GPL-733 As a developer I would like to run a Manual MLWH and DART Migration #144
Closed
9 tasks done
Labels
Beckman integration
Beckman integration
User story
As developers we want to ensure that if a sample upload to the MLWH or DART databases fails with a critical exception, but the upload to mongo succeeds, we are able to run a manual process to pull samples from mongodb added between timestamps and attempt to re-add them to MLWH and DART (see existing manual MLWH migration that already exists in crawler - we want a combined version that keeps both in sync). The process needs to be idempotent, such that data that already exists is not overwritten. It will help to ensure the mongo, MLWH and DART databases remain in-sync through MLWH and DART upload failures. NB. DART inserts should only be done if samples for a particular plate have NOT already been cherrypicked. This does not handle changing filtered positive rules. See GPL-709 for this
Acceptance criteria
To be considered successful the solution must:
filtered_positive_version
,lh_sample_uuid
andlh_source_plate_uuid
properties. If not we shouldn't add them as DART would become inconsistent/Beckman workflow expects these propertiesNeeds discussing: Re-determine if filtered positive? Is this required? Filtered positive fields should already be recorded in mongo, and there is a separate migration to update databases following filtered positive rule changes (GPL-709)Needs discussing: Re-add to Mongo & MLWH (only necessary if re-determining filtered positive fields)in the normal file processing, don't attempt to add to DART if the MLWH insert failsSplit to GPL-764 Do not update DART if the MLWH insert fails #162Dependencies
This story is blocked by the following dependencies:
Additional context
NB. There are specific error types that will show for a file in the Lighthouse-UI imports screen. This will need to be monitored to indicate to us when we need to run the migration.
The migration should also be able to insert legacy data for us, e.g. to get the initial plates into DART for cherrypicking.
The text was updated successfully, but these errors were encountered: