feat!(backend): add column in backend for raw original data that should never be modified

[anna-parker]

#3262

Adds a new field to the database called unprocessed_data. It is for now populated with a copy of the original_data jsonb column. unprocessed_data will be used instead of original_data for calls to preprocessing/requests for original data. original_data on the other hand will not be used and serves only as a backup/storage of the initial metadata we recieved. This will simplify the reprocessing process as we can migrate unprocessed_data in a db migration which is not stored in the code but still keep a record of the data originally submitted to us.

The biggest code change is the subsequent field renaming that I did to make the distrinction between original and unprocessed data clearer in the code.

Breaking change

The get-original-data endpoint is now called get-unprocessed-data endpoint.

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)
  Preview comes up, revision (which uses affected endpoint) via the webpage is WAI: LOC_00001V8.2

🚀 Preview: https://add-new-column.loculus.org

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[maverbiest]

Looks good!

Connected to the DB via port forwarding. Checked that this query returns all rows:

SELECT * FROM public.sequence_entries
WHERE original_data = unprocessed_data;

And that this query returns no rows:

SELECT * FROM public.sequence_entries
WHERE original_data != unprocessed_data;

I also did spot checks on the preview web page and data seems to be coming up fine, with all the data as expected in the sequence details view I checked.

[anna-parker]

I guess it could now be a tad confusing the the prepro pipeline calls extract-unprocessed-data to get unprocessed data - previously original data (updates the status to IN_PROCESSING) and the get-unprocessed-data (previously get-original-data does the same thing just without setting the status to IN_PROCESSING.

This is a preexisting naming issue, but might need a second thought, @theosanderson what do you think?