Series filtering out based on description string

[jeremyestein]

Fixes #346

Apply series filtering in orthanc anon, just before anonymisation. So if series filtering settings change, we can re-run without having to re-download everything to orthanc raw.

TODO

• System tests
  • In _check_dcm_tags_from_zip, need to set the expected study/series/instance structure (see _populate_vna for the source data)
  • Explain why unhashed AccNum and PatientID ended up in orthanc-anon(!) Might be due to throwing in the received instance callback. Investigate separately? (see orthanc-anon ReceivedInstanceCallback must not allow exceptions to go uncaught or anonymisation will be compromised #368 )
  • Fix whatever is causing instances in the same study to not end up in the same zip file - The in-test generated dicom data is not mixing well with the static dicom files, so they are not recognised as the same study.
  • Decide if we need anything more complicated than series exclusion based on simple substring matches (eg. an include list, regexes...)

Approaches considered but not implemented

• Download study to orthanc raw as before, then delete series/instances that we don't want using the API
• Query VNA (without /retrieve) for series matching criteria, locally filter, and then instead of downloading studies as we do now, query (with /retrieve) for series IDs that we do want

Problems with the above:

• Would need to requery VNA if series filtering criteria changed
• Would need to keep track of downloads in the DB at a series level. Would CLI know what to re-request?
• Some series level operations not even possible?

[stefpiatek]

Explain why unhashed AccNum and PatientID ended up in orthanc-anon(!) Might be due to throwing in the received instance callback. Investigate separately?

I think create a separate issue. Probably worth querying for the pseudonymised study ID in PIXL DB before exporting, then updating the ORM instance with the export date after the export? That way if the pseudonymised ID hasn't been added, it won't be found in the database and an exception will be thrown and it won't be exported

Fix whatever is causing instances in the same study to not end up in the same zip file - The in-test generated dicom data is not mixing well with the static dicom files, so they are not recognised as the same study.

IIRC for system tests we decreased the waiting time for a study to be taken as stable, might be that? Or are they seemingly a different study altogether?

[stefpiatek]

Looking good, thanks Jeremy!