Cell Line Datasets Separated, Improved Error Handling for all Build Scripts

[jjacobson95]

Ready to merge.

Multiple Changes will be coming with this PR.

Summary:

• Broad_sanger is separated into all of its component datasets. These include:

  • CCLE
  • gCSI
  • GDSCv1
  • GDSCv2
  • CTRPv2
  • FIMM
  • PRISM

• This separation process includes several steps, that listed here to address Break down broad_sanger datasets into individual datasets with no duplicates #227:

  • build/broad_sanger/05_separate_datasets.py script - This performs the separation script.
  • build/broad_sanger/build_misc.sh - This calls the separation script in the broad_sanger_omics dockerfile.
  • build/build_all.py and build/build_dataset.py were modified to have a process_misc step. This is only called with broad_sanger but maybe other future datasets will require an additional final processing step as well.
  • Validation and schema test for each listed dataset and associated updated to check_all_schemas.py.

• Error Handling improved. Every bash script in the build process has been modified to appropriately pass an exit code to docker when its R or Python scripts fail.

  • This will address the uncaught validation error in Validation ending with a success message even when failing. #233.
  • It also has some potential to affect Drug generation Uncaught Failures #125, though we need a more direct method to handle this issue.

• Schema Checking revamped. All of the schema checking code has been moved into three files: schema/coderdata.yaml, schema/expected_files.yaml, scripts/check_schema.py.

• Multiple Bug fixes and code updates to align to schema. These include Validation ending with a success message even when failing. #233, Should the schema be modified to allow "inf" values in dose_response_value? #234, NCI60 - Several incorrect values in chem_name #236, NCI60 Polars Version Error #238, Add more robustness in Synapse file download scripts. #239, Schema Issues Found #240.

[sgosline]

Is there a way to generalize the validation script so we only need one for all datasets? It feels like this is redundant and increases the burden on anyone wanting to add a dataset. Since all the files have the same file name structure ([datasetname]_[datatype]), can we just iterate through a directory and validate each one according to the file name?

[jjacobson95]

Yes, that is a more maintainable approach, I'll make that change!

Likely will use a decent amount of hard coded file names to ensure that all expected files are present (as is currently done) rather than simple pattern matching.

[sgosline]

I'd avoid hard-coded files. Perhaps use a config file that can be updated so the validation code itself remains the same.

[jjacobson95]

Good call, that sounds more elegant

[sgosline]

Please merge #242 before this.

[jjacobson95]

Merged. I've got one bug that I'm working on (The broad_sanger data split is working in the individual dataset script, but not in the build_all script) and then I'll rebuild all of the data.

[jjacobson95]

Just a note, I'm running into consistent memory issues when splitting the broad_sanger data in docker on AWS. I'm reworking this script to reduce the memory usage.

[jjacobson95]

Resolves #227. This is the most notable update.
Resolves #233, #234, #236 (with a schema fix to allow "Any"), #238, and #240. These are all bug and schema updates.
-We never made this an issue, but this moves all of the schema checking scripts into a single unified script that runs schema checking in parallel. Old schema code has been removed.

Closes #239. Not because this was resolved, but because it is a non-issue.

[jjacobson95]

Ready to merge!

[sgosline]

I reviewed this, and it looks solid!