diff --git a/docs/fix_plan.md b/docs/fix_plan.md new file mode 100644 index 0000000..d559cb0 --- /dev/null +++ b/docs/fix_plan.md @@ -0,0 +1,236 @@ +# Fix Plan: Pipeline & Import Errors + +Tracking doc for the errors identified in the first-pass review (2026-08-01). +Each item lists the defect, the fix, and how to verify it. Work top to +bottom — later Nextflow stages depend on earlier ones, so the pipeline +can't be smoke-tested end-to-end until items 1-3 are resolved. + +**Status (2026-08-02, branch `fix/pipeline-wiring-bugs`):** Items 1-7 +implemented. Item 8 (dead code) left untouched — needs an author decision, +not a mechanical fix. + +**Additional bugs found during implementation, beyond this plan's original +scope**, all in `PROCESS_ALL_CONTACTS` and `PRODUCE_NEO4J_FILES` — the +same two processes items 1-3 were already fixing. Verified by diffing each +process's declared Nextflow `output:`/script CLI flags against what the +underlying Python script's argparse and `to_csv()` calls actually use: + +- `PROCESS_ALL_CONTACTS` script call used `--contacts` but + `process_all_pdb_contacts.py`'s argparse expects `--contacts_file` — + would have failed immediately with an "unrecognized arguments" error. +- `PROCESS_ALL_CONTACTS` never passed `--scop2_descriptions_file` at all + (same missing-param class of bug as item 2, confirmed to also apply + here, not just to the file being absent from `nextflow.config`). +- `PROCESS_ALL_CONTACTS`'s declared outputs `scop2b_fa_pdb_residue_interactions.csv.gz` + / `scop2b_sf_pdb_residue_interactions.csv.gz` don't match what the script + actually writes — `scop2_fa_...` / `scop2_sf_...` (no "b") — a second, + previously-uncaught instance of the same bug class as item 3. +- `PRODUCE_NEO4J_FILES` input `superfamily_domain_ownership` was declared + as a process input but never actually passed as a `--superfamily_domain_ownership` + flag on the script's command line at all. +- `PRODUCE_NEO4J_FILES`'s entire declared `output:` block (58 entries) was + stale relative to the script: every filename used `.csv.gz` while the + script writes `.tsv.gz` throughout, several files were renamed + (`g3dsa_domains_nodes` → `gene3d_domains_nodes`, `scop2b_fa`/`scop2b_sf` + → `scop2_fa`/`scop2_sf`, etc.), and 13 files the script writes + (SCOP2 hierarchy nodes/rels, `superfamily_fold_rels`, + `cath_topology_domain_rels`) weren't declared as outputs at all. The + corrected 71-entry list was verified two ways: (1) extracting every + `to_csv(f"...")` call from `produce_neo4j_files.py` directly, and (2) + cross-checking against `nextflow/bin/import_neo4j_data.sh`, which already + references the correct actual filenames (e.g. `gene3d_domains_nodes.tsv.gz`), + confirming the import script was kept in sync with the Python script + while `main.nf`'s output declarations were not. +- `PRODUCE_NEO4J_FILES` was also missing `scop2_domains_info`/ + `scop2_domains_description` inputs entirely (same root cause as item 2 — + `produce_neo4j_files.py` requires `--scop2_domains_info_file` and + `--scop2_descriptions_file`, neither of which existed as inputs or was + passed on the command line). + +None of this changes the plan's original fixes — it's the same items 1-3, +just more extensively broken than the original review (a static read, +without cross-referencing every script's argparse/output against +`main.nf`) caught. Fixed on the same branch as part of the same items. + +## 1. Undefined `interpro_domain_ownership` in `PRODUCE_NEO4J_FILES` + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** [`nextflow/main.nf:205`](../nextflow/main.nf#L205) (call site), + input block [`nextflow/main.nf:126-142`](../nextflow/main.nf#L126) +- **Problem:** the process call references + `--interpro_domain_ownership ${interpro_domain_ownership}`, but no such + variable is declared as a process input. Declared inputs are + `superfamily_domain_ownership`, `g3dsa_domain_ownership`, + `scop2b_sf_domain_ownership`, `scop2b_fa_domain_ownership`. +- **Fix:** + 1. Check `produce_neo4j_files.py`'s argparse definition to confirm the + full, current set of expected `--*_domain_ownership` flags. + 2. Add any missing `path` inputs to the `PRODUCE_NEO4J_FILES` input block + and wire them from the correct upstream channel in the workflow block + (likely the CATH-Gene3D/SCOP2/InterPro domain ownership channels + produced earlier in `main.nf`). + 3. Fix the call line to reference only variables that exist as inputs, + with names matching what the script expects + (`--superfamily_domain_ownership`, `--gene3dsa_domain_ownership`, + `--scop2_sf_domain_ownership`, `--scop2_fa_domain_ownership`, etc.). +- **Verify:** `nextflow run main.nf -profile standard -stub-run` (or a real + run against a small structure set) reaches `PRODUCE_NEO4J_FILES` without + an undefined-variable/null error, and all expected `*_domain_ownership` + files are non-empty in the output. + +## 2. Missing SCOP2 descriptions file for `PROCESS_ALL_CONTACTS` + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** `PROCESS_ALL_CONTACTS` process call in `nextflow/main.nf` + (around line 214); consumed by + [`nextflow/bin/process_all_pdb_contacts.py`](../nextflow/bin/process_all_pdb_contacts.py) + via `utils.get_scop2_domains_info`. +- **Problem:** the script requires a SCOP2 descriptions file path, but + `main.nf` never passes one, and no corresponding `params.*` entry exists + in `nextflow.config`. +- **Fix:** + 1. Determine where the SCOP2 descriptions file is expected to come from + (a downloaded reference file, similar to other SCOP2/CATH reference + data already staged elsewhere in the pipeline). + 2. Add a `params.scop2_descriptions_file` (or equivalent) entry to + `nextflow.config`, wire it as a process input, and pass it on the + `process_all_pdb_contacts.py` command line. + 3. Confirm the flag name matches the script's argparse definition exactly. +- **Verify:** `PROCESS_ALL_CONTACTS` runs without the + `TypeError: expected str, bytes or os.PathLike object, not NoneType` + currently produced, and SCOP2 domain info appears correctly in its output. + +## 3. Output filename mismatch: `g3dsa_pdb_residue_interactions.csv.gz` + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** declared output + [`nextflow/main.nf:91`](../nextflow/main.nf#L91) (`g3dsa_...`) vs. actual + file written in + [`nextflow/bin/process_all_pdb_contacts.py:343,346`](../nextflow/bin/process_all_pdb_contacts.py#L343) + (`gene3dsa_...`). +- **Problem:** Nextflow will report a missing output file because the + process declares a different filename than the script produces. +- **Fix:** Pick one name and use it consistently — recommend standardizing + on `gene3dsa_...` (matches the script and is the less ambiguous + abbreviation) and updating `main.nf:91` to match, rather than changing + the Python script. Grep the rest of `main.nf` and `nextflow/bin/` for + `g3dsa_pdb_residue_interactions` to make sure no other reference still + uses the old name. +- **Verify:** the process's declared `output:` block and the file(s) + actually written by the script are identical strings; pipeline proceeds + past this process without a missing-output error. + +## 4. Missing comma silently drops SCOP2/Pfam domains from filter + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** [`nextflow/bin/process_pdb_contacts.py:188`](../nextflow/bin/process_pdb_contacts.py#L188) + ```python + .isin(["CATH", "SCOP", "SCOP2B", "SCOP2" "Pfam", "InterPro"]) + ``` +- **Problem:** adjacent string literals `"SCOP2" "Pfam"` concatenate into a + single string `"SCOP2Pfam"` that matches nothing in the data. Structures + with genuine SCOP2 or Pfam domain annotations are silently excluded from + this filter — no error, just missing data downstream. +- **Fix:** add the missing comma: + ```python + .isin(["CATH", "SCOP", "SCOP2B", "SCOP2", "Pfam", "InterPro"]) + ``` +- **Verify:** re-run on a structure known to have SCOP2 and/or Pfam + annotations (spot-check via SIFTS) and confirm `mmcif_domains` / + `domain_info_dataframe_filtered` now includes them. Since this bug + produces no exception, add a quick assertion or log line counting rows + per `xref_db` value so future regressions of this kind are caught + immediately rather than silently. + +## 5. Copy-paste error: SCOP class node mislabeled in Neo4j import + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** [`nextflow/bin/import_neo4j_data.sh:35`](../nextflow/bin/import_neo4j_data.sh#L35) + ``` + --relationships=IS_IN_SCOP_CLASS=import/scop_fold_class_rels.tsv.gz \ + --nodes=IS_IN_SCOP_CLASS=import/scop_class_nodes.tsv.gz \ + ``` +- **Problem:** the `--nodes` flag reuses the relationship type + `IS_IN_SCOP_CLASS` as a node label instead of a proper label like + `scopClass`. Every SCOP class node imported this way is mislabeled, + breaking any Cypher query, constraint, or dashboard panel that expects + the correct label. +- **Fix:** change the node line's label to `scopClass` (matching the + naming convention of the sibling lines, e.g. `scopSuperfamily`, + `scopFold`, `cathClass`). Double-check `import_neo4j_data_minimal.sh` and + any Cypher/NeoDash queries under `procogdash/` don't already assume the + broken label (i.e. that nothing was built to work around this bug). +- **Verify:** after a fresh `compose_build.yaml` import, run + `MATCH (n:scopClass) RETURN count(n)` in Neo4j Browser and confirm it + returns the expected non-zero count, and `MATCH (n:IS_IN_SCOP_CLASS)` + returns zero. + +## 6. Stale Zenodo version reference in docs (lower confidence) + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** `docs/installation.md` vs. the Zenodo record/version + downloaded by `setup_docker_linux.sh` / `setup_docker_windows.ps1`. +- **Problem:** the doc appears to reference an older data release than what + the setup scripts actually pull. +- **Fix:** confirm the current Zenodo record ID/version in the setup + scripts, and update `docs/installation.md` (and any other doc/README + mentioning a version) to match. Consider referencing the version from a + single place (e.g. an env var or top-of-script constant) to avoid drift + in future. +- **Verify:** every doc/script mentioning the Zenodo version string agrees. + +## 7. Hardcoded author-specific conda path (lower confidence) + +**Status: fixed on `fix/pipeline-wiring-bugs`.** + +- **Where:** [`nextflow/main.nf:25`](../nextflow/main.nf#L25) vs. the + portable `$baseDir`-relative conda path pattern used in + `nextflow.config`. +- **Problem:** an absolute, machine-specific conda environment path breaks + the pipeline for anyone who isn't the original author running it + directly (outside the Docker-based prebuilt-flat-files path). +- **Fix:** replace the hardcoded path with the same `$baseDir`/relative + pattern used elsewhere, so a fresh clone + `envs/procoggraph.yaml` setup + works without editing `main.nf`. +- **Verify:** clone into a clean directory (or `git worktree add`), build + the conda env from `envs/procoggraph.yaml`, and confirm `main.nf` doesn't + reference any path outside the checkout. + +## 8. Possible dead code (lower confidence, needs a decision) + +- **Where:** `nextflow/bin/produce_minimal_neo4j_files.py`, + `nextflow/bin/produce_flat_files.py`, + `nextflow/bin/import_neo4j_data_minimal.sh`. +- **Problem:** none of these appear referenced from `main.nf`, the setup + scripts, or docs. Either they're dead code left over from a prior + approach, or they support an undocumented "minimal" install path. +- **Fix (needs author decision, not a mechanical fix):** determine which + case it is. + - If dead: remove the files (git history preserves them if needed later). + - If it's a real, supported minimal-install path: wire it into `main.nf` + or the setup scripts explicitly, and document it in + `docs/installation.md`. +- **Verify:** N/A until the decision is made; then verify per whichever + branch is chosen above. + +## Suggested order of work + +1. Items 1-3 (pipeline wiring) — the pipeline cannot run end-to-end without + these, so fix and smoke-test together against a small structure set. +2. Item 4 (silent data-correctness bug) — high impact, cheap fix. +3. Item 5 (Neo4j import mislabeling) — verify against a fresh + `compose_build.yaml` import once 1-3 are stable, since a full pipeline + run + import is the only real way to confirm it. +4. Items 6-8 — low risk, can be done independently at any point; item 8 + needs a decision from the author before any file is touched. + +No automated test suite exists yet. Once items 1-4 are fixed, consider +adding at least one smoke test (a tiny fixture structure run through +`process_pdb_contacts.py` / `process_all_pdb_contacts.py`) so regressions +like #4 fail loudly instead of silently next time. diff --git a/docs/installation.md b/docs/installation.md index 8094c48..1be39ce 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -58,12 +58,12 @@ NOTE: Currently, the NeoDash Docker image does not contain a build for arm based Installation instructions for running the database on bare metal, rather than Docker, are described below. -1. Download the latest database flat files from Zenodo [here](https://zenodo.org/records/13165852) and clone the ProCogGraph repository: +1. Download the latest database flat files from Zenodo [here](https://zenodo.org/records/14046116) and clone the ProCogGraph repository: ``` bash git clone Bashton-Lab/ProCogGraph - curl https://zenodo.org/records/13165852/files/procoggraph_flat_files_v1-0.zip?download=1 -o /PATH/TO/DATABASE_FLAT_FILES/procoggraph_flat_files_v1-0.zip - unzip /PATH/TO/DATABASE_FLAT_FILES/procoggraph_flat_files_v1-0.zip + curl https://zenodo.org/records/14046116/files/procoggraph_flat_files_v1-0-2.zip?download=1 -o /PATH/TO/DATABASE_FLAT_FILES/procoggraph_flat_files_v1-0-2.zip + unzip /PATH/TO/DATABASE_FLAT_FILES/procoggraph_flat_files_v1-0-2.zip ``` 2. Download and install Neo4j community edition from the [Neo4j website](https://neo4j.com/download/). The database was built using Neo4j version 5. diff --git a/nextflow/bin/import_neo4j_data.sh b/nextflow/bin/import_neo4j_data.sh index 7205cd3..1b6ab66 100755 --- a/nextflow/bin/import_neo4j_data.sh +++ b/nextflow/bin/import_neo4j_data.sh @@ -32,7 +32,7 @@ bin/neo4j-admin database import full \ --relationships=IS_IN_SCOP_FOLD=import/superfamily_fold_rels.tsv.gz \ --nodes=scopFold=import/scop_fold_nodes.tsv.gz \ --relationships=IS_IN_SCOP_CLASS=import/scop_fold_class_rels.tsv.gz \ ---nodes=IS_IN_SCOP_CLASS=import/scop_class_nodes.tsv.gz \ +--nodes=scopClass=import/scop_class_nodes.tsv.gz \ --nodes=cathClass=import/cath_class_nodes.tsv.gz \ --relationships=IS_IN_CATH_CLASS=import/cath_class_architecture_rels.tsv.gz \ --nodes=cathArchitecture=import/cath_architecture_nodes.tsv.gz \ diff --git a/nextflow/bin/process_pdb_contacts.py b/nextflow/bin/process_pdb_contacts.py index 5cbf4d0..f5bc555 100644 --- a/nextflow/bin/process_pdb_contacts.py +++ b/nextflow/bin/process_pdb_contacts.py @@ -185,7 +185,7 @@ def process_manifest_row(row, cutoff): ##we check in process_pdb_structure that domains are present for the structure, so no need to fail here. domain_info_dataframe = pd.DataFrame(block.find("_pdbx_sifts_xref_db_segments.", ["entity_id", "asym_id", "xref_db", "xref_db_acc", "domain_name", "segment_id", "instance_id", "seq_id_start", "seq_id_end"]), columns = ["entity_id", "asym_id", "xref_db", "xref_db_acc", "domain_name", "segment_id", "instance_id", "seq_id_start", "seq_id_end"]) - domain_info_dataframe_filtered = domain_info_dataframe.loc[domain_info_dataframe.xref_db.isin(["CATH", "SCOP", "SCOP2B", "SCOP2" "Pfam", "InterPro"])] + domain_info_dataframe_filtered = domain_info_dataframe.loc[domain_info_dataframe.xref_db.isin(["CATH", "SCOP", "SCOP2B", "SCOP2", "Pfam", "InterPro"])] if len(domain_info_dataframe_filtered) > 0: mmcif_domains = domain_info_dataframe_filtered.xref_db.unique().tolist() diff --git a/nextflow/main.nf b/nextflow/main.nf index 9e39555..ebab36c 100644 --- a/nextflow/main.nf +++ b/nextflow/main.nf @@ -22,7 +22,7 @@ process PROCESS_MMCIF { process RUN_ARPEGGIO { label 'arpeggio' //gives 18gb mem for arpeggio in slurm submission (offering overhead for complex structures) cache 'lenient' - conda '/raid/MattC/repos/envs/arpeggio-env.yaml' + conda "${workflow.projectDir}/envs/arpeggio-env.yaml" publishDir "${params.publish_dir}/arpeggio", mode: 'copy' input: path arpeggio_batch @@ -71,6 +71,7 @@ process PROCESS_ALL_CONTACTS { path scop_domains_info path scop_domains_description path scop2_domains_info + path scop2_domains_description path interpro_xml path cath_names path cddf @@ -88,13 +89,13 @@ process PROCESS_ALL_CONTACTS { path("scop_pdb_residue_interactions.csv.gz"), emit: scop path("pfam_pdb_residue_interactions.csv.gz"), emit: pfam path("superfamily_pdb_residue_interactions.csv.gz"), emit: superfamily - path("g3dsa_pdb_residue_interactions.csv.gz"), emit: g3dsa - path("scop2b_fa_pdb_residue_interactions.csv.gz"), emit: scop2b_fa - path("scop2b_sf_pdb_residue_interactions.csv.gz"), emit: scop2b_sf + path("gene3dsa_pdb_residue_interactions.csv.gz"), emit: g3dsa + path("scop2_fa_pdb_residue_interactions.csv.gz"), emit: scop2b_fa + path("scop2_sf_pdb_residue_interactions.csv.gz"), emit: scop2b_sf script: """ - python3 ${workflow.projectDir}/bin/process_all_pdb_contacts.py --contacts ${combined_contacts} --ccd_cif ${ccd_cif} --pfam_a_file ${pfam_a} --pfam_clan_rels ${pfam_clan_rels} --pfam_clans ${pfam_clans} --scop_domains_info_file ${scop_domains_info} --scop_descriptions_file ${scop_domains_description} --scop2_domains_info_file ${scop2_domains_info} --interpro_xml ${interpro_xml} --cath_names ${cath_names} --cddf ${cddf} --glycoct_cache ${glycoct_cache} --smiles_cache ${smiles_cache} --csdb_linear_cache ${csdb_linear_cache} --enzyme_dat_file ${enzyme_dat_file} --enzyme_class_file ${enzyme_class_file} --sifts_ec_mapping ${sifts_ec_mapping} + python3 ${workflow.projectDir}/bin/process_all_pdb_contacts.py --contacts_file ${combined_contacts} --ccd_cif ${ccd_cif} --pfam_a_file ${pfam_a} --pfam_clan_rels ${pfam_clan_rels} --pfam_clans ${pfam_clans} --scop_domains_info_file ${scop_domains_info} --scop_descriptions_file ${scop_domains_description} --scop2_domains_info_file ${scop2_domains_info} --scop2_descriptions_file ${scop2_domains_description} --interpro_xml ${interpro_xml} --cath_names ${cath_names} --cddf ${cddf} --glycoct_cache ${glycoct_cache} --smiles_cache ${smiles_cache} --csdb_linear_cache ${csdb_linear_cache} --enzyme_dat_file ${enzyme_dat_file} --enzyme_class_file ${enzyme_class_file} --sifts_ec_mapping ${sifts_ec_mapping} """ } @@ -140,69 +141,84 @@ process PRODUCE_NEO4J_FILES { path rhea2ec path rhea_directions path rhea_reaction_smiles + path scop2_domains_info + path scop2_domains_description output: - path "ec_id_nodes.csv.gz" - path "ec_nodes_class.csv.gz" - path "ec_nodes_subclass.csv.gz" - path "ec_nodes_subsubclass.csv.gz" - path "ec_class_subclass_rel.csv.gz" - path "ec_subclass_subsubclass_rel.csv.gz" - path "ec_subsubclass_id_rel.csv.gz" - path "cognate_ligand_nodes.csv.gz" - path "cognate_ligands_ec.csv.gz" - path "pdb_entry_nodes.csv.gz" - path "pdb_protein_chain_nodes.csv.gz" - path "protein_ec_rels.csv.gz" - path "pdb_protein_rels.csv.gz" - path "scop_domains_nodes.csv.gz" - path "cath_domains_nodes.csv.gz" - path "superfamily_domains_nodes.csv.gz" - path "g3dsa_domains_nodes.csv.gz" - path "scop2b_fa_domains_nodes.csv.gz" - path "scop2b_sf_domains_nodes.csv.gz" - path "pfam_domains_nodes.csv.gz" - path "scop_family_nodes.csv.gz" - path "scop_superfamily_nodes.csv.gz" - path "scop_fold_nodes.csv.gz" - path "scop_class_nodes.csv.gz" - path "scop_domain_family_rels.csv.gz" - path "scop_family_superfam_rels.csv.gz" - path "scop_superfam_fold_rels.csv.gz" - path "scop_fold_class_rels.csv.gz" - path "cath_class_nodes.csv.gz" - path "cath_architecture_nodes.csv.gz" - path "cath_topology_nodes.csv.gz" - path "cath_homologous_superfamily_nodes.csv.gz" - path "cath_class_architecture_rels.csv.gz" - path "cath_architecture_topology_rels.csv.gz" - path "cath_topology_homology_rels.csv.gz" - path "cath_homologous_superfamily_domain_rels.csv.gz" - path "pfam_clans.csv.gz" - path "pfam_clan_rels.csv.gz" - path "bound_descriptors.csv.gz" - path "bound_entities.csv.gz" - path "be_bd_rels.csv.gz" - path "be_pdb_rels.csv.gz" - path "bound_entity_parity_score_rels.csv.gz" - path "cath_domain_ligand_interactions.csv.gz" - path "scop_domain_ligand_interactions.csv.gz" - path "pfam_domain_ligand_interactions.csv.gz" - path "superfamily_domain_ligand_interactions.csv.gz" - path "g3dsa_domain_ligand_interactions.csv.gz" - path "scop2b_fa_domain_ligand_interactions.csv.gz" - path "scop2b_sf_domain_ligand_interactions.csv.gz" - path "cath_protein_rels.csv.gz" - path "scop_protein_rels.csv.gz" - path "pfam_protein_rels.csv.gz" - path "superfamily_protein_rels.csv.gz" - path "g3dsa_protein_rels.csv.gz" - path "scop2b_fa_protein_rels.csv.gz" - path "scop2b_sf_protein_rels.csv.gz" - path "procoggraph_node.csv.gz" + path "ec_id_nodes.tsv.gz" + path "ec_nodes_class.tsv.gz" + path "ec_nodes_subclass.tsv.gz" + path "ec_nodes_subsubclass.tsv.gz" + path "ec_class_subclass_rel.tsv.gz" + path "ec_subclass_subsubclass_rel.tsv.gz" + path "ec_subsubclass_id_rel.tsv.gz" + path "cognate_ligand_nodes.tsv.gz" + path "cognate_ligands_ec.tsv.gz" + path "pdb_entry_nodes.tsv.gz" + path "pdb_protein_chain_nodes.tsv.gz" + path "protein_ec_rels.tsv.gz" + path "scop_domains_nodes.tsv.gz" + path "scop2_fa_domains_nodes.tsv.gz" + path "scop2_sf_domains_nodes.tsv.gz" + path "scop2_fa_nodes.tsv.gz" + path "scop2_sf_nodes.tsv.gz" + path "scop2_cf_nodes.tsv.gz" + path "scop2_cl_nodes.tsv.gz" + path "scop2_tp_nodes.tsv.gz" + path "scop2_sf_fa_rels.tsv.gz" + path "scop2_sf_cf_rels.tsv.gz" + path "scop2_cf_cl_rels.tsv.gz" + path "scop2_cl_tp_rels.tsv.gz" + path "scop2_fa_domains_rels.tsv.gz" + path "scop2_sf_domains_rels.tsv.gz" + path "cath_domains_nodes.tsv.gz" + path "superfamily_domains_nodes.tsv.gz" + path "gene3d_domains_nodes.tsv.gz" + path "pfam_domains_nodes.tsv.gz" + path "scop_family_nodes.tsv.gz" + path "scop_superfamily_nodes.tsv.gz" + path "scop_fold_nodes.tsv.gz" + path "scop_class_nodes.tsv.gz" + path "scop_domain_family_rels.tsv.gz" + path "superfamily_fold_rels.tsv.gz" + path "scop_family_superfam_rels.tsv.gz" + path "scop_superfam_fold_rels.tsv.gz" + path "scop_fold_class_rels.tsv.gz" + path "cath_class_nodes.tsv.gz" + path "cath_architecture_nodes.tsv.gz" + path "cath_topology_nodes.tsv.gz" + path "cath_homologous_superfamily_nodes.tsv.gz" + path "cath_class_architecture_rels.tsv.gz" + path "cath_architecture_topology_rels.tsv.gz" + path "cath_topology_homology_rels.tsv.gz" + path "cath_homologous_superfamily_domain_rels.tsv.gz" + path "cath_topology_domain_rels.tsv.gz" + path "pfam_clans.tsv.gz" + path "pfam_clan_rels.tsv.gz" + path "bound_descriptors.tsv.gz" + path "be_bd_rels.tsv.gz" + path "bound_entities.tsv.gz" + path "be_pdb_rels.tsv.gz" + path "bound_entity_parity_score_rels.tsv.gz" + path "cath_domain_ligand_interactions.tsv.gz" + path "scop_domain_ligand_interactions.tsv.gz" + path "pfam_domain_ligand_interactions.tsv.gz" + path "superfamily_domain_ligand_interactions.tsv.gz" + path "gene3dsa_domain_ligand_interactions.tsv.gz" + path "scop2_sf_domain_ligand_interactions.tsv.gz" + path "scop2_fa_domain_ligand_interactions.tsv.gz" + path "pdb_protein_rels.tsv.gz" + path "cath_protein_rels.tsv.gz" + path "scop_protein_rels.tsv.gz" + path "pfam_protein_rels.tsv.gz" + path "scop2_fa_protein_rels.tsv.gz" + path "scop2_sf_protein_rels.tsv.gz" + path "gene3d_protein_rels.tsv.gz" + path "superfamily_protein_rels.tsv.gz" + path "procoggraph_node.tsv.gz" script: """ - python3 ${workflow.projectDir}/bin/produce_neo4j_files.py --enzyme_dat_file ${enzyme_dat} --enzyme_class_file ${enzyme_class} --cognate_ligands ${cognate_ligands} --cath_domain_ownership ${cath_domain_ownership} --scop_domain_ownership ${scop_domain_ownership} --interpro_domain_ownership ${interpro_domain_ownership} --pfam_domain_ownership ${pfam_domain_ownership} --bound_entities ${bound_entities} --parity_calcs ${all_parity_calcs} --parity_threshold ${parity_threshold} --rhea2ec ${rhea2ec} --rhea_dir ${rhea_directions} --rhea_reaction_smiles ${rhea_reaction_smiles} + python3 ${workflow.projectDir}/bin/produce_neo4j_files.py --enzyme_dat_file ${enzyme_dat} --enzyme_class_file ${enzyme_class} --cognate_ligands ${cognate_ligands} --cath_domain_ownership ${cath_domain_ownership} --scop_domain_ownership ${scop_domain_ownership} --scop2_sf_domain_ownership ${scop2b_sf_domain_ownership} --scop2_fa_domain_ownership ${scop2b_fa_domain_ownership} --pfam_domain_ownership ${pfam_domain_ownership} --superfamily_domain_ownership ${superfamily_domain_ownership} --gene3dsa_domain_ownership ${g3dsa_domain_ownership} --scop2_domains_info_file ${scop2_domains_info} --scop2_descriptions_file ${scop2_domains_description} --bound_entities ${bound_entities} --parity_calcs ${all_parity_calcs} --parity_threshold ${parity_threshold} --rhea2ec ${rhea2ec} --rhea_dir ${rhea_directions} --rhea_reaction_smiles ${rhea_reaction_smiles} """ } @@ -211,7 +227,7 @@ workflow { arpeggio_batches = processed_struct_manifest.arpeggio_batch.flatten() arpeggio = RUN_ARPEGGIO( arpeggio_batches ).collect() contacts = PROCESS_CONTACTS( arpeggio, processed_struct_manifest.updated_manifest, params.domain_contact_cutoff ) - all_contacts = PROCESS_ALL_CONTACTS( contacts.combined_contacts, Channel.fromPath("${params.ccd_cif}"), Channel.fromPath("${params.pfam_a_file}"), Channel.fromPath("${params.pfam_clan_rels}"), Channel.fromPath("${params.pfam_clans}"), Channel.fromPath("${params.scop_domains_info_file}"), Channel.fromPath("${params.scop_descriptions_file}"), Channel.fromPath("${params.scop2_domains_info_file}"), Channel.fromPath("${params.interpro_xml}"), Channel.fromPath("${params.cath_names}"), Channel.fromPath("${params.cddf}"), Channel.fromPath("${params.glycoct_cache}"), Channel.fromPath("${params.smiles_cache}"), Channel.fromPath("${params.csdb_linear_cache}"), Channel.fromPath("${params.enzyme_dat_file}"), Channel.fromPath("${params.enzyme_class_file}"), Channel.fromPath("${params.sifts_file}") ) + all_contacts = PROCESS_ALL_CONTACTS( contacts.combined_contacts, Channel.fromPath("${params.ccd_cif}"), Channel.fromPath("${params.pfam_a_file}"), Channel.fromPath("${params.pfam_clan_rels}"), Channel.fromPath("${params.pfam_clans}"), Channel.fromPath("${params.scop_domains_info_file}"), Channel.fromPath("${params.scop_descriptions_file}"), Channel.fromPath("${params.scop2_domains_info_file}"), Channel.fromPath("${params.scop2_descriptions_file}"), Channel.fromPath("${params.interpro_xml}"), Channel.fromPath("${params.cath_names}"), Channel.fromPath("${params.cddf}"), Channel.fromPath("${params.glycoct_cache}"), Channel.fromPath("${params.smiles_cache}"), Channel.fromPath("${params.csdb_linear_cache}"), Channel.fromPath("${params.enzyme_dat_file}"), Channel.fromPath("${params.enzyme_class_file}"), Channel.fromPath("${params.sifts_file}") ) score_ligands = SCORE_LIGANDS( all_contacts.bound_entities, Channel.fromPath(params.cognate_ligands), Channel.fromPath(params.parity_cache), Channel.from(params.parity_threshold) ) - produce_neo4j_files = PRODUCE_NEO4J_FILES( score_ligands.all_parity_calcs, Channel.fromPath(params.cognate_ligands) , all_contacts.bound_entities, all_contacts.cath, all_contacts.scop, all_contacts.pfam, all_contacts.superfamily, all_contacts.g3dsa, all_contacts.scop2b_sf, all_contacts.scop2b_fa, Channel.fromPath("${params.enzyme_dat_file}"), Channel.fromPath("${params.enzyme_class_file}"), Channel.from(params.parity_threshold), Channel.fromPath("${params.rhea2ec}"), Channel.fromPath("${params.rhea_directions}"), Channel.fromPath("${params.rhea_reactions_smiles}") ) + produce_neo4j_files = PRODUCE_NEO4J_FILES( score_ligands.all_parity_calcs, Channel.fromPath(params.cognate_ligands) , all_contacts.bound_entities, all_contacts.cath, all_contacts.scop, all_contacts.pfam, all_contacts.superfamily, all_contacts.g3dsa, all_contacts.scop2b_sf, all_contacts.scop2b_fa, Channel.fromPath("${params.enzyme_dat_file}"), Channel.fromPath("${params.enzyme_class_file}"), Channel.from(params.parity_threshold), Channel.fromPath("${params.rhea2ec}"), Channel.fromPath("${params.rhea_directions}"), Channel.fromPath("${params.rhea_reactions_smiles}"), Channel.fromPath("${params.scop2_domains_info_file}"), Channel.fromPath("${params.scop2_descriptions_file}") ) } \ No newline at end of file diff --git a/nextflow/nextflow.config b/nextflow/nextflow.config index 10d3001..a154192 100644 --- a/nextflow/nextflow.config +++ b/nextflow/nextflow.config @@ -89,6 +89,7 @@ params { scop_domains_info_file = "${params.data_dir}/dir.cla.scop.1_75.txt" scop_descriptions_file = "${params.data_dir}/dir.des.scop.1_75.txt" scop2_domains_info_file = "${params.data_dir}/scop2-cla-latest.txt" + scop2_descriptions_file = "${params.data_dir}/scop2-des-latest.txt" interpro_xml = "${params.data_dir}/interpro.xml.gz" pfam_clans = "${params.data_dir}/clan.txt.gz" pfam_clan_rels = "${params.data_dir}/clan_membership.txt.gz"