Get ref genome from meta_study instead of command line #7969

SRodenburg · 2020-10-15T15:22:53Z

The reference genome (and thereby species and build) is now retrieved from the meta_study.txt file after the optional reference_genome tag.

Allowed values are hg19, hg38 and mm10. The validator will raise an error when any other value is encountered, and will default to hg19.

I also added a more specific error when validating the NCBI_Build column in MAF files, validating whether the correct reference genome is used.

This was already present in the FIle Formats documentation.

SRodenburg · 2020-10-15T15:23:55Z

core/src/main/scripts/importer/cbioportal_common.py

@@ -37,7 +37,7 @@
 ADD_CASE_LIST_CLASS = "org.mskcc.cbio.portal.scripts.AddCaseList"
 VERSION_UTIL_CLASS = "org.mskcc.cbio.portal.util.VersionUtil"

-# provides a key for data types to metafile specification dict.  
+# provides a key for data types to metafile specification dict.


somehow my editor automatically removes trailing whitespace

SRodenburg

some sidenotes

SRodenburg · 2020-10-15T15:26:19Z

core/src/main/scripts/importer/cbioportal_common.py

@@ -685,7 +686,6 @@ def validate_types_and_id(meta_dictionary, logger, filename):
 def parse_metadata_file(filename,
                        logger,
                        study_id=None,
-                        genome_name=None,


somehow this parameter is passed on but never used.

SRodenburg · 2020-10-15T15:26:52Z

core/src/main/scripts/importer/metaImport.py

-                        required=False)   
-    parser.add_argument('-ncbi', '--ncbi_build_number', type=str, default='37',
-                         help='NCBI reference genome build number (default: assumed 37 for UCSC reference genome build hg19)',
-                         required=False)                                                                  


arguments became obsolete now

SRodenburg · 2020-10-15T15:27:48Z

core/src/main/scripts/importer/validateData.py

+        'hg38':('human','38','hg38'),
+        'mm10':('mouse','38','mm10')
+    }
+


not sure if we should also allow GRCxXX values

I think we should follow the values defined in portal.properties, so it's good like this. If you want to add the GRCxXX values we should change it in the properties. And just a thought: if we want to change the genome build properties in portal.properties I think we should simplify it to one property: hg19/hg38/mm10 already gives you all the information given from the three properties that we currently have.

SRodenburg · 2020-10-15T15:28:17Z

core/src/main/scripts/importer/validateData.py

@@ -4864,8 +4892,6 @@ def validate_data_relations(validators_by_meta_type, logger):
 def request_from_portal_api(server_url, api_name, logger):
    """Send a request to the portal API and return the decoded JSON object."""

-    print(api_name)
-


relic from previous work

inodb

Thanks! LGTM! Could you also update the docs https://docs.cbioportal.org/5.1-data-loading/data-loading/using-the-dataset-validator#validation-of-non-human-data ?

inodb · 2020-10-15T19:43:47Z

And maybe double check if https://github.com/cBioPortal/cbioportal-docker-compose still works for grch38. Thanks!

SRodenburg · 2020-10-16T08:29:18Z

And maybe double check if https://github.com/cBioPortal/cbioportal-docker-compose still works for grch38. Thanks!

Just tested, it still works. I did not change anything there.

oplantalech · 2020-10-16T07:45:52Z

core/src/main/scripts/importer/validateData.py

+                    portal_instance.species, portal_instance.ncbi_build, portal_instance.genome_build = \
+                        reference_genome_map[meta_dictionary['reference_genome']]
+            else:
+                logger.info('No reference genome specified -- using default (hg19)',


I think it is wrong to assume that the default is hg19. The default is the genome specified in portal.properties, right?

Another option would be to set the default to hg19 instead of being defined in portal.properties. But it should be checked if the genome build properties in portal.properties are only used here before removing them.

The portal.properties are not used in the validator. AFAIK, those are only used in the Java code, and some parameters are passed on to the frontend.

@SRodenburg I see. I was confused by the portal_instance name.

Defaults for these properties are, indeed, set here: https://github.com/cBioPortal/cbioportal/pull/7969/files/e805ff2f295eae71032d52dfcd754849a07a401c#diff-9be14d6564536099d9c431cf330dc2ad62abfecbbc191ac7c5814855dc37685eL305

So I think this three values can be simplified into one. As I mentioned, hg19/hg38/mm10 already gives you all the information given from the three properties that we currently have, and there is no need to deviate from the reference_genome property

oplantalech · 2020-10-16T07:50:59Z

core/src/main/scripts/importer/validateData.py

+        'hg38':('human','38','hg38'),
+        'mm10':('mouse','38','mm10')
+    }
+


I think we should follow the values defined in portal.properties, so it's good like this. If you want to add the GRCxXX values we should change it in the properties. And just a thought: if we want to change the genome build properties in portal.properties I think we should simplify it to one property: hg19/hg38/mm10 already gives you all the information given from the three properties that we currently have.

SRodenburg · 2020-10-16T09:44:14Z

@inodb docs are updated ✔️

inodb · 2020-10-16T12:30:18Z

docs/Support-multiple-reference-genomes.md

-
-```
-
+* You can also overwrite default genome values by using the following options of the importer script. For that, you have to specify the `reference_genome` field in the `meta_study.txt` file of your study.


not entirely clear here what the values are supposed to be but will update that

* In accordance with cBioPortal/cbioportal#7969 which is merged since version 3.4.17

impl. ref genome parsing from meta_study

e805ff2

SRodenburg requested review from pvannierop, oplantalech and inodb and removed request for pvannierop October 15, 2020 15:22

SRodenburg commented Oct 15, 2020

View reviewed changes

SRodenburg added the validator label Oct 15, 2020

inodb approved these changes Oct 15, 2020

View reviewed changes

inodb added the enhancement label Oct 15, 2020

oplantalech reviewed Oct 16, 2020

View reviewed changes

update docs

860a0a2

oplantalech approved these changes Oct 16, 2020

View reviewed changes

inodb changed the title ~~impl. ref genome parsing from meta_study~~ Get ref genome from meta_study instead of command line Oct 16, 2020

inodb merged commit 0f23ed4 into cBioPortal:master Oct 16, 2020

inodb reviewed Oct 16, 2020

View reviewed changes

oplantalech deleted the metafile_ref_genome_parsing branch November 19, 2020 09:50

oplantalech added a commit to thehyve/cbioportal-staging that referenced this pull request Feb 26, 2021

Remove genomic info from validation command

e28069a

* In accordance with cBioPortal/cbioportal#7969 which is merged since version 3.4.17

oplantalech mentioned this pull request Feb 26, 2021

Remove genomic info from validation command thehyve/cbioportal-staging#47

Merged

oplantalech added a commit to thehyve/cbioportal-staging that referenced this pull request Feb 26, 2021

Remove genomic info from validation command

66e3064

* In accordance with cBioPortal/cbioportal#7969 which is merged since version 3.4.17

oplantalech added a commit to thehyve/cbioportal-staging that referenced this pull request Mar 1, 2021

Remove genomic info from validation command

18637ff

* In accordance with cBioPortal/cbioportal#7969 which is merged since version 3.4.17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get ref genome from meta_study instead of command line #7969

Get ref genome from meta_study instead of command line #7969

SRodenburg commented Oct 15, 2020

SRodenburg Oct 15, 2020

SRodenburg left a comment

SRodenburg Oct 15, 2020

SRodenburg Oct 15, 2020

SRodenburg Oct 15, 2020

oplantalech Oct 16, 2020

SRodenburg Oct 15, 2020

inodb left a comment

inodb commented Oct 15, 2020

SRodenburg commented Oct 16, 2020

oplantalech Oct 16, 2020

oplantalech Oct 16, 2020

SRodenburg Oct 16, 2020

oplantalech Oct 16, 2020

oplantalech Oct 16, 2020

SRodenburg commented Oct 16, 2020

inodb Oct 16, 2020


		```

		* You can also overwrite default genome values by using the following options of the importer script. For that, you have to specify the `reference_genome` field in the `meta_study.txt` file of your study.

Get ref genome from meta_study instead of command line #7969

Get ref genome from meta_study instead of command line #7969

Conversation

SRodenburg commented Oct 15, 2020

Choose a reason for hiding this comment

SRodenburg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inodb left a comment

Choose a reason for hiding this comment

inodb commented Oct 15, 2020

SRodenburg commented Oct 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SRodenburg commented Oct 16, 2020

Choose a reason for hiding this comment