Annotations schema updates #1281

jameshadfield · 2023-08-16T09:50:31Z

This PR is ready for review, but note that I won't merge it until the companion Auspice PR has been merged and that there may be some very slight schema changes (e.g. property names changing) before then. Neither of those should hold up review.

See commit messages for details.

The annotations schema is updated to more accurately represent how this data is used by Auspice version 2.46.1. Specifically, the strand of the 'nuc' property is not used by auspice, and start/end coordinates are required properties. Tests are updated because the tests were incorrect. Currently the annotations section of node-data JSONs are produced by `augur ancestral` and `augur translate`. Each adds a 'nuc' property, and it's required for Auspice, so I think it's appropriate to make it a required property for the schema. The changes to the schema here are only reflected when parsing node-data files, the next commit will make this available for our typical auspice JSON validation.

codecov · 2023-08-16T10:31:01Z

Codecov Report

Patch coverage: 91.66% and project coverage change: +0.12% 🎉

Comparison is base (17a2746) 69.74% compared to head (c530539) 69.86%.
Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1281      +/-   ##
==========================================
+ Coverage   69.74%   69.86%   +0.12%     
==========================================
  Files          67       67              
  Lines        7146     7155       +9     
  Branches     1742     1744       +2     
==========================================
+ Hits         4984     4999      +15     
+ Misses       1855     1849       -6     
  Partials      307      307

Files Changed	Coverage Δ
augur/validate.py	`79.32% <91.66%> (+4.62%)`	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

victorlin · 2023-08-16T16:23:15Z

augur/data/schema-annotations.json

-                "strand": {
-                    "description": "Positive or negative strand",
-                    "type": "string",
-                    "enum": ["-","+"]


Suggestion: Add enum back with additional values.

Even though Auspice only cares if the value is '-' or not, Augur can also export values '?' and None as suggested and implemented in 31f0b26. Defining the possible values here will improve the effectiveness of validation.

I started this in #1279 but that PR can be closed and the change included in here.

Thanks for the review -- really appreciated!

We're going to have to think through this. All annotations are interpreted by Auspice as CDSs, so a strand of "?" or "." (which we represent as None) doesn't make sense as a CDS - neither for Auspice to display nor for Augur to translate.

I can allow them in the schema and then have Auspice filter to ["+", "-"], which is probably the most technically correct, but I would think that happily translating "?" / "." features is questionable. For augur translate + GenBank annotations, only CDS features are read and (I believe) it's not possible to encode a CDS in GenBank that's not +/-. I'm not sure what we do for augur translate + GFF annotation.

Update: Auspice PR now ignores any non-nuc annotation which is not explicitly +/- strand

Have shifted this conversation to #1279

augur/data/schema-export-v2.json

augur/validate.py

augur/data/schema-annotations.json

Matches the new schema id (to be) in Augur, see <nextstrain/augur#1281>. Resolves <#704>.

tsibley

Just a couple additional small changes.

I've done the .org side of things in nextstrain/nextstrain.org#705.

augur/validate.py

tsibley · 2023-08-18T20:51:13Z

augur/validate.py

+    if refs:
+        # Make the validator aware of additional schemas
+        schema_store = {k: json.loads(resource_string(__package__, os.path.join("data", v))) for k,v in refs.items()}
+        resolver = jsonschema.RefResolver.from_schema(schema,store=schema_store)
+        schema_validator = Validator(schema, resolver=resolver)
+    else:
+        schema_validator = Validator(schema)
+
+    # By default $ref URLs which we don't define in a schema_store are fetched
+    # by jsonschema. This often indicates a typo (the $ref doesn't match the key
+    # of the schema_store) or we forgot to add a local mapping for a new $ref.
+    # Either way, Augur should not be accessing the network. 
+    def resolve_remote(url):
+        # The exception type is not important as jsonschema will catch & re-raise as a RefResolutionError
+        raise Exception(f"The schema used for validation attempted to fetch the remote URL '{url}'. " +
+                        "Augur should resolve schema references to local files, please check the schema used " +
+                        "and update the appropriate schema_store as needed." )
+    schema_validator.resolver.resolve_remote = resolve_remote
+
+    return schema_validator


non-blocking

If we upgraded on dep on jsonschema to the 4.x series, we could use the newer and nicer APIs for reference handling here. Doesn't have to be now, though.

augur/data/schema-export-v2.json

See added comments for details

See <nextstrain/auspice#1684> for the context for these additions.

jameshadfield · 2023-08-20T21:42:07Z

Schema updated to remove the name property on individual CDS segments as that functionality was removed from the Auspice PR.

repr() will add appropriate quotes around the value. Related-to: <#1281 (comment)>

corneliusroemer · 2023-12-08T18:32:51Z

augur/data/schema-annotations.json

-                    "description": "Type of the feature. could be mRNA, CDS, or similar",
-                    "$comment": "Unused by Auspice 2.0",
-                    "type": "string"
+                "strand": {


Late to the party, but shouldn't we in principle allow each cds fragment to have its own strandedness? Here the strand is fixed for all fragments, which will be fine in most cases but who knows, maybe sometimes fragments might come from opposite strands?

ChatGPT tells me that strandedness never changes within a CDS so then what we have here should work in all cases.

With trans-splicing it might work, but doesn't seem to happen in viruses so we should be good for now: https://en.wikipedia.org/wiki/Trans-splicing

Given a SeqFeqture with a CompoundLocation we now correctly write out the CDS/gene using segmented coordinates. Auspice can now handle such coordinates (see <nextstrain/auspice#1684> and <#1281> for the corresponding schema updates). Note that the translations (via augur translate) of complex CDSs are not modified in this commit. Supersedes #1333

Given a SeqFeqture with a CompoundLocation we now correctly write out the CDS/gene using segmented coordinates. Auspice can now handle such coordinates (see <nextstrain/auspice#1684> and <#1281> for the corresponding schema updates). Note that the translations (via augur translate) of complex CDSs did not need modifying as they already used BioPython's SeqFeature.extract method. Supersedes #1333

jameshadfield requested a review from a team August 16, 2023 09:50

jameshadfield force-pushed the annotations-schema-updates branch from 34692f4 to d763c05 Compare August 16, 2023 10:14

jameshadfield mentioned this pull request Aug 16, 2023

Genome annotations for segmented CDSs etc #1280

Open

3 tasks

victorlin reviewed Aug 16, 2023

View reviewed changes

jameshadfield force-pushed the annotations-schema-updates branch from d763c05 to 35c688c Compare August 17, 2023 00:09

tsibley requested changes Aug 17, 2023

View reviewed changes

augur/data/schema-export-v2.json Outdated Show resolved Hide resolved

augur/validate.py Outdated Show resolved Hide resolved

augur/data/schema-annotations.json Outdated Show resolved Hide resolved

jameshadfield mentioned this pull request Aug 17, 2023

Add annotations schema URL nextstrain/nextstrain.org#704

Closed

jameshadfield force-pushed the annotations-schema-updates branch from 35c688c to 7aea997 Compare August 17, 2023 23:27

tsibley added a commit to nextstrain/nextstrain.org that referenced this pull request Aug 18, 2023

Add canonical URL for Augur's annotations schema

eaf140f

Matches the new schema id (to be) in Augur, see <nextstrain/augur#1281>. Resolves <#704>.

tsibley mentioned this pull request Aug 18, 2023

Add canonical URL for Augur's annotations schema nextstrain/nextstrain.org#705

Merged

2 tasks

tsibley requested changes Aug 18, 2023

View reviewed changes

[validate] allow local $refs

37f63b4

See added comments for details

jameshadfield force-pushed the annotations-schema-updates branch from 7aea997 to 56fb8a8 Compare August 20, 2023 20:59

Extend annotations schema

c530539

See <nextstrain/auspice#1684> for the context for these additions.

jameshadfield force-pushed the annotations-schema-updates branch from 56fb8a8 to c530539 Compare August 20, 2023 21:40

jameshadfield mentioned this pull request Aug 20, 2023

Update annotation schema for strand changes #1279

Draft

2 tasks

jameshadfield merged commit 37a0a07 into master Aug 20, 2023
27 checks passed

jameshadfield deleted the annotations-schema-updates branch August 20, 2023 22:11

tsibley added a commit that referenced this pull request Aug 21, 2023

validate: Remove duplicative single quotes around !r interpolation

14d2781

repr() will add appropriate quotes around the value. Related-to: <#1281 (comment)>

corneliusroemer reviewed Dec 8, 2023

View reviewed changes

tsibley mentioned this pull request Jan 10, 2024

Augur errors when jsonschema version 4 is installed #1358

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Annotations schema updates #1281

Annotations schema updates #1281

jameshadfield commented Aug 16, 2023

codecov bot commented Aug 16, 2023 •

edited

Loading

victorlin Aug 16, 2023

jameshadfield Aug 17, 2023 •

edited

Loading

jameshadfield Aug 20, 2023

tsibley left a comment

tsibley Aug 18, 2023

jameshadfield commented Aug 20, 2023

corneliusroemer Dec 8, 2023

corneliusroemer Dec 8, 2023

corneliusroemer Dec 8, 2023

Annotations schema updates #1281

Annotations schema updates #1281

Conversation

jameshadfield commented Aug 16, 2023

codecov bot commented Aug 16, 2023 • edited Loading

Codecov Report

victorlin Aug 16, 2023

Choose a reason for hiding this comment

jameshadfield Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

jameshadfield Aug 20, 2023

Choose a reason for hiding this comment

tsibley left a comment

Choose a reason for hiding this comment

tsibley Aug 18, 2023

Choose a reason for hiding this comment

jameshadfield commented Aug 20, 2023

corneliusroemer Dec 8, 2023

Choose a reason for hiding this comment

corneliusroemer Dec 8, 2023

Choose a reason for hiding this comment

corneliusroemer Dec 8, 2023

Choose a reason for hiding this comment

codecov bot commented Aug 16, 2023 •

edited

Loading

jameshadfield Aug 17, 2023 •

edited

Loading