Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to STAC spec 1.0.0 #86

Merged
merged 14 commits into from
Jun 25, 2021

Conversation

moradology
Copy link
Collaborator

@moradology moradology commented Jun 9, 2021

This PR is aimed at updating models to correspond to updates in the STAC spec since 1.0.0-beta. Changes introduced here are based on the relevant updates to STAC-Spec listed here in rc1, rc2, rc3, rc4, and 1.0.0: https://github.com/radiantearth/stac-spec/blob/v1.0.0/CHANGELOG.md

@moradology moradology changed the title Update to STAC spec 1.0.0 [WIP] Update to STAC spec 1.0.0 Jun 14, 2021
@moradology moradology marked this pull request as ready for review June 14, 2021 02:52
@moradology
Copy link
Collaborator Author

Here's a list of the changes to the spec with those that triggered updates in pydantic modeling called out at the end of relevant lines ("noop" when no change applied to here, "changed" when a change has been applied, and a longer note in more complicated or questionable cases):

v1.0.0 - 2021-05-25
Added

  • Updated best practices to add a recommendation to include title in links where possible. (#1133, noop)
    Changed
  • Updated version numbers for 1.0.0 release. (noop)
  • Final UML tweaks for latest changes. (#1134, noop)
  • Removed ItemCollection from the STAC detection heuristic in Best Practices. It can't easily be differentiated from GeoJSON FeatureCollections any longer. (API#141, noop)
    v1.0.0-rc.4 - 2021-05-11
    Changed
  • An empty Catalog is now allowed, removed the requirement that it must have a child or item link. (#1115, noop)
  • An open date range to both sides is now allowed in the Collection's temporal extents. (#1125, noop - already supported by existing model)
  • Catalog and Collection JSON Schemas don't have a common JSON Schema any more. (#1122, noop)
    Removed
  • Catalogs don't support summaries any more. (#1122, changed)
    Fixed
  • Added clarification around when an extension should be included in stac_extensions. (#1123, noop)
  • JSON Schemas don't allow "shortcuts" for core extensions any longer. (#1121, noop)
  • Various examples fixes. (noop)
    v1.0.0-rc.3 - 2021-04-29
    Added
  • Summaries are allowed to specify JSON Schema in addition to ranges and sets of values. (#1045, changed)
  • Added preview relation type for interoperable thumbnails to best practices. (#1090, changed)
  • Recommendation to include both root and parent relation types when they point at the same file. (#1098, noop)
  • Overview section linking to various foundational standards. (#1111, noop)
    Changed
  • The first extent in a Collection is always the overall extent, followed by more specific extents. (#1064, opengeospatial/ogcapi-features; #520, noop - enforcement is likely possible but not worth it)
  • Updated examples for automatic collection creation from code and validation (#1080, noop)
  • Clarified that stac_extensions should also list extensions that are used in Collection summaries. (#1077, noop)
  • The Stats Object for Summaries has been renamed to Range Object (no functional change). (#1093, changed)
    changed, created (common metadata) and temporal extents (collections): Timestamps must be always in UTC (#1095, changed)
  • Clarified that collection summaries do not require that all property fields are summarized. (#1106, noop)
  • Clarified that gsd should only be used on an asset to represent the sensor, not just different processing. (#1105, changed)
  • Clarified that leaving a field out is not equivalent to setting it as null. (#1111, noop: a new method was added but the subclassing of Feature and FeatureCollection make this messy; as it stands, users appear to be responsible for ensuring the appropriate flags are set during serialization to dict or json)
    v1.0.0-rc.2 - 2021-03-30
    Changed
  • Required properties of type string require a minimum length of 1. (#1065)
  • gsd must be greater than 0. (#1068)
    Removed
  • The remaining extensions in the spec (EO, Projection, Scientific Citation and View) have been moved out of the core specification, into their own repos in the stac-extensions GitHub organization. They must now be referred to by their schemas directly - the name shortcuts for them are no longer valid. The concept of the 'named shortcut' also goes away entirely. (#1070, noop)
    Fixed
    Examples
  • Collection Assets were specified as required (only in written text, not in JSON Schema), but that was incorrectly copied over from the former collection-assets extension. Collection Assets are not required. (changed)
  • Clarified that the values in summaries (both for ranges and sets of values) must follow the data type of the property they summarize. (#1069, noop: unclear how to enforce this relationship)
    v1.0.0-rc.1 - 2021-03-03
    Added
  • Catalog and Collection now require a type parameter, to be set to Catalog or Collection for clients to more easily distinguish them easily. (#971, changed)
  • Collection specification adds Assets (previously needed Collections Asset extension to do that). (#1008)
    'via' and 'canonical' rel types are now options in Items. (#884, changed)
  • In Extensions list there is now reference to three new (non-core) extensions: processing, file info and card4l. These would have been added to the stac-spec repo, except all non-core extensions were moved to stac-extensions org. (#1024, noop)
  • 'summaries' are now available in the Catalog spec, so both catalogs and collections can make use of it. (#903, noop: this change is rolled back in later updates to the spec)
  • There is a new recommendation to enable CORS. (#940, noop)
  • A Best Practice section on 'requester pays' cloud buckets was added. (#1021, noop)
  • A new Best Practice section explains Asset Roles, plus some lists of potential roles for people to use (in best practices, sar and eo). (#989, noop)
  • There is a new Best Practice recommendation to keep collections at consistent levels. (#1009, noop)
    Changed
  • The Stats Object for Collection summaries changed min to minimum and max to maximum to align with JSON Schema. (#967, changed)
  • URIs (usually found int properties like href, url) are now validated using the iri-reference format in JSON Schema (allows international characters in URIs) (#953, noop)
  • Enhanced the way the spec talks about ID's to encourage more global uniqueness. (#883, noop)
  • Clarified how collection-level asset object properties do not remove the need for item-level asset object properties in the item-assets extension (#880, noop)
  • Made summaries to be strongly recommended - everyone should strive to implement them, as they are very useful. (#985, noop)
  • Moved examples from individual directories into a single /examples folder at the root, and evolved them to be more representative. (#955, noop)
  • Renamed "Scientific Extension" to "Scientific Citation Extension" (#990, noop)
  • Relaxed the regular expression for DOIs in the scientific extension (#910, noop)
  • proj:geometry allows all GeoJSON geometries instead of just a polygon. (#995, changed)
    Removed
  • Checksum extension (field checksum:multihash). Use File Info extension (field file:checksum) instead for assets. There's no replacement for links. (#934, noop)
  • Collection Assets extension, as the core construct of Assets in a Collection is now part of the core Collection spec. No change is required except removing collection-assets from the list of stac_extensions. (#1008, changed)
  • Numerous extensions (Data Cube, Item Assets, Point Cloud, SAR, Single File STAC, Tiled Assets, Timestamps & Versioning) have been moved out of the core specification, into their own repos in the stac-extensions GitHub organization. They must now be referred to by their schemas directly - the name shortcuts for them are no longer valid. (#1024, this appears to be unnecessary to mirror in stac-pydantic)
    Fixed
  • Fixed JSON Schema for providers (Collections and Items) to be an object and require a name. (#924, already true in pydantic)

@moradology
Copy link
Collaborator Author

moradology commented Jun 14, 2021

This is ready to be reviewed/inspected for sense but work continues to get the tests functioning correctly. Currently the majority of the tests are failing with 404s as they request 1.0.0 compliant example JSON

The lack of 1.0.0 examples makes this all a bit problematic and suggestions are very welcome.

@@ -1 +1 @@
STAC_VERSION = "1.0.0-beta.2"
STAC_VERSION = "1.0.0"
Copy link
Collaborator Author

@moradology moradology Jun 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change, if omitted, allows tests to complete successfully as the prior files all appear to be 1.0.0 compliant

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm rolling some example files that simply extend the previously working versions to use some 1.0.0 features. More canonical examples can be swapped in later if that's desirable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone ahead and folded the testing files within this repository

{
"stac_version": "1.0.0",
"stac_extensions": [
"scientific",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extension ids need to be fully qualified in 1.0.0 I think?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're absolutely right

Copy link
Collaborator Author

@moradology moradology Jun 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to fix this in the test files but I wonder if more validation isn't desirable in the model itself to avoid this kind of oversight

Copy link
Contributor

@kylebarron kylebarron Jun 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

@geospatial-jeff geospatial-jeff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Some minor comments.

stac_pydantic/api/utils/link_factory.py Outdated Show resolved Hide resolved
stac_pydantic/catalog.py Outdated Show resolved Hide resolved
stac_pydantic/collection.py Outdated Show resolved Hide resolved
stac_pydantic/extensions.py Show resolved Hide resolved
stac_pydantic/shared.py Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented Jun 25, 2021

Codecov Report

Merging #86 (cf9f613) into master (0e48d2f) will decrease coverage by 1.95%.
The diff coverage is 94.52%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #86      +/-   ##
==========================================
- Coverage   98.33%   96.38%   -1.96%     
==========================================
  Files          33       19      -14     
  Lines         779      442     -337     
==========================================
- Hits          766      426     -340     
- Misses         13       16       +3     
Flag Coverage Δ
unittests 96.38% <94.52%> (-1.96%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
stac_pydantic/utils.py 100.00% <ø> (ø)
stac_pydantic/extensions.py 86.20% <86.20%> (ø)
stac_pydantic/__init__.py 100.00% <100.00%> (ø)
stac_pydantic/api/extensions/sort.py 100.00% <100.00%> (ø)
stac_pydantic/api/landing.py 100.00% <100.00%> (ø)
stac_pydantic/api/utils/link_factory.py 100.00% <100.00%> (ø)
stac_pydantic/catalog.py 100.00% <100.00%> (ø)
stac_pydantic/collection.py 100.00% <100.00%> (ø)
stac_pydantic/item.py 93.33% <100.00%> (-1.19%) ⬇️
stac_pydantic/links.py 100.00% <100.00%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e48d2f...cf9f613. Read the comment docs.

@geospatial-jeff geospatial-jeff self-requested a review June 25, 2021 02:49
Copy link
Collaborator

@geospatial-jeff geospatial-jeff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moradology This looks great!

@geospatial-jeff geospatial-jeff merged commit d222206 into stac-utils:master Jun 25, 2021
@moradology moradology mentioned this pull request Jun 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants