DM-41688: Add Pydantic data model to felis #25
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #25 +/- ##
==========================================
+ Coverage 98.12% 98.47% +0.35%
==========================================
Files 6 7 +1
Lines 427 592 +165
==========================================
+ Hits 419 583 +164
- Misses 8 9 +1 ☔ View full report in Codecov by Sentry. |
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
from
e31a3fd
to
b48c686
Compare
|
Sample output of new validation using Pydantic on |
|
deescription? (at start of validate.log) |
Someone misspelled "description" in the SDM schema itself, and the strict validation has caught the error. Before this would not have been caught because extra attributes were allowed in the YAML files and they are disallowed now. |
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
3 times, most recently
from
aa5f06a
to
6b2f62d
Compare
timj
approved these changes
Jan 2, 2024
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
3 times, most recently
from
e7bb6b1
to
6c62c6e
Compare
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
from
6c62c6e
to
73b200b
Compare
gpdf
requested changes
Jan 10, 2024
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
4 times, most recently
from
49d9916
to
56f5cea
Compare
This update adds a Pydantic v2 data model in the datamodel module along with unit tests for the new functionality. A new CLI command "validate" supports loading a YAML file in the top-level Schema class and validating it. Validation errors will result in a non-zero return code. Any number of YAML files may be provided at the command line and wildcards may be used to test all the files in a directory. The data model explicitly defines fields for all allowed keys, such that the usage of undefined keys, including misspellings or unknown annotations, will result in validation errors. All optional annotations currently found in the SDM Schemas YAML files are supported by the data model. Units and IVOA UCDs are checked using modules from astropy. Validation of the contents of other fields is initially loose, some accepting any string value. It is forseen that additional validation rules for fields and data model classes will be added in the future.
JeremyMcCormick
force-pushed
the
tickets/DM-41688
branch
from
56f5cea
to
28f202c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a Pydantic v2 data model to felis describing the YAML schema format, as well as unit tests for the new module. A
validatecommand was added to the CLI which will load any number of YAML files into Pydantic and print validation errors. This has been tested on all of the schemas insdm_schemas/yml.To run tests covering the new module:
To run validation on all of the SDM schemas using the CLI tool:
These changes make no attempt to replace usage of the existing "simple" model that is used throughout Felis. All existing functionality should be unaffected.
For more details, please see the message on this PR's single commit. Updates to the Felis documentation will be forthcoming.