Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Default Attributes, Overwrite Policy, and Attribute Validation #100

Merged
merged 6 commits into from
Oct 26, 2023

Conversation

Alrobbertz
Copy link
Member

@Alrobbertz Alrobbertz commented Oct 18, 2023

  • Updates the default FORMAT variable metadata attribute
  • Updates the schema policy so FORMAT, VALIDMIN/MAX, FILLVAL are not overwritten.
  • Updates variable attribute validation to skip FORMAT, and FILLVAL which were based on the data type of the attribute
  • Updates validation for VALIDMIN/MAX from spacepy which was based on the data type of the attribute. Modified version still checks range and dimensionality, but not that the data types are en exact match.

closes #98
closes #99

@codecov
Copy link

codecov bot commented Oct 18, 2023

Codecov Report

Merging #100 (4e1ac85) into main (ba1fe82) will decrease coverage by 0.02%.
The diff coverage is 98.48%.

@@            Coverage Diff             @@
##             main     #100      +/-   ##
==========================================
- Coverage   94.49%   94.47%   -0.02%     
==========================================
  Files          10       10              
  Lines        1707     1756      +49     
==========================================
+ Hits         1613     1659      +46     
- Misses         94       97       +3     
Files Coverage Δ
hermes_core/util/schema.py 95.21% <100.00%> (-0.04%) ⬇️
hermes_core/util/validation.py 96.07% <98.43%> (+1.07%) ⬆️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@ehsteve ehsteve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few small comments added.



def test_min_max_Epoch16():
"""Get min/max values for Epoch16 types"""
minval, maxval = HermesDataSchema()._get_minmax(const.CDF_EPOCH16)
# Make sure the minimum isn't just plain invalid
assert minval == datetime.datetime(1, 1, 1)
assert maxval == datetime.datetime(9999, 12, 31, 23, 59, 59)
assert minval == datetime.datetime(1900, 1, 1, 0, 0, 0, 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this 01-Jan-0000 00:00:00.000.000.000.000?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same is true for epoch 8 bit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table on Page 57 looks like its referring to the FILLVAL attribute rather than the VALIDMIN/MAX attributes. From a brief skim through the PDF it does not look like there are recommendations for VALIDMIN/MAX default values.

I updated these here because we were getting overflow errors when loading and saving CDF files so I updated them to more reasonable MIN and MAX values.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry i misunderstood the spec.

@@ -306,24 +306,24 @@ def test_min_max_TT2000():
"""Get min/max values for TT2000 types"""
minval, maxval = HermesDataSchema()._get_minmax(const.CDF_TIME_TT2000)
# Make sure the minimum isn't just plain invalid
assert minval == datetime.datetime(1, 1, 1)
assert maxval == datetime.datetime(2292, 4, 11, 11, 46, 7, 670776)
assert minval == datetime.datetime(1900, 1, 1, 0, 0, 0, 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we be using astropy time objects here? since we are supposed to be using them everywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacepy loads these attributes as datetime objects when loading a CDF file. So if we want to make everything astropy time everywhere, we would need to create wrappers around the spacepy attribute loading as well as update these derivation functions. If we want to go that route that might warrant a separate issue and PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrrr. spacepy does support astropy time but does not use it "natively".

hermes_core/util/tests/test_schema.py Show resolved Hide resolved
@Alrobbertz Alrobbertz merged commit 65ec6b1 into HERMES-SOC:main Oct 26, 2023
14 checks passed
@Alrobbertz Alrobbertz deleted the update_format_and_schema_attrs branch October 26, 2023 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants