MDIO Release 1.0.0
#365
Draft
MDIO Release 1.0.0
#365
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added base schemas for MDIO v0 and v1 variables, datasets, units etc. Significantly improved documentation (still some work to do) Switched from a huge stack of flake8+black+isort+plugins to ruff With new ruff rules Everything is type hinted now (still need to run mypy and fix errors though) Fixed a lot of ugly code Found some bugs and fixed them Updated almost all dependencies to latest versions Things should be a little faster because removed a lot of dict/list etc calls to literals Moved a lot of type hint imports to type-checking block, this should also speed up imports and help avoid circular dependencies Improved some testing logic. NOTE: Ignoring coverage right now because a lot of code was added. Before v1 release, will have a separate branch to add more tests.
The code change made the metadata field in the Dataset schema necessary, it is no longer optional. This ensures that all instances of the Dataset schema will include metadata.
The commit simplifies the StructuredField and StructuredType classes by removing unnecessary attributes. In the StructuredField class, 'offset' has been removed, and the 'name' attribute now requires a value. Similarly, in the StructuredType class, the 'item_size' attribute has been eliminated. 'From __future__ import annotations' was also added at the beginning of the file to support post-poned evaluation of type hints.
The commit reflects the updates made to different packages in the pyproject.toml file. It includes upgrade of versions for several dependencies such as zarr, dask, tqdm, numba, fsspec, pydantic and others. Additionally, a new dependency, pydantic-settings, has been added. Various optional and dev dependencies were also updated.
Additional fields have been added to the VersionedMetadataConvention class in the metadata schema. These include the version of the metadata convention, a link to its documentation (homepage), and URL of the JSON schema. Also, the 'units_v1' field is renamed to 'units' in the AllUnits class within the units schema.
The commit includes 'rechunking' in the list of tutorials listed in the index. This is to ensure that readers can easily navigate to the 'rechunking' tutorial right from the index page.
This reverts commit c3fadb1.
The Python 3.10-style type hinting syntax has been applied across several modules in the MDIO schemas. The use of `from __future__ import annotations` enables forward references, and simplifies optional type hints by using `float | None` instead of `Optional[float]`, improving readability and consistency. This also involved conditional importing within TYPE_CHECKING block for avoiding circular dependencies.
The version of the numba package was updated from 0.59.0rc1 to 0.59.1 in the pyproject.toml file. This change ensures we are using the latest stable version of the numba package for better performance and stability.
Enums in compressors.py were changed from Enum to IntEnum and StrEnum. Instance methods were also added to the ZFP and BloscShuffle schemas to convert parameters to compressor arguments. Additionally, the order of the ZFPMode enum was slightly altered and an integer code property was added.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.