Second set of data flow and data service pre-work #144
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py).
Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker.
Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type.
Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job.
Extending DataFormat metadata type to initialize with indices and (if needed) implied indices (i.e., not also data fields).
Adding new ContinuousRestriction, DiscreteRestriction, and DataDomain types for defining and filtering data domains.
Updating DataDomain type to also contain a DataFormat property, to let it encapsulate the data fields in the domain effectively.
Updating DataRequirement type to use the DataDomain to define its domain instead of a generic type variable; with this done, making it a concrete type and removing deprecated CatchmentDataRequirement subtype.
Renaming to test_data_requirement.py after making DataRequirement type concrete and removing CatchmentDataRequirement type.
Updating tests after recent redesign that involved removing the CatchmentDataRequirement subtype.
Removing URI in favor of simple string access location property, and adding UUID properties to Dataset and DatasetManager.
Adding new DatasetUser type and functions within DatasetManager for managing known users of a Dataset.
christophertubbs
approved these changes
Feb 24, 2022
|
Relates to #128. |
6 tasks
donaldwj
pushed a commit
to donaldwj/DMOD
that referenced
this pull request
Jun 21, 2022
* Fix bug related to job types and ModelExecRequest. Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py). * Add uri as dependency for scheduler package. * New WorkerDataRequirement helper type for Jobs. Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker. * Add new JobExecStep values and update docstring. * Redesign JobStatus type. Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type. * Update tests to account for redesigned JobStatus. * Adopt DataRequirement type from modeldata. Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job. * Update scheduler package to depend on modeldata. * Update DataFormat with is_time_series and fields. * Add TimeRange importable from dmod.modeldata.data. * Add abstract Dataset and DatasetManager types. * Extending features of DataFormat. Extending DataFormat metadata type to initialize with indices and (if needed) implied indices (i.e., not also data fields). * Add data domain and restriction metadata types. Adding new ContinuousRestriction, DiscreteRestriction, and DataDomain types for defining and filtering data domains. * Adjust TimeRange to extend ContinuousRestriction. * Refactor meta_data.py var/property names and doc. * Refactor class order in meta_data.py. * Update DataDomain to contain a data format. Updating DataDomain type to also contain a DataFormat property, to let it encapsulate the data fields in the domain effectively. * Update DataRequirement to use DataDomain directly. Updating DataRequirement type to use the DataDomain to define its domain instead of a generic type variable; with this done, making it a concrete type and removing deprecated CatchmentDataRequirement subtype. * Update modeldata.data init for new/removed types. * Rename test_catchment_data_requirement.py file. Renaming to test_data_requirement.py after making DataRequirement type concrete and removing CatchmentDataRequirement type. * Add DataDomain to modeldata.data package init. * Fix typo in type hints for DataFormat.__init__. * Adjust modeldata.data init for circular imports. * Update DataRequirement unit tests. Updating tests after recent redesign that involved removing the CatchmentDataRequirement subtype. * Remove URI and add UUIDs in dataset and manager. Removing URI in favor of simple string access location property, and adding UUID properties to Dataset and DatasetManager. * Add DatasetManager uuid property function. * Add DatasetUser and manager funcs to link. Adding new DatasetUser type and functions within DatasetManager for managing known users of a Dataset. * Fix dataset.py imports after changes to package. * Update some Dataset docstring and func signatures. * Add abstract Dataset list_files function.
christophertubbs
pushed a commit
to christophertubbs/DMOD
that referenced
this pull request
Jan 24, 2023
* Fix bug related to job types and ModelExecRequest. Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py). * Add uri as dependency for scheduler package. * New WorkerDataRequirement helper type for Jobs. Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker. * Add new JobExecStep values and update docstring. * Redesign JobStatus type. Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type. * Update tests to account for redesigned JobStatus. * Adopt DataRequirement type from modeldata. Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job. * Update scheduler package to depend on modeldata. * Update DataFormat with is_time_series and fields. * Add TimeRange importable from dmod.modeldata.data. * Add abstract Dataset and DatasetManager types. * Extending features of DataFormat. Extending DataFormat metadata type to initialize with indices and (if needed) implied indices (i.e., not also data fields). * Add data domain and restriction metadata types. Adding new ContinuousRestriction, DiscreteRestriction, and DataDomain types for defining and filtering data domains. * Adjust TimeRange to extend ContinuousRestriction. * Refactor meta_data.py var/property names and doc. * Refactor class order in meta_data.py. * Update DataDomain to contain a data format. Updating DataDomain type to also contain a DataFormat property, to let it encapsulate the data fields in the domain effectively. * Update DataRequirement to use DataDomain directly. Updating DataRequirement type to use the DataDomain to define its domain instead of a generic type variable; with this done, making it a concrete type and removing deprecated CatchmentDataRequirement subtype. * Update modeldata.data init for new/removed types. * Rename test_catchment_data_requirement.py file. Renaming to test_data_requirement.py after making DataRequirement type concrete and removing CatchmentDataRequirement type. * Add DataDomain to modeldata.data package init. * Fix typo in type hints for DataFormat.__init__. * Adjust modeldata.data init for circular imports. * Update DataRequirement unit tests. Updating tests after recent redesign that involved removing the CatchmentDataRequirement subtype. * Remove URI and add UUIDs in dataset and manager. Removing URI in favor of simple string access location property, and adding UUID properties to Dataset and DatasetManager. * Add DatasetManager uuid property function. * Add DatasetUser and manager funcs to link. Adding new DatasetUser type and functions within DatasetManager for managing known users of a Dataset. * Fix dataset.py imports after changes to package. * Update some Dataset docstring and func signatures. * Add abstract Dataset list_files function.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continuation of work started in #142: