Initial data flow and data service pre-work #142
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This won't pass all test until the changes in #134 are merged in (ironically, it probably will also fail silently, and thus actually pass the checks). |
Adding SimpleDockerUtil utility class inside scheduler package with basic Docker functionality and properties, and adjusting the Launcher class to inherit from it.
Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py).
Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker.
Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type.
Adding to compensate for similar function no longer being in JobStatus after its recent redesign.
Fixing placement of @AbstractMethod annotation for AbstractDataSubset property storage_size, and fixing ordering of AbstractDataSubset subclasses that was causing a resolution order problem.
Initial few unit tests for CatchmentDataRequirement.
Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job.
Accounting for change from ALLOCATED to AWAITING_SCHEDULING.
Account for workflow changes in job_manager.py, required as a direct or indirect result of adding steps for various execution states in the workflow related to data requirements.
Accounting for renaming of ALLOCATED to AWAITING_SCHEDULING within unit tests for monitor service.
robertbartel
force-pushed
the
launch_jobs/forcings
branch
from
aab5985
to
ac2d209
Compare
Actually committing change to name of JobExecStep value that is already reflected elsewhere.
|
Relates to #128. |
6 tasks
donaldwj
pushed a commit
to donaldwj/DMOD
that referenced
this pull request
Jun 21, 2022
* Add simple Docker util class. Adding SimpleDockerUtil utility class inside scheduler package with basic Docker functionality and properties, and adjusting the Launcher class to inherit from it. * Expose scheduler package SimpleDockerUtil class. * Fix bug related to job types and ModelExecRequest. Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py). * Add uri as dependency for scheduler package. * New WorkerDataRequirement helper type for Jobs. Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker. * Add new JobExecStep values and update docstring. * Add get_for_name funcs for job phase and step. * Redesign JobStatus type. Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type. * Update tests to account for redesigned JobStatus. * Minor fixes to docstrings. * Add Job.should_release_resources property. Adding to compensate for similar function no longer being in JobStatus after its recent redesign. * Rename one of the JobExecStatus values. * Fix data_subset.py bug with abstract declarations. Fixing placement of @AbstractMethod annotation for AbstractDataSubset property storage_size, and fixing ordering of AbstractDataSubset subclasses that was causing a resolution order problem. * Add dmod.modeldata.data metadata classes. * Initial unit tests for one new metadata class. Initial few unit tests for CatchmentDataRequirement. * Adopt DataRequirement type from modeldata. Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job. * Account for JobExecStep name change in manager. Accounting for change from ALLOCATED to AWAITING_SCHEDULING. * Account for changes in job management workflow. Account for workflow changes in job_manager.py, required as a direct or indirect result of adding steps for various execution states in the workflow related to data requirements. * Account for change of JobExecStep value in tests. Accounting for renaming of ALLOCATED to AWAITING_SCHEDULING within unit tests for monitor service. * Update scheduler package to depend on modeldata. * Update schedulerservice dep version of scheduler. * Fix bad monitorservice dep version for monitor. * Fix change of ALLOCATED to AWAITING_SCHEDULING. Actually committing change to name of JobExecStep value that is already reflected elsewhere.
christophertubbs
pushed a commit
to christophertubbs/DMOD
that referenced
this pull request
Jan 24, 2023
* Add simple Docker util class. Adding SimpleDockerUtil utility class inside scheduler package with basic Docker functionality and properties, and adjusting the Launcher class to inherit from it. * Expose scheduler package SimpleDockerUtil class. * Fix bug related to job types and ModelExecRequest. Fixing bug resulting from extending MaaSRequest to ModelExecRequest but not updating all usage (here, in job.py). * Add uri as dependency for scheduler package. * New WorkerDataRequirement helper type for Jobs. Adding new type in job.py to encapsulate the definition of data input or output needs for a scheduled worker. * Add new JobExecStep values and update docstring. * Add get_for_name funcs for job phase and step. * Redesign JobStatus type. Redesigning JobStatus class to be a serializable composite of JobExecPhase and JobExecStep, which it was before anyway, but without itself redunantly being another enum type. * Update tests to account for redesigned JobStatus. * Minor fixes to docstrings. * Add Job.should_release_resources property. Adding to compensate for similar function no longer being in JobStatus after its recent redesign. * Rename one of the JobExecStatus values. * Fix data_subset.py bug with abstract declarations. Fixing placement of @AbstractMethod annotation for AbstractDataSubset property storage_size, and fixing ordering of AbstractDataSubset subclasses that was causing a resolution order problem. * Add dmod.modeldata.data metadata classes. * Initial unit tests for one new metadata class. Initial few unit tests for CatchmentDataRequirement. * Adopt DataRequirement type from modeldata. Replacing internal WorkerDataRequirement type with DataRequirement type from dmod.modeldata, and adopting its use within Job. * Account for JobExecStep name change in manager. Accounting for change from ALLOCATED to AWAITING_SCHEDULING. * Account for changes in job management workflow. Account for workflow changes in job_manager.py, required as a direct or indirect result of adding steps for various execution states in the workflow related to data requirements. * Account for change of JobExecStep value in tests. Accounting for renaming of ALLOCATED to AWAITING_SCHEDULING within unit tests for monitor service. * Update scheduler package to depend on modeldata. * Update schedulerservice dep version of scheduler. * Fix bad monitorservice dep version for monitor. * Fix change of ALLOCATED to AWAITING_SCHEDULING. Actually committing change to name of JobExecStep value that is already reflected elsewhere.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adjustments to jobs, job status and execution state classes, and job services, in preparation to include the necessary pieces for getting forcing, observation, hydrofabric, etc. data in and out of where it needs to be.
The biggest pieces are: