Max job retained pipelined runs #608
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Later used to retrieve job runs to delete when max_pipeline_runs retained has a valid value.
A job level property must be used to keep track of how many job pipeline runs have been scheduled because job pipeline runs can now be deleted, also the parameters of a (cron)job can be edited.
For some reason alembic did not like this migration (i.e. the previous verison) when creating database from scratch.
|
Nice job! LGTM |
Makes it so that the deletion of a job pipeline run directory is done by the celery worker rather than the orchest webserver. This allows to avoid a bidirectional relationship between the orchest-webserver and the orchest-api. The logic related to max_retained_pipeline_runs has been refactored into a 2PF, DeleteNonRetainedJobPipelineRuns.
yannickperrenet
approved these changes
Dec 23, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds to the BE the functionality to automatically clean up the oldest job pipeline runs. This is done through a new field of the
Jobmodel,max_retained_pipeline_runs. This can be specified both during the creation of a job (POST) or update (PUT). This will make it so that 1) when new pipeline runs are scheduled for the job 2) when a pipeline run of the job gets into an end state the logic will check for the oldest job pipeline runs to delete. A requirement for a pipeline run to be deleted is to be in an end state (SUCCESS,FAILED,ABORTED).Some details:
orchest-apidoes not control theuserdir, but theorchest-webserverdoes.pipeline_run_index, this is used to find out what are the oldest job pipeline runs to delete, i.e.highest_index - max_retained_pipeline_runswould be the threshold to be considered for deletionmax_retained_pipeline_runs = 0is a valid value, i.e. every run will cleanup itself after reaching an end stateRelated to #550
@yannickperrenet The deletion of the directory of a job pipeline run could have been done in different manners, I went for preserving the constraint that the
orchest-apidoes not touch theuserdir, and for reusing the existingorchest-webserverendpoint, so now theorchest-apiis actually getting in touch with the webserver and it's quite awkward since it will result in a call to theorchest-api. Keen to know your opinion :).Checklist
./orchest status --extis not reporting failures when Orchest is running.models.pyI have performed the appropriate database migrations.