Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store image job results indefinitely #57

Closed
ssube opened this issue Jan 15, 2023 · 1 comment
Closed

store image job results indefinitely #57

ssube opened this issue Jan 15, 2023 · 1 comment
Labels
status/fixed issues that have been fixed and released type/bug broken features
Milestone

Comments

@ssube
Copy link
Owner

ssube commented Jan 15, 2023

Jobs are immediately removed from memory if the client comes back to check /ready, with #55, but abandoned jobs are not removed until the Executor starts to prune them, which defaults to 50.

Look for jobs that finished more than N minutes ago and remove their futures, probably as part of the /ready endpoint.

@ssube ssube added status/new issues that have not been confirmed yet type/bug broken features labels Jan 15, 2023
@ssube ssube added this to the v0.5 milestone Jan 16, 2023
@ssube ssube changed the title remove old jobs from memory after a while store image job results indefinitely Jan 19, 2023
@ssube
Copy link
Owner Author

ssube commented Jan 19, 2023

The FutureCollection used to store jobs does not record when they were started, making it difficult to prune jobs on a regular basis, but it does limit the overall growth.

If a job has been removed from the collection and the client comes to check on that job, they will not get an affirmative value. Adding a check for the output file might work, or keeping a database of all the image parameters and their start/end times.

@ssube ssube removed this from the v0.5 milestone Jan 19, 2023
@ssube ssube added status/progress issues that are in progress and have a branch and removed status/new issues that have not been confirmed yet labels Jan 23, 2023
@ssube ssube added this to the v0.5 milestone Jan 23, 2023
@ssube ssube added status/fixed issues that have been fixed and released and removed status/progress issues that are in progress and have a branch labels Jan 23, 2023
@ssube ssube closed this as completed Jan 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/fixed issues that have been fixed and released type/bug broken features
Projects
None yet
Development

No branches or pull requests

1 participant