Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete output files on re-run & UI run check #421

Merged
merged 4 commits into from
Nov 30, 2020

Conversation

ajstewart
Copy link
Contributor

This fixes #418 and removes the parquet and arrow files before the run is re-run.

It also adds a check when running though the UI that the run is not already running or is queued (maybe an old tab is open or something).

For future reference, two things will need to changed:

@ajstewart ajstewart added enhancement New feature or request UI User Interface labels Nov 25, 2020
@ajstewart ajstewart self-assigned this Nov 25, 2020
@github-actions github-actions bot added this to In progress in Pipeline Backlog Nov 25, 2020
Copy link
Contributor

@srggrs srggrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment below

vast_pipeline/management/commands/runpipeline.py Outdated Show resolved Hide resolved
@ajstewart
Copy link
Contributor Author

@srggrs the forced measurements parquet was a little confusing in the current state - as it stands, until we are able to add an image, and because we don't store the job running config in the DB, when re-running a pipeline the forced measurements should always be checked for and removed.

So I've removed the removal step in the actual forced extraction part of the code and made the removal of the old data in the run pipeline command take care of everything. It will check for the forced measurements to delete from the database on every re-run and delete all the parquets.

Pipeline Backlog automation moved this from In progress to Review in progress Nov 30, 2020
Copy link
Contributor

@srggrs srggrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good! Merge master after merging #401 then I'll approve

@ajstewart
Copy link
Contributor Author

All good! Merge master after merging #401 then I'll approve

@srggrs all merged.

Pipeline Backlog automation moved this from Review in progress to Reviewer approved Nov 30, 2020
@ajstewart ajstewart merged commit 1040cf2 into master Nov 30, 2020
Pipeline Backlog automation moved this from Reviewer approved to Done Nov 30, 2020
@ajstewart ajstewart deleted the issue-418-clear-files-before-rerun branch November 30, 2020 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request UI User Interface
Projects
Development

Successfully merging this pull request may close these issues.

Parquets (and arrow files) not being removed before a run (not a bug)
2 participants