GCP Housekeeping jobs?

[MattWellie]

We have a number of jobs which nominate a temporary data directory, and write potentially huge amounts of data to them (e.g. AnnotateCohort which generates multiple checkpoints, or GATK-SV in general which is a storage-hungry beast).

I would like to consider follow-on jobs which would run if the main job completes successfully, clearing the temporary storage directory which was used.

We'd need to poke the numbers, and see if this is worth the effort.

[vivbak]

We'd need to poke the numbers, and see if this is worth the effort.

I agree with this. I'm scared of deleting things in the pipeline :)

Tag this under: Investigate?

[vivbak]

Also tag this under cost optimisation