Doing things inside /data/rw/pit in interactive jobs is very painful because of sshfs, a git status can take tens of seconds to complete. But doing things outside of the snakepit mounts means risking losing data if something goes wrong unexpectedly and your job gets stopped/killed, or if you stop it and forget to copy things first.
If we could provide a cleanup script that is executed as part of job end, then I could copy any critical folders (checkpoints, etc) into networked folders to make sure nothing is forgotten.
Doing things inside
/data/rw/pitin interactive jobs is very painful because of sshfs, agit statuscan take tens of seconds to complete. But doing things outside of the snakepit mounts means risking losing data if something goes wrong unexpectedly and your job gets stopped/killed, or if you stop it and forget to copy things first.If we could provide a cleanup script that is executed as part of job end, then I could copy any critical folders (checkpoints, etc) into networked folders to make sure nothing is forgotten.