[<Ray component: Tune>] TensorflowCheckpoint Temp Folder is hard coded to system variable

[drukoz]

Description

We need to change this as this way we can set where we ant to save the tmp files that are used while trials are paused .
we have a HPC that has a small drive and a huge network storage we had to change the tmp folder for the windows env and i am guessing that linux has the same issue as well when this can be configured by python env variable or passing it to the class.

Use case

Configure the directory the temporary "model.keras" which are of a decent size is saved when using raytune tensorflow checkpoint or any derivative like ReportCheckpointCallback for tensorflow

[matthewdeng]

@drukoz are you using the ReportCheckpointCallback? If so, would it suffice if we deleted the tempdir after the checkpoint is reported?

[drukoz]

@drukoz are you using the ReportCheckpointCallback? If so, would it suffice if we deleted the tempdir after the checkpoint is reported?

Hello Matthew,
Yes and maybe i think that would work my only issue that might be that depending on the type of search algo those tmp files might be required to restart paused trials and depending on samples and concurrency/resources it might fill up the tmp folder regardless!
hence why setting the folder would really be the best solution. this obviously being an optional parameter.

[matthewdeng]

If you're using Ray Tune, the paused trials should be pointing to checkpoints that are persisted after calling train.report. These would be in a location defined in RunConfig.storage_path, rather than the temporary ones here.

[drukoz]

ah ok i was not aware of that! i had noticed if i delete the tmp files that are not in use mid trials that it fails but maybe i did it wrong and deleted some that were mid use!

[drukoz]

@matthewdeng do you know this this was included in the nightly build?