How to change Flyte's temp storage directory

[samuel-sujith]

Hello All,

I am running flyte tasks as docker containers while mounting the host /tmp into container /tmp.

What happens is that after every task run there is a lot of data in the /tmp folder which is causing the machine to clog up due to tasks running everyday.

Could the community please help in configuring the /tmp location for the flyte. Where do I change the setting so that Flyte can use some other temporary folder.

Kindly help.

[kumare3]

Cc @samhita-alla / @pingsutw / @wild-endeavor ?

[samhita-alla]

I'm assuming it has to do with configuring https://docs.flyte.org/en/latest/concepts/data_management.html#raw-data-prefix, but I'm not very sure. @pingsutw, could you confirm, please?

[samuel-sujith]

Hello community. Please help in the above configuration. How to configure the non default tmp directory for flyte. @kumare3

[pingsutw]

@samuel-sujith
flytekit will write task output to /tmp first, then upload to s3. task output -> /tmp -> s3
What you want is to write the output to other directories first. like task output -> /home/user -> s3 , is it correct?

[pingsutw]

For now, task output is always written to /tmp/flyte, we can make it configurable.

[samuel-sujith]

yes, the requirement is to direct the task output to a configurable folder location.

[wild-endeavor]

Can I get more information about what you're trying to do? I don't entirely understand the question. Are you running things locally (like on your laptop)? Or are you running these tasks on a Flyte backend?

flytekit generally just uses the tempfile library for managing temporary disk usage. According to documentation there, you should be able to control where that goes by "setting the TMPDIR, TEMP or TMP environment variables".

I'm confused what you mean by running "tasks as docker containers". If you're running this on a Flyte backend, which launches Pods that run containers... these Pods should be cleaned up afterwards. They shouldn't clog up anything.

[samuel-sujith]

@pingsutw @wild-endeavor Our current setup involves running flask images which are packaged by pyflyte binary as docker containers on an IBM LSF cluster.

The LSF cluster uses the docker run command to run these jobs and by default the /tmp on the LSF host is mounted into the /tmp of the flask running container. Since Flyte is writing a lot of data into the /tmp, the host directories are getting full which causes problems for us.

[kumare3]

@samuel-sujith would love to catch up on VC and understand. Also definitely ping back on slack.

I think what @wild-endeavor suggested should help