Cannot set root temporary path with ray tune

[jpgard]

What happened + What you expected to happen

I have a working pipeline where I load a dataset with ray.data and train/tune models with ray tune.

I'd like to change the location of the root temporary directory which is /tmp/ray by default (this fills up the /tmp directory for some tuning runs).

However, the suggestions on the linked page above (change the root temporary directory by passing --temp-dir={your temp path} to ray start, or provide the _temp_dir argument to ray.init()). don't apply to workflows like mine: when using the datasets and tune APIs, I don't ever explicitly call ray start or ray.init(). I also can't identify where these calls are happening, or where I would provide either of those parameters.

Is there a way to specify the location of the root temporary directory in this case? If not, I think it is needed.

Versions / Dependencies

ray 2.2.0
Python 3.8.15

Reproduction script

See above; this should apply to any call to ray Tuner.fit()

Issue Severity

High: It blocks me from completing my task.

[richardliaw]

You can call ray.init(...) before you run Ray Tune or Datasets. I think we can definitely add this to the documentation though

[jpgard]

Thank you! That is what I was suspecting was the fix...happy to submit a PR if you have a suggestion where in the docs this should go.

[richardliaw]

@justinvyu could you help shepherd @jpgard for a docs fix here?

thanks!

[justinvyu]

Looks like another user ran into the same issue: stack overflow.

The confusion seems to be that ray.init() gets called for you automatically when using AIR (tuner, trainers, datasets). We can clarify this by:

1. Have ray.init() in the section you found link to the to ray.init API ref (it's referenced by ray-init-ref in the doc code).
   • Take a look here on how to link: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-a-custom-anchor
2. In the ray.init() docstring here, we can add a note at the bottom saying that if you are using AIR libraries, then ray.init() will be automatically called for you (ex: on tuner.fit or trainer.fit). If you want to customize the configurations to ray.init(), then manually call it at the beginning of your script.

This guide shows you how to build the docs locally + other tips: https://docs.ray.io/en/latest/ray-contribute/docs.html#contributing-to-the-ray-documentation.

Would the changes above be useful when you were looking into how to do this initially?

Let me know if you have any other questions, I'm happy to help out!

[justinvyu]

Hey @jpgard, just checking in to see if you still want to post a PR for this. I can put it on my backlog otherwise!

[jpgard]

Thanks for following up! Maybe you should put it in your backlog :)

[ndvbd]

What is the ray.init line for changing or disabling output to the /tmp directory?

[justinvyu]

ray.init(_temp_dir="<your_custom_dir>")

You may also want to check out this thread, if you're using Ray Tune: #32776 (comment)