fix examples: bad K8s log config causing logs to be lost

[davidxia]

across Container restarts

Ray logs are written to /tmp/ray by default. Right now
various RayCluster example YAMLs use volumeMounts for /var/log
which doesn't have anything written to it.

So right now /tmp/ray logs are deleted after Container restarts.

This change sets volumeMounts.mountPath to /tmp/ray so that
logs in this dir are persisted across Container restarts for the life
of the Pod.

This change also adds volumes for head Pods which was missing before.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[DmitriGekhtman]

Thanks for the fix! This looks good to me.
@davidxia just to confirm, is this PR ready for review? If so, could you click the button that marks it as such? :)
(It's currently marked as a draft.)

[davidxia]

thanks, Moved out of draft. I wanted to add some tests for this. Do you think that'd be helpful? I can do that in another PR if so.

[DmitriGekhtman]

Tests are always appreciated! What kind of testing did you have in mind?

[davidxia]

That the ray logs dir /tmp/ray is mounted in these example YAMLs. Are there existing tests for these example YAMLs that check a certain value is used correctly?

Also is /tmp/ray in some constant that a test can use?

[DmitriGekhtman]

That the ray logs dir /tmp/ray is mounted in these example YAMLs. Are there existing tests for these example YAMLs that check a certain value is used correctly?

I don't think we systematically test the sample YAMLs, but this would be an excellent opportunity to start doing that.
(At the moment, it looks like all of the tests define their own RayCluster structs, rather than reading from the files.)
Just validating the volumes/mounts in the examples would be great.

Also is /tmp/ray in some constant that a test can use?

Looks like we added that here:

kuberay/ray-operator/controllers/ray/common/pod.go

Line 25 in 92d1ffb

RayLogVolumeMountPath = "/tmp/ray"

[davidxia]

Thanks, feel free to merge this PR first to unblock. I'll try to write tests for this soon.