Release v0.5.0 doc validation part 2 #999

architkulkarni · 2023-03-31T17:31:33Z

Why are these changes needed?

Continuation of #997. As in that PR,

This PR mainly focuses on correctness validation rather than document improvement.
We use the nightly KubeRay operator image to test the documents.

https://github.com/ray-project/kuberay/blob/master/docs/guidance/ingress.md
- The script does not always work when run all at once, likely due to some race condition. Added a troubleshooting step to unblock the user if this happens. Ideally we should figure out the correct "wait" step to insert in the script to fix the issue.
https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayjob.md
- Passes 10/10 times on Ray 2.3 after [Hotfix][release blocker][RayJob] HTTP client from submitting jobs before dashboard initialization completes #1000.
https://github.com/ray-project/kuberay/blob/master/docs/guidance/tls.md
- Passes. Only tested "TL;DR" path. We should explicitly add the command for "# Install v0.5.0 KubeRay operator". I used kind create cluster followed by helm install kuberay-operator --set image.repository=kuberay/operator --set image.tag=nightly ~/kuberay/helm-chart/kuberay-operator
https://github.com/ray-project/kuberay/blob/master/docs/guidance/pod-command.md
- Passes

https://github.com/ray-project/kuberay/blob/master/docs/guidance/pod-security.md

Passes. Note that the job "succeeds" but the script itself raises an error:

Traceback (most recent call last):
File "samples/xgboost_example.py", line 28, in <module>
  result = trainer.fit()
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/train/base_trainer.py", line 363, in fit
  result_grid = tuner.fit()
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/tuner.py", line 292, in fit
  return self._local_tuner.fit()
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/impl/tuner_internal.py", line 455, in fit
  analysis = self._fit_internal(trainable, param_space)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/impl/tuner_internal.py", line 573, in _fit_internal
  **args,
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/tune.py", line 679, in run
  callbacks, sync_config, metric=metric, progress_metrics=progress_metrics
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/utils/callback.py", line 105, in _create_default_callbacks
  callbacks.append(TBXLoggerCallback())
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/logger/tensorboardx.py", line 165, in __init__
  from tensorboardX import SummaryWriter
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/__init__.py", line 5, in <module>
  from .torchvis import TorchVis
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/torchvis.py", line 10, in <module>
  from .writer import SummaryWriter
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/writer.py", line 16, in <module>
  from .comet_utils import CometLogger
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
  from .summary import _clean_tag
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/summary.py", line 12, in <module>
  from .proto.summary_pb2 import Summary
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
  from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
  from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
File "/home/ray/anaconda3/lib/python3.7/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 42, in <module>
  serialized_options=None, file=DESCRIPTOR),
File "/home/ray/anaconda3/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 561, in __new__
  _message.Message._CheckCalledFromGeneratedFile()

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower)

this doesn't affect the main point of the doc, which is to demonstrate security settings.

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

kevin85421 · 2023-04-01T00:34:43Z

docs/guidance/ingress.md

@@ -201,3 +201,12 @@ kubectl describe ingress raycluster-ingress-head-ingress
 #        [Note] The forward slash at the end of the address is necessary. `<ip>/raycluster-ingress`
 #               will report "404 Not Found".
 ```
+


Maybe we just need to add a comment in Step 2 that says to execute the kubectl wait command after the Pod for the NGINX ingress controller is created or just sleep 10. This is an expected behavior for kubectl wait.

The commands are copied from here.

kevin85421 · 2023-04-01T00:36:41Z

For pod-security.md, this bug is totally from Ray Docker images. It has been fixed by Ray 2.2 but happened again in Ray 2.3 images. See #873 for more details. Maybe we should update the example script in Step 5.2.

kevin85421

Remove the section "Example: NGINX Ingress on KinD (built-in ingress support)" in ingress.md and remove https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.ingress.yaml.

gvspraveen · 2023-04-02T23:00:47Z

For

does not gracefully handle multiple jobs with the same submission ID submitted in quick succession
May be in the description here link to your PR in 2.4 which fixes this

architkulkarni · 2023-04-03T15:58:11Z

For

does not gracefully handle multiple jobs with the same submission ID submitted in quick succession
May be in the description here link to your PR in 2.4 which fixes this

Actually it works now after Kai-Hsun's fix #1000, I've updated the PR description

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni · 2023-04-04T20:32:40Z

For pod-security.md, this bug is totally from Ray Docker images. It has been fixed by Ray 2.2 but happened again in Ray 2.3 images. See #873 for more details. Maybe we should update the example script in Step 5.2.

I confirmed that this script passes with rayproject/ray-ml:nightly, so I've updated the pod security sample YAML to use ray-ml:latest. When the Ray 2.4 docker images are released, the test should pass.

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni · 2023-04-04T22:52:14Z

Discussed offline, we'll use Ray 2.2.0 for the pod security doc for now until we can confirm ray:latest works (likely after 2.4.0 release of Ray)

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

kevin85421

LGTM

docs/guidance/ingress.md

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Release v0.5.0 doc validation part 2

Add troubleshooting for ingress.md

c14a65e

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni requested review from gvspraveen and kevin85421 March 31, 2023 23:18

kevin85421 reviewed Apr 1, 2023

View reviewed changes

kevin85421 mentioned this pull request Apr 1, 2023

[Bug] Encounter error in Step 5.2 in pod-security.md due to ray-ml docker images #873

Closed

2 tasks

kevin85421 reviewed Apr 1, 2023

View reviewed changes

architkulkarni added 2 commits April 4, 2023 10:21

Remove built-in ingress doc and YAML

3a23226

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Update ray ml image to latest in pod security example YAML

10eacd3

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Change version to 2.2.0 in security yaml

484bd73

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni marked this pull request as ready for review April 4, 2023 22:50

Fix typo

cfa9f5c

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Update ingress.md to add sleep and remove retry instruction

393381b

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

kevin85421 approved these changes Apr 4, 2023

View reviewed changes

architkulkarni commented Apr 4, 2023

View reviewed changes

docs/guidance/ingress.md Outdated Show resolved Hide resolved

Update docs/guidance/ingress.md

e660a7b

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni changed the title ~~[WIP] Release v0.5.0 doc validation part 2~~ Release v0.5.0 doc validation part 2 Apr 4, 2023

kevin85421 merged commit 31c1e6a into ray-project:master Apr 4, 2023

kevin85421 mentioned this pull request Apr 5, 2023

[Bug][Docs] Ingress EKS bug and documentation #441

Closed

2 tasks

architkulkarni deleted the doc-updates branch June 12, 2023 21:53

kevin85421 mentioned this pull request Jun 23, 2023

Operator oc support #1185

Closed

1 task

lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023

Release v0.5.0 doc validation part 2 (ray-project#999)

a763e1e

Release v0.5.0 doc validation part 2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release v0.5.0 doc validation part 2 #999

Release v0.5.0 doc validation part 2 #999

architkulkarni commented Mar 31, 2023 •

edited

Loading

kevin85421 Apr 1, 2023

kevin85421 Apr 3, 2023

kevin85421 commented Apr 1, 2023 •

edited

Loading

kevin85421 left a comment

gvspraveen commented Apr 2, 2023

architkulkarni commented Apr 3, 2023 •

edited

Loading

architkulkarni commented Apr 4, 2023

architkulkarni commented Apr 4, 2023

kevin85421 left a comment

Release v0.5.0 doc validation part 2 #999

Release v0.5.0 doc validation part 2 #999

Conversation

architkulkarni commented Mar 31, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

kevin85421 Apr 1, 2023

Choose a reason for hiding this comment

kevin85421 Apr 3, 2023

Choose a reason for hiding this comment

kevin85421 commented Apr 1, 2023 • edited Loading

kevin85421 left a comment

Choose a reason for hiding this comment

gvspraveen commented Apr 2, 2023

architkulkarni commented Apr 3, 2023 • edited Loading

architkulkarni commented Apr 4, 2023

architkulkarni commented Apr 4, 2023

kevin85421 left a comment

Choose a reason for hiding this comment

architkulkarni commented Mar 31, 2023 •

edited

Loading

kevin85421 commented Apr 1, 2023 •

edited

Loading

architkulkarni commented Apr 3, 2023 •

edited

Loading