[Data][Train] Read example data anonymously in download/quickstart doc examples - #64060
Closed
ArturNiederfahrenhorst wants to merge 6 commits into
Closed
Conversation
…c examples Three CI-executed doc examples read public example buckets with ambient AWS credentials, which fail with AccessDenied on the restricted premerge runner role when the doc-test cache misses (same root cause as ray-project#64059, which fixes loading-data.rst): - doc/source/data/working-with-images.rst: download("image_url") -> pass an anonymous pyarrow S3 filesystem. - doc/source/train/doc_code/{xgboost,lightgbm}_quickstart.py: pd.read_csv of s3://ray-example-data/iris/... -> storage_options={"anon": True}. The data is public, so anonymous access is correct and removes the dependency on runner credentials. Verified the anonymous reads against the live bucket. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates documentation and quickstart examples to ensure S3 data is accessed anonymously. Specifically, it configures pyarrow.fs.S3FileSystem with anonymous access in the image-working guide, and adds storage_options={'anon': True} to pandas S3 read operations in the LightGBM and XGBoost quickstart scripts. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
ray-gardener
Bot
added
docs
TimothySeah
approved these changes
Jun 12, 2026
TimothySeah
left a comment
TimothySeah
left a comment
Contributor
There was a problem hiding this comment.
lgtm but looks like tests are still failing
`download(filesystem=pyarrow.fs.S3FileSystem(...))` silently ignored the filesystem's credential configuration when obstore was the backend (as in CI). The obstore download path re-derives S3 credentials from the supplied filesystem via `getattr`, but a native PyArrow `S3FileSystem` exposes only `region` to Python -- `anonymous`/keys/assume-role live in the underlying C++ object. So obstore fell back to the ambient AWS credential chain: signing requests that 403 against public buckets on credentialed runners (anonymous), or quietly using the wrong identity (explicit keys). This is why the `data: doc tests` suite intermittently failed on loading-data.rst. Translate the filesystem to obstore options from its pickle state instead -- PyArrow round-trips the full construction config there (it must, to ship filesystems to workers) -- keeping the fast obstore path for anonymous, static keys, and the default credential chain. Configs that can't be represented as static options fall back to the threaded path, which applies the filesystem verbatim: assume-role (PyArrow refreshes those creds; a static obstore snapshot would go stale) and any unrecognized pickle shape (caught defensively, so a future PyArrow change degrades to "slower", never "wrong"). Replace the introspection unit tests -- including one that asserted anonymity through a MagicMock faking an attribute real PyArrow never exposes, which is why CI stayed green while the path was broken -- with tests that drive real S3FileSystem objects through the translation. Update the loading-data and working-with-images docs to pass an anonymous filesystem to `download()`, which now works and stays on the fast path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>
…non-s3 Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>
…mples pandas already auto-retries S3 reads with anon=True on credential errors (pandas/io/common.py, GH 34626), so the public iris CSVs load fine without the explicit flag. It only saved a failed signed first attempt (~60ms-1s depending on creds/IMDS), which isn't worth diverging these examples from the plain read_csv form. The Ray Data download/read_parquet examples in this PR keep their anonymous filesystem -- those have no such fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>
Brings in ray-project#64059 and ray-project#64089, which already landed every change this PR carried. Net diff vs master should now be empty. Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com>
Contributor
Author
|
Superceded |
ArturNiederfahrenhorst
added a commit
that referenced
this pull request
Jun 17, 2026
…e tests (#64033) ## Description In recent PyTorch (2.9+), `torch.onnx.export` routes through the rewritten "dynamo" exporter, which imports `onnxscript` at call time. The rllib CI image doesn't install `onnxscript`, so the ONNX-export example tests fail with `ModuleNotFoundError: No module named 'onnxscript'`: - `//rllib:examples/_old_api_stack/inference/policy_inference_after_training_with_attention_torch` — actively failing in postmerge (e.g. builds 17958, 17990) - `//rllib:examples/inference/policy_inference_after_training_w_connector_w_onnx` — same `--use-onnx-for-inference` → `torch.onnx.export` path; fails whenever it's scheduled Example failed build: https://buildkite.com/ray-project/postmerge/builds/17979#019eb34e-018e-4ca0-bf51-877a5b569b48 Prerequisites because of seemingly unrelated Ray Data issue: #64059 #64060 --------- Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
limarkdcunha
pushed a commit
to limarkdcunha/ray
that referenced
this pull request
Jun 30, 2026
…e tests (ray-project#64033) ## Description In recent PyTorch (2.9+), `torch.onnx.export` routes through the rewritten "dynamo" exporter, which imports `onnxscript` at call time. The rllib CI image doesn't install `onnxscript`, so the ONNX-export example tests fail with `ModuleNotFoundError: No module named 'onnxscript'`: - `//rllib:examples/_old_api_stack/inference/policy_inference_after_training_with_attention_torch` — actively failing in postmerge (e.g. builds 17958, 17990) - `//rllib:examples/inference/policy_inference_after_training_w_connector_w_onnx` — same `--use-onnx-for-inference` → `torch.onnx.export` path; fails whenever it's scheduled Example failed build: https://buildkite.com/ray-project/postmerge/builds/17979#019eb34e-018e-4ca0-bf51-877a5b569b48 Prerequisites because of seemingly unrelated Ray Data issue: ray-project#64059 ray-project#64060 --------- Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
elliot-barn
added a commit
that referenced
this pull request
Jul 8, 2026
…e tests (#64033) ## Description In recent PyTorch (2.9+), `torch.onnx.export` routes through the rewritten "dynamo" exporter, which imports `onnxscript` at call time. The rllib CI image doesn't install `onnxscript`, so the ONNX-export example tests fail with `ModuleNotFoundError: No module named 'onnxscript'`: - `//rllib:examples/_old_api_stack/inference/policy_inference_after_training_with_attention_torch` — actively failing in postmerge (e.g. builds 17958, 17990) - `//rllib:examples/inference/policy_inference_after_training_w_connector_w_onnx` — same `--use-onnx-for-inference` → `torch.onnx.export` path; fails whenever it's scheduled Example failed build: https://buildkite.com/ray-project/postmerge/builds/17979#019eb34e-018e-4ca0-bf51-877a5b569b48 Prerequisites because of seemingly unrelated Ray Data issue: #64059 #64060 --------- Signed-off-by: Artur Niederfahrenhorst <artur@anyscale.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Elliot Barnwell <elliot.barnwell@anyscale.com> (cherry picked from commit 043f430) Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Three more CI-executed doc examples read public example buckets with ambient AWS credentials, so they fail with
AccessDeniedon the restricted premerge runner role whenever the cache misses — same root cause as #64059 (which fixesloading-data.rst).Prerequisite: #64089