Skip to content

[release] Fix huggingface_accelerate release test: floor peft>=0.18 for transformers 5.x - #65062

Merged
elliot-barn merged 3 commits into
masterfrom
hf-update
Jul 29, 2026
Merged

[release] Fix huggingface_accelerate release test: floor peft>=0.18 for transformers 5.x#65062
elliot-barn merged 3 commits into
masterfrom
hf-update

Conversation

@elliot-barn

@elliot-barn elliot-barn commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Why are these changes needed?

The huggingface_accelerate nightly release test fails at startup with:

ImportError: cannot import name 'HybridCache' from 'transformers'

Root cause: after the HuggingFace stack upgrade (#64054), the ray-ml
base-extra-testdeps lock pairs transformers==5.11.0 with peft==0.17.1.
peft has no source pin — it is pulled in transitively by lm_eval — so uv
left it at a stale resolution. peft 0.17.x imports HybridCache from
transformers at module scope, but transformers 5.x removed that class, and
transformers' trainer_utils imports peft whenever it is installed, so any
import transformers on the image crashes.

Fix: add a peft>=0.18 floor (0.18 made the HybridCache import lazy and
version-guarded; verified 0.19.1 only references it inside a deferred
Gemma-specific code path) to the ml byod source requirements for both py3.10
and py3.13, and recompile the affected locks:

  • python/deplocks/base_extra_testdeps/ray-ml-base_extra_testdeps_py3.10.lock: peft 0.17.1 → 0.19.1 (no other resolution changes)
  • release/ray_release/byod/ml_torchft_py3.13.lock: already resolved peft 0.19.1; only picks up the new via annotation

Related issue number

N/A — nightly release-test failure (huggingface_accelerate, Train tests group).

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR. (pre-commit hooks ran at commit time)
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/

Testing:

  • Locks regenerated with bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/rayimg.depsets.yaml --name ray_ml_base_extra_testdeps_cuda_310 and ... build ci/raydepsets/configs/release_ml_torchft_tests.depsets.yaml (both compiled successfully; diffs are minimal as described above).
  • Verified against the peft 0.19.1 wheel that peft/peft_model.py no longer imports HybridCache at module scope.
  • End-to-end verification requires a nightly image rebuild + release-test run of huggingface_accelerate.

AI assistance: this PR was prepared with AI assistance (Claude Code); I reviewed every changed line. Not a duplicate: searched open PRs for peft and huggingface_accelerate — no existing PR addresses this failure.

🤖 Generated with Claude Code

Successful release test run: https://buildkite.com/ray-project/release/builds/102272/canvas

…or transformers 5.x

The ray-ml base-extra-testdeps lock paired transformers==5.11.0 with
peft==0.17.1 (pulled in transitively by lm_eval, unconstrained). peft
0.17.x unconditionally imports HybridCache from transformers at module
import time, but transformers 5.x removed it, so any 'import
transformers' path that touches trainer_utils crashes with
ImportError: cannot import name 'HybridCache'. This broke the
huggingface_accelerate release test at startup.

Add a peft>=0.18 floor to the ml byod source requirements (0.18 made
the HybridCache import lazy/guarded) and recompile the affected locks:
the py3.10 ray-ml testdeps lock moves peft 0.17.1 -> 0.19.1; the py3.13
torchft lock was already on 0.19.1 and only picks up the annotation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the peft dependency requirement to >=0.18 in the BYOD ML requirements files (requirements_ml_byod_3.10.in and requirements_ml_byod_3.13.in) and updates the corresponding lockfiles. This change is made to prevent runtime import errors with transformers 5.x due to the removal of HybridCache in older peft versions (0.17.x). I have no feedback to provide.

@ray-gardener ray-gardener Bot added train Ray Train Related Issue release-test release test labels Jul 28, 2026
elliot-barn and others added 2 commits July 28, 2026 18:39
- Import AdamW from torch.optim (removed from transformers in 5.x)
- Use namespaced dataset id Yelp/yelp_review_full (huggingface-hub 1.x
  rejects unnamespaced repo ids in hf:// URIs)
- Rename evaluation_strategy to eval_strategy (removed in transformers 5.x)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
…s test image

The post-build script pinned accelerate==0.32.0 and peft==0.10.0, but the
base ray-ml image now ships accelerate 1.14.0 / peft 0.19.1, and
transformers 5.x requires accelerate>=1.1.0 for Trainer. Use the base
image versions like huggingface_accelerate already does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
@elliot-barn
elliot-barn requested a review from a team as a code owner July 28, 2026 21:02
@elliot-barn elliot-barn added the go add ONLY when ready to merge, run all tests label Jul 28, 2026
@elliot-barn
elliot-barn enabled auto-merge (squash) July 28, 2026 22:41
@elliot-barn
elliot-barn merged commit 6e160e6 into master Jul 29, 2026
8 of 9 checks passed
@elliot-barn
elliot-barn deleted the hf-update branch July 29, 2026 00:37
elliot-barn added a commit that referenced this pull request Jul 29, 2026
…oor peft>=0.18 for transformers 5.x (#65062) (#65101)

Cherry-pick of #65062 onto `releases/2.57.0`.

## Why are these changes needed?

The `huggingface_accelerate` nightly release test fails at startup with:

```
ImportError: cannot import name 'HybridCache' from 'transformers'
```

Root cause: after the HuggingFace stack upgrade (#64054), the ray-ml
base-extra-testdeps lock pairs `transformers==5.11.0` with
`peft==0.17.1`. `peft` has no source pin — it is pulled in transitively
by `lm_eval` — so uv left it at a stale resolution. peft 0.17.x imports
`HybridCache` from transformers at module scope, but transformers 5.x
removed that class, and transformers' `trainer_utils` imports peft
whenever it is installed, so any `import transformers` on the image
crashes.

Fix: add a `peft>=0.18` floor (0.18 made the `HybridCache` import lazy
and version-guarded) to the ml byod source requirements for both py3.10
and py3.13, and recompile the affected locks.

Clean cherry-pick of merge commit
6e160e6 — no conflicts, diff identical
to master.

Successful release test run on master:
https://buildkite.com/ray-project/release/builds/102272/canvas

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests release-test release test train Ray Train Related Issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants