Skip to content

Fix cached_files silently returning stale file on read-only filesystem (EROFS) - #47852

Merged
ydshieh merged 2 commits into
mainfrom
fix_erofs_cached_files
Aug 9, 2026
Merged

Fix cached_files silently returning stale file on read-only filesystem (EROFS)#47852
ydshieh merged 2 commits into
mainfrom
fix_erofs_cached_files

Conversation

@ydshieh

@ydshieh ydshieh commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

CI

Problem

When the HF Hub cache is on a read-only filesystem (in some rare CI environments, a pre-populated cache may be mounted read-only, potentially for security reasons), hf_hub_download still contacts the Hub to resolve the latest commit hash. If a newer revision exists, it then tries to write the new snapshot pointer / blob to disk and fails with OSError: [Errno 30] Read-only file system (EROFS).

On the cached_files side, EROFS is not handled: Python only promotes EACCES (errno 13) to its own PermissionError subclass, so the existing isinstance(e, PermissionError) guard catches EACCES but misses EROFS (errno 30), which is a plain OSError. Without this fix, the EROFS error falls through to the stale-cache recovery block, which silently returns the old cached file — even when the Hub has a newer revision.

Fix

Add an explicit elif isinstance(e, OSError) and e.errno == errno.EROFS guard that re-raises the error before the stale-cache recovery block, giving callers a chance to catch it and retry with a writable cache path (our patch in conftest.py).

Why EROFS reaches cached_files but EACCES doesn't

The difference comes down to where the kernel rejects the write:

Method FS writable? Rejected at errno Python type
chmod 555 / chown root Yes DAC permission check 13 (EACCES) PermissionError (subclass of OSError)
mount -o ro / read-only volume No VFS mount flag check 30 (EROFS) plain OSError
  • With chmod/chown, the filesystem is still writable — the kernel reaches the DAC check, finds the user lacks permission on that path, and returns EACCES. Python promotes this to PermissionError, so the existing guard in cached_files catches it.
  • With a read-only mount, the kernel rejects the write at the VFS layer (mount flag MS_RDONLY) before any permission check. This returns a plain OSError(errno.EROFS) that Python does not promote to PermissionError, so it slips past the existing guard.

Impact

This changes behaviour only when cached_files is called with a cache directory on a filesystem mounted read-only via mount -o ro (or equivalent kernel-level read-only mount). This is a rare setup — primarily used in our own CI for security/isolation purposes (pre-populated read-only cache volumes). The behaviour change for regular users (writable cache, network errors, permission-bit errors) is zero.

Testing

Verified by:

  1. Confirming CI fails (old stale chat_template returned) without this fix: https://github.com/huggingface/transformers/actions/runs/31317535108/job/93255227530
  2. Confirming CI passes (new chat_template loaded correctly) with this fix applied: https://github.com/huggingface/transformers/actions/runs/31317927760

…m (EROFS)

When the HF Hub cache lives on a read-only filesystem (EROFS, errno 30),
`hf_hub_download` contacts the Hub, resolves a newer commit hash, and then
fails with an `OSError(EROFS)` when trying to write the new snapshot pointer
or blob to disk.  That error is not caught inside `hf_hub_download` itself
and bubbles up to the `except` block in `cached_files`.

The existing guards there only re-raise for `RepositoryNotFoundError`,
`RevisionNotFoundError`, `PermissionError` (EACCES, errno 13), and
`ValueError`.  Python maps EACCES to its own `PermissionError` subclass, but
EROFS (errno 30) is a plain `OSError` that does **not** satisfy
`isinstance(e, PermissionError)`.  Without this fix it falls through to the
stale-cache recovery block, which silently returns whatever old file was
previously cached — even though the Hub has a newer revision.

Fix: add an explicit `elif isinstance(e, OSError) and e.errno == errno.EROFS`
guard that re-raises the error before the recovery block, giving callers
(e.g. the CI `_with_tmpdir_cache_fallback` wrapper) a chance to catch it and
retry the download against a writable temporary cache directory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 31319427357:1
Result: success | Jobs: 16 | Tests: 179,409 | Failures: 0 | Duration: 15h 38m

@ydshieh
ydshieh merged commit fd12552 into main Aug 9, 2026
112 checks passed
@ydshieh
ydshieh deleted the fix_erofs_cached_files branch August 9, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants