osdc/hf-cache: tier rclone memory by GPU count - #883
Merged
Conversation
huydhn
force-pushed
the
hf-cache-multigpu-mem
branch
from
July 8, 2026 07:04
b953f3d to
c650466
Compare
huydhn
changed the title
huydhn
marked this pull request as ready for review
July 8, 2026 07:04
huydhn
force-pushed
the
hf-cache-multigpu-mem
branch
4 times, most recently
from
July 8, 2026 19:01
6cb5bfe to
bf44172
Compare
huydhn
changed the title
huydhn
force-pushed
the
hf-cache-multigpu-mem
branch
10 times, most recently
from
July 8, 2026 20:55
ec4fcb5 to
80cd34b
Compare
jeanschmidt
approved these changes
Jul 8, 2026
Contributor
|
overall LGTM :) |
rclone RSS OOMs track job concurrency ~ GPU count, not GPU model: an 8-GPU g5.48xlarge (a10g) OOMs its 1Gi mount the same way a p5 (h100) does. pytorch#874 only covered h100/b200, so a10g still wedged. Render one mount DaemonSet per GPU-count tier (instance-gpu-count), reserved (request == limit) memory scaled to concurrency: 8-GPU -> 4Gi, 4-GPU -> 2Gi, 2-GPU -> 1Gi, 1-GPU -> 512Mi, CPU/rest -> 256Mi. The CPU catch-all (NotIn [1,2,4,8], also matches non-GPU nodes) guarantees every node gets a mount to clear the startup taint. Retires -largegpu. Account for hf-cache in the packing analysis: it's a .tpl the YAML scan skips, so it was never counted. daemonset_overhead adds a 256Mi base on all nodes; analyze_node_utilization adds the per-GPU-count top-up (hf_cache_gpu_topup_mib) since a flat constant can't express the tiers.
The GPU runners are sized to ~full node memory, so hf-cache reserving memory per GPU count (this PR) pushes the memory-maximal ones past allocatable (analyze-utilization: 0 pods, MEM-bound). Trim job memory by the tier reservation so they fit: 8-GPU: a10g-8 704->700Gi, t4-8 344->340Gi (-4Gi) 4-GPU: a10g-4 167->165Gi, t4-4/l4-4 172->170Gi (opt 173->171Gi) (-2Gi) Labels keep their -NNN- memory (names stable, no GHA label change). 1-GPU runners still fit on memory; the remaining ~30-70m CPU gap is hf-cache's pre-existing 100m CPU request (since pytorch#874), now merely visible.
huydhn
force-pushed
the
hf-cache-multigpu-mem
branch
from
July 8, 2026 21:06
80cd34b to
d5e4d74
Compare
This was referenced Jul 8, 2026
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 9, 2026
…runner trim (pytorch#886) pytorch#883 trimmed the `l-x86iavx512-45-172-t4-4` runner 172→170Gi (room for hf-cache's 4-GPU reserve) but left the integration test's hardcoded `EXPECTED` at 172Gi. `TORCH_CI_MAX_MEMORY` = the runner's memory in bytes, so `test-gpu-t4-multi` now fails (172 ≠ 170). Update `EXPECTED` to 170Gi (`182536110080`). (The separate "Duplicate runner names" smoke failure is **not** from pytorch#883 — it's the `arc-runners-opt` module from pytorch#871 sharing names with `arc-runners`; needs its own fix.)
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 9, 2026
…les (pytorch#887) The `test_names_are_unique` smoke test loaded the **union** of every `arc-runners*/defs`, so it flags all 44 names as duplicates now that `arc-runners-opt` (pytorch#871) exists with names identical to `arc-runners`. Those two are **mutually exclusive** (only meta-prod-aws-ue1 uses `-opt`; no cluster enables both), so the shared names are by design. Scope the check to the cluster's **enabled** arc-runners* modules — mirroring the existing `test_configmaps_for_all_defs`. Per cluster exactly one variant is enabled, so names are unique; a real collision (two name-overlapping variants enabled together) is still caught. Not related to pytorch#883 (that was memory-only); this fixes a pre-existing collision introduced by pytorch#871.
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rclone OOMs on high-concurrency GPU nodes regardless of GPU model — observed on an a10g
g5.48xlarge, not just H100/B200. #874 keyed the 4Gi split oninstance-gpu-name In [h100,b200], leaving a10g at 1Gi.Example failure (a10g): https://github.com/pytorch/pytorch/actions/runs/28908017258/job/85761532112 — node
ip-10-4-176-7(g5.48xlarge, 8× a10g) on meta-prod-aws-ue2, rclone OOMKilled →/mnt/hf_cachedead FUSE → job podCreateContainerError.Render one mount DaemonSet per GPU-count tier (
karpenter.k8s.aws/instance-gpu-count), reserved (request == limit) memory scaled to concurrency:In [8]In [4]In [2]In [1]NotIn [1,2,4,8]Affinity keeps tiers mutually exclusive (the
NotIncatch-all also matches non-GPU nodes, so CPU is covered and every node still gets a mount to clear the startup taint). Retires the pre-tier-largegpuDaemonSet.Also counts hf-cache in the packing analysis (it's a
.tplthe YAML scan skipped):daemonset_overheadadds a 256Mi base on all nodes, andanalyze_node_utilizationadds the per-GPU-count top-up (hf_cache_gpu_topup_mib) — a flat constant can't express the tiers.Packing impact (
just analyze-utilization)Large-RAM GPU nodes (p5/p6-b200/p4d, 2TB / 1TB) have 50–86Gi of slack — the reservation is negligible there. The small-RAM GPU nodes run a single runner sized to ~full node memory, so reserving there needs a matching trim. Job memory dropped by the tier reservation (labels keep their
-NNN-, names stable):After the trims every GPU runner fits on memory. The 1-GPU runners already fit; the remaining ~30–70m CPU gap on the small-GPU runners is hf-cache's pre-existing 100m CPU request (since #874), now merely visible in the overhead — it's <0.15% of node CPU and within the tool's kubelet estimate, not a new failure.
Deploy: redeploy
hf-cacheandarc-runners(+arc-runners-opt) to uw1 + ue2 (the runner-memory trims ship with the reservation).