Fix Daily CI reporting issues

[tarekziade]

What this PR does

Fix 1

The scheduled model CI collated report can fail before uploading reports with:

collated_reports.py: error: argument --machine-type/-m: expected one argument

.github/workflows/model_jobs.yml was passing machine_type to the reusable collated report workflow from needs.run_models_gpu.outputs.machine_type.

That job is matrixed, so relying on a matrix job output here can produce an empty value for the downstream reusable workflow.

This moves the normalized report machine type into a small non-matrix report_metadata job and passes needs.report_metadata.outputs.machine_type to collated-reports.yml. The per-matrix test job still sets machine_type in GITHUB_ENV for artifact names exactly as before.

Fix 2

The generated folder matrix crossed GitHub’s 256-job limit:

• July 15: 512 dirs -> [256, 256]
• July 16: 514 dirs -> [257, 257]

Added a slice

Fix 3

Some model CI jobs can upload a partial report artifact without summary_short.txt when pytest does
not reach its terminal summary hook. The collated report job then fails with:

FileNotFoundError: [Errno 2] No such file or directory:
'/summary_short.txt'

This makes utils/collated_reports.py tolerate missing summary_short.txt files. Instead of crashing
the entire collated report job, it records that artifact as an error entry and continues processing
the remaining reports.

This preserves the useful collated report output while still surfacing the malformed/incomplete
artifact as a failure signal.

[tarekziade]

Triggered a new daily https://github.com/huggingface/transformers/actions/runs/29483227997

[vasqu]

Thanks, do you know why this happened all of a sudden? It only started failing a few days ago it seems

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[tarekziade]

Thanks, do you know why this happened all of a sudden? It only started failing a few days ago it seems

the July 13 issues are yet another issue I am looking at

[tarekziade]

Checked run 29502875190:

https://github.com/huggingface/transformers/actions/runs/29502875190

Result: failed, but the reporting fix path worked.

Key signal:

All 4 Model CI / Collated Reports jobs succeeded

Those are:

87641539498 single-gpu slice 0
87642640285 single-gpu slice 1
87643694669 multi-gpu slice 0
87644784416 multi-gpu slice 1

So #47364’s machine-type/matrix fix is doing what we wanted here: matrix jobs materialized,
machine_type was not empty, and collated reports completed.

The run is red because model jobs failed:

llama single + multi
wav2vec2 single + multi
whisper single + multi
blip multi

Examples:

• llama: generation assertion mismatch in test_llama_3_1_hard
• wav2vec2: ONNXRuntime Expand node broadcast failure
• whisper: many integration failures, including float vs half conv mismatch
• blip: model parallelism device mismatch, cuda:0 vs cuda:1

No summary_short.txt / created_at / empty machine_type failure in the collated report jobs on this
run.

[tarekziade]

run-slow: whisper, wav2vec2, llama

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/llama", "models/wav2vec2", "models/whisper"]
quantizations: []

[tarekziade]

new run with the 3 fixes https://github.com/huggingface/transformers/actions/runs/29512139671

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29511760410:1
Result: success | Jobs: 2 | Tests: 13 | Failures: 0 | Duration: 44s

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	3592959d	workflow commit (merge commit)
PR	82618664	branch commit (from PR)
main	0f33294a	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !