Profile the Transformers compaction path with torch.profiler

[DaBestCode]

Identify where the measured single-request Apple M4 overhead comes from before any kernel optimization is proposed.

Acceptance criteria:

• Add an opt-in profiling script or flag; normal generation must remain unchanged.
• Profile dense and RandKV runs after warm-up using the same model, prompt, and generation length.
• Summarize the top CPU/device operators by self time and total time.
• Record exact hardware, PyTorch, Transformers, model revision, budget, and buffer.
• Document how to reproduce the profile without checking large trace files into Git.
• Make no speedup claim and add no runtime dependency to the core package.

A CPU or MPS profile is sufficient for this issue; GPU kernel work is tracked separately.

[RezaAshrafii]

Hi, I'd like to work on this issue if it is still available.

My plan is to:

1. Add an opt-in profiling script or flag without changing normal generation behavior.
2. Profile dense and RandKV runs after warm-up using the same model, prompt, and generation length.
3. Summarize the top CPU/device operators by self time and total time.
4. Record the exact hardware, PyTorch version, Transformers version, model revision, budget, and buffer size.
5. Document how to reproduce the profile without committing large trace files.
6. Avoid making any speedup claim or adding a runtime dependency to the core package.

A CPU or MPS profile should be sufficient for the initial contribution. Please let me know if you would prefer a specific model or profiling setup.

[DaBestCode]

Thanks — this is also a useful proposal, and the scope you described matches the issue well.

To keep multiple starter tasks available to new contributors, I assigned you #4 first and will leave this one unassigned for now. If profiling is the task you would rather take, say so and I can switch the assignment.

For whoever picks this up, the preferred baseline is the existing Qwen/Qwen3-0.6B microbenchmark configuration: pin and report the exact model revision, use budget 32, recency buffer 8, warmup 32, max-new-tokens 128, and three trials. Profile one post-warmup run for dense and one for RandKV, then commit only a compact JSON/Markdown summary of the dominant CPU/MPS or CUDA operators and run metadata — not profiler traces or new runtime dependencies.