Migrate scmp_llm: SC + MP + application/Diffusion + evaluation + archived

[heroarmor]

Summary

End-to-end migration of scmp_llm into scmp_kernels. Every Python file in scmp_llm (SC kernels, Q-DiT integration, evaluation tools, legacy reference code) now has a home here, with a working runtime path for the diffusion application.

What landed

SC kernels (scmp_kernels/sc/)

• sc_triton.py (single-file replacement of the previous kernels.py+matmul.py split), sng.py, rng.py, lfsr_taps.py, config_helpers.py, constants.py — copied verbatim from scmp_llm/SC/, imports rewritten to relative.
• Quant clipping fully deleted. Both unipolar (q_lo, q_hi = 2, q_max - 2) and bipolar (q_clip = q_norm - 2) margins are gone, including the q_clip/q_clip_min/q_norm Triton kernel parameters and _grouped_symmetric_quant's clip_margin kwarg. 8-bit unipolar now uses full [0, 255] (256 levels); 8-bit bipolar uses full [-127, 127].

MP dispatch (scmp_kernels/mp/)

• config.py ported from scmp_llm/SC/mp_config.py (762 lines).
• __init__.py re-exports the public API: MPConfig, AdaptiveMPConfig, RangeMPConfig, RowAssignment, classify_rows_by_metric, adaptive_classify_rows, classify_groups_by_range, MPDistributionLogger, MetricProfiler.

Application: application/Diffusion/

• 91 source files copied from scmp_llm/Q-DiT/ (including models/evaluations/{evaluator.py, convert_npz.py, ...} — 92MB Inception checkpoint excluded; download separately).
• All bare imports rewritten: from sc_triton/sng/config_helpers/mp_config import ... → from scmp_kernels.{sc,mp}.* import ....
• Dead sys.path.insert(0, .../SC) shims removed throughout.
• qdit/sc_integration/mp_config.py is now a thin re-export shim of scmp_kernels.mp.
• application/{ViT,WorldModel}/ reserved as .gitkeep placeholders.

Evaluation (evaluation/)

• evaluation/Diffusion/: kid.py, compare_images.py, build_full_mosaic.py, build_sample_grids.py (from scmp_llm/evaluation/).
• evaluation/Diffusion/imagenet_ref/: extract.py, parallel_npz.py, compute_fid_kid.py, compare_grid.py (from scmp_llm/imagenet256_ref/).
• evaluation/{ViT,WorldModel}/ placeholders.

Archived (archived/)

• archived/origin_cpu/: sc.py, sc_enable.py — original NumPy/PyTorch-CPU SC reference impls.
• archived/bench/: bench_table_vs_compact.py, compare_cbsg.py, compare_enable.py, compare_matmul.py, compare_unarysim.py, test_kernel_opt.py.
• archived/tools/: dse.py, noise_model_calibration.py.
• Each subfolder has a README explaining contents and the bare-import caveat.

Repo hygiene

• .gitignore for __pycache__/, *.py[cod], *.egg-info/, build/, dist/, .pytest_cache/.
• README.md documents the new layout.
• MIGRATION_PLAN.md deleted (executed; stale).

Coverage audit

Source	Files	Funcs/Classes	Covered
scmp_llm/SC/ (active migration: 6 files)	6	127	127
scmp_llm/SC/ (archived: 10 files)	10	46	46
scmp_llm/Q-DiT/	59	277	277
scmp_llm/evaluation/	4	15	15
scmp_llm/imagenet256_ref/	4	–	4 files
TOTAL	83	465 + 4	100%

Known not-migrated (vit_sc-only; never existed in scmp_llm)

• det_kernel_tuning
• make_sobol_antithetic_config, make_sobol_altseed_config
• auto_calibrator.py (RidgeFitter, auto_calibrate_mp, AutoMPBudgetLogger)
• set_current_block_idx / _CURRENT_BLOCK_IDX global hook
• FreeBoundaryMPConfig, fixed_levels extra field on AdaptiveMPConfig

These are vit_sc-side conveniences. Add when porting vit_sc into application/ViT/.

Verified

• Imports: import scmp_kernels, from scmp_kernels.{sc,mp,sc.sc_triton,sc.sng,sc.config_helpers}, plus all qdit.sc_integration.* modules — clean under torch 2.10.0+cu128 / triton 3.6.0.
• Symbol parity: every top-level def/class in the migrated scmp_llm files exists in the corresponding scmp_kernels file (AST-checked).
• MP wiring: calibration scripts (calibrate_mp_thresholds.py etc.) write a JSON threshold table; runtime constructs AdaptiveMPConfig(threshold_table_path=...) at quant_sc_main.py:576; sc_attention.py / sc_mlp.py call adaptive_classify_rows per-forward and dispatch each row's precision through the rng_levels= arg of the SC Triton kernels.

Not verified (no GPU on the dev login node)

• Actual Triton kernel execution / numerics. Run on a CUDA box:

  PYTHONPATH=. python -m pytest tests/test_sc_smoke.py -v
  PYTHONPATH=application/Diffusion python -c \"import qdit.sc_integration.sc_attention\"
  

Test plan

• pytest tests/test_sc_smoke.py on a CUDA + Triton box.
• Diffusion pilot: run a short FID sweep with --mp_config_path <calibrated_table.json> and confirm metric is in expected band given the wider quant range (256 levels vs 252).
• Confirm vit_sc-only items (det_kernel_tuning, auto-calibrator) are acceptable as follow-up when ViT lands.

[Allenjin123]

We want a clear separation between the SC implementation and the application implementation, so no diffusion-related code should be included. We should also remove all legacy code(that is what initial commit is doing), since we are primarily using agents to write and maintain the code now. Deleting stale content will help reduce context length.

Please use this repository as a submodule for the other repository. Also, please commit changes directly to the existing repository instead of rebasing. Let's actually learn how to collaborate as a team! Thanks.