Contributing to the MetaMathQA Benchmark

[BenjaminBossan]

A place to discuss community contributions to the MetaMathQA benchmark. Before starting your experiments, read the contribution guideline and discuss your ideas with the maintainers here.

The results can be seen in this Gradio Space. Failed experiments (i.e. ones that didn't lead to an improvement) go to the benchmark graveyard -- check it to see past experiments and upload your failed experiments there.

[szzhoujiarui]

@BenjaminBossan Thanks for setting up this discussion. I’ll continue the VBLoRA MetaMathQA work here.
I’ll start by trying unsloth/Llama-3.2-3B with the pad token explicitly set to the EOS token, as you suggested. I’ll first reproduce the existing VBLoRA baseline and compare the training loss before exploring any new configurations.

I’ll share the results here once the baseline check is complete.

[szzhoujiarui]

@BenjaminBossan

VBLoRA baseline diagnosis update

Completed a full B0 baseline run with the setup discussed earlier:

• Model: unsloth/Llama-3.2-3B (snapshot d4446454, local snapshot, offline)
• PEFT: 0.20.1.dev0 (6b2eda7 on my fork's vblora-metamathqa-optimized)
• Transformers: 5.14.1, PyTorch: 2.13.0+cu130, datasets: 5.0.1
• Config: VBLoRA default (r=4, num_vectors=256, vector_length=256, topk=2, q_proj + v_proj), batch_size=4, max_steps=5000, eval_steps=250, AdamW lr=1e-4 + cosine, seed 0
• Data: MetaMathQA (revision aa4f34d) / GSM8K (revision 740312a), matches official revision
• Hardware: RTX 5080 16 GB, use_gc=true

Metric	Official (Meta Llama)	Local (Unsloth, GC)
Step 250 train loss	1.3084	1.6055
Step 5000 train loss	0.7142	1.0322
GSM8K test accuracy	33.36%	1.7%
Wiki loss before training	1.58611	1.58627

Diagnostics to isolate the gap:

• GC on/off produces identical step 250 loss (1.6055 vs 1.6056).
• Per-step loss analysis shows the deviation starts at step 1.
• VBLoRA adapter initial perturbation is negligible (base vs. adapter loss diff = 0.00125).
• VBLoRA implementation, training loop, optimizer, and scheduler are unchanged from official.
• Data revisions match official, RNG state is deterministic.
• No OOM / NaN / crash across all runs.

The base model forward loss matches closely, but the gradient path diverges — likely due to the Unsloth weights or the Transformers version delta. I'm unable to access huggingface.co from this environment to download the original Meta Llama checkpoint for validation.

Question: Is the Unsloth baseline acceptable as a reference for tuning experiments (e.g., num_vectors to 512, LR sweep), or should we wait until the Meta Llama weights can be loaded?

[BenjaminBossan]

Thanks for reporting the results. It's fine if the new experiment presents a trade-off, it doesn't need to improve on all fronts. If you have the time, you could try even higher values (larger VB, higher lr) to see if even more improvement is possible.

[szzhoujiarui]

Thanks! I’ll run a small controlled follow-up search, testing a larger vector bank and a moderately higher learning rate while changing one factor at a time. I’ll report the complete comparison and use only the strongest candidate for the configuration PR.

[szzhoujiarui]

I completed the follow-up experiments, changing one factor at a time before testing their combination.
These were single seed-0 local runs.

Run	num_vectors	LR	Step 5000 loss	GSM8K test accuracy	Forgetting
C2	512	3e-4	0.671287	36.8461%	0.0643
C3	1024	3e-4	0.658788	39.7271%	0.0555
C4	512	5e-4	0.661107	38.9689%	0.0654
C5	1024	5e-4	0.648648	41.2434%	0.0591

Both single-factor changes improved over C2, and their combination in C5 performed best. Compared with the reproduced default baseline at 33.2828%, C5 improved GSM8K test accuracy by about 7.96 percentage points.

The trade-off is that the 1024-vector configuration has approximately 4x the trainable adapter parameters and checkpoint size of the 256-vector baseline. The checkpoint was about 18.5 MB, while peak GPU memory remainedaround 13 GB in the local gradient-checkpointed run.

I’m stopping the search at this predefined boundary rather than continuing to tune against the test benchmark. I’ll prepare a minimal configuration-only PR for C5, containing no local paths, gradient-checkpointing override, result files, or runner changes.

[BenjaminBossan]

Nice to see even better results. Note that VB-LoRA shares the vector bank, so even if we double the number of vectors there, it doesn't necessarily mean we require a lot more memory. Do you have the exact memory numbers, especially max memory? That's more interesting to look at than pure parameter count.

[szzhoujiarui]

Good point. I checked the exact accelerator_memory_max values recorded by the benchmark. This metric is the peak reserved accelerator memory after subtracting the initial allocation.

Run	num_vectors	LR	Max memory
B0	256	1e-4	12.6426 GiB
C1	512	1e-4	12.5332 GiB
C2	512	3e-4	12.6621 GiB
C3	1024	3e-4	12.5703 GiB
C4	512	5e-4	12.6621 GiB
C5	1024	5e-4	12.6973 GiB

C5 used only 56 MiB more peak reserved memory than B0, an increase of about 0.43%. At the same learning rate, C3 was actually slightly below C2, while C5 was only 36 MiB above C4, so these small differences are likely within CUDA allocator variation.

Therefore, although the trainable parameter count and checkpoint size increase by approximately 4x, the measured training peak memory is essentially unchanged in these local gradient-checkpointed runs. This is consistent with the vector bank being shared.

[zdd3143]

Hi @BenjaminBossan,

I'd like to contribute a MetaMathQA experiment. Before running anything,
I wanted to check the direction with you.

Proposal: add a CARTRIDGE entry

As far as I can tell, CARTRIDGE (peft/tuners/cartridge/, its docs page and
tests/test_cartridge.py all exist) currently has no experiment in any method_comparison benchmark — I checked image-gen, MetaMathQA and text_generation_benchmark. The "add a new PEFT method" checklist in the contribution guideline lists method_comparison configs as a step, so this looks like a missing item rather than a new idea.

Reasons I think it fits this harness:

• CartridgeConfig inherits PromptLearningConfig and is served as
  past_key_values, like prefix tuning — and MetaMathQA already has a prefixtuning experiment.
• The harness already supports its initialisation: run.py imports
  initialize_kv_prefix_from_text and init_kv_cache_prefix is an existing
  training param, so paper option 3 (KV from a prompt) is directly usable;
  paper option 2 (random KV) needs nothing extra.
• method_comparison/peft-shop/app.py already carries a
  "CARTRIDGE": "cartridges" docs-slug override.

Planned config, mirroring
prefixtuning/llama-3.2-3B-tokens50-init_prefix-lr_0.005:
num_virtual_tokens=50, num_frozen_tokens=1, base model
meta-llama/Llama-3.2-3B, everything else at defaults.

If CARTRIDGE isn't a good target, I'm equally happy to do hyperparameter work on an existing method in the same style as the VBLoRA runs above
— I have a 16 GB GPU and can report accelerator_memory_max.

AI assistance disclosure: I used an AI assistant for repository
exploration and drafting this. I verified each claim above against the local
source, and I'll run the experiment myself.

Thanks!

[zdd3143]

Thanks for the explanation on CARTRIDGE — that makes sense, I'll drop it.

On the hardware question:

I can get a 32 GB GPU (Colab / Kaggle), or rent something larger if that's
what it takes. Before I spend that budget, which would you steer me toward:

1. MetaMathQA on a 32 GB cloud GPU, or
2. the image-gen benchmark, as you suggested.

And if there's a specific method or family you'd like covered in either
benchmark, I'd much rather work on that than propose another method that
turns out to be a poor fit — you clearly have context about these methods
that isn't visible from reading the code.

AI assistance disclosure: I used an AI assistant for repo exploration
and drafting this reply.

Thanks!

[zdd3143]

Quick correction to my previous comment — I skipped past your actual question,
sorry about that.

To be precise on hardware: my local GPU is a laptop RTX 4060 with 8 GB, so
MetaMathQA will OOM locally (based on the VBLoRA numbers above it needs
around 12.6 GiB). What I really have access to is a 16 GB cloud GPU on the
Colab / Kaggle free tiers, or I can rent 24-32 GB if it's needed.

So the question stands: would you rather I target MetaMathQA on a rented GPU,
or start with image-gen on the 16 GB free tier, as you suggested? Either
works for me — I just don't want to spend the budget in the wrong direction.

[BenjaminBossan]

About which benchmark: Why not go with the image-gen one if you can run it for free. If I would good a cut for sending you two a paid cloud provider, I'd send you there ;-)

Regarding the PEFT method: I think it makes much more sense to check a specific one that is underexplored (you can see if there is only the default config), read the claims in the paper, then make your own judgment if you think it should be able to fare better. I would take this as learning opportunity much rather than just blindly going through the motions. Don't take me wrong, if you just intend to spend your time and resources to run grid search for PEFT, that's also fine, though I don't quite see what your personal benefit would be.

[zdd3143]

While looking for an underexplored method in the image-gen benchmark, I noticed that ln_tuning is the lowest-scoring run of all 39 in image-gen/results/ (test dino_similarity 0.3415, vs. 0.7431 for lora rank64 and 0.6972 for lora default). Its validation metric looked wrong, so I looked into it.

(Note: this is about the image-gen benchmark rather than MetaMathQA — sorry if this is the wrong discussion. Happy to move it if there's a better place.)

The default config is effectively a no-op

results/ln_tuning--flux2-klein-default.json:

• valid dino_similarity is bit-identical at all 7 eval checkpoints (0.23442742228507996)
• drift is exactly 0.0
• 6,400 of 3,875,550,976 parameters are trainable (target_modules: ["norm_q", "norm_k"])

A bit-identical validation metric plus exactly-zero drift suggests the adapter is not affecting the model at all, rather than merely performing poorly.

It reproduces on a different machine and different library versions

I re-ran the same experiment on an RTX 4080 SUPER with torch 2.14.0+cu130 / transformers 5.17.0 / diffusers 0.40.0, against the reference L40S with torch 2.12.0+cu130 / transformers 5.10.2 / diffusers 0.38.0:

	reference (L40S)	my run (4080 SUPER)
valid dino_similarity	0.23442742228507996, identical at all 7 checkpoints	0.21392177045345306, identical at all 7 checkpoints
drift	0.0	0.0
test dino_similarity	0.3415	0.3379
trainable params	6,400	6,400

Same qualitative signature and the same parameter count. The absolute values differ slightly — most likely because the validation split has only 2 rows and the base model's forward pass differs across library versions.

Root cause: the learning rate

The experiment has no training_params.json, so it runs at the default lr=1e-4. With only 6,400 trainable parameters, that is far too small to move the model within 750 steps.

I ran a 4-point sweep changing only the LR (everything else, including weight_decay, unchanged):

lr	test dino_similarity	drift
1e-4 (default)	0.3379	0.0000
1e-2	0.4115	0.0820
3e-2	0.4764	0.1111
1e-1	0.3751	0.2073

A clean inverted U peaking at 3e-2: test dino_similarity goes from 0.3379 to 0.4764 (+41%) and drift goes from exactly 0 to 0.111 — the adapter actually starts affecting the model. At 1e-1 the run becomes unstable: the validation metric peaks at 0.5480 at step 400 and then decays to 0.3070, so 3e-2 is bounded from above as well.

The default is therefore roughly 300x below the useful range, and ln_tuning is currently shown on the leaderboard as the worst method when it is really just an untuned configuration.

Caveats

• The validation split has only 2 rows and its metric is very noisy — at lr=3e-2 it oscillates between 0.295 and 0.525 with no monotonicity. I don't think it can be used for hyperparameter selection, so I used the test metric only to establish the trend.
• I stopped after 4 points rather than searching further, to avoid tuning against the test benchmark. The exact optimum is not known.

Secondary observation

Every run emits:

UserWarning: Unsupported layer type '<class 'torch.nn.modules.normalization.RMSNorm'>' encountered, proceed at your own risk.

and the model dump shows norm_q / norm_k wrapped as LNTuningLayer while norm_added_q / norm_added_k are left as plain RMSNorm. Possibly worth a separate look, though it does not affect the LR finding above.

Proposal

Add experiments/ln_tuning/flux2-klein-lr3e-2/, configuration-only:

• adapter_config.json: identical to flux2-klein-default
• training_params.json: {"optimizer_kwargs": {"lr": 0.03, "weight_decay": 0.0001}}

I'm happy to open a PR for that, and equally happy to run additional LR values first if you'd prefer a different point or a different protocol.

AI assistance disclosure: I used an AI assistant for repository exploration, environment debugging and drafting this message. I ran all experiments myself and verified every number above directly from the result JSONs.

[BenjaminBossan]

Thanks for running this experiment, please proceed with the PR. I think you can update the default experiment, no need to create a separate one, as the changed parameters are general parameters, not something specific to the PEFT method, so it's still "default".

Btw. for the image-gen benchmark, please use discussion #3522 in the future.

[Sergasgr]

Hi @BenjaminBossan,

I'd like to add VeLoRA to MetaMathQA (there are no VeLoRA experiments yet). Its main claim is lower training memory at similar accuracy, which this benchmark measures directly, so it seems like a good fit.

I ran two configs, each identical to an existing LoRA experiment plus velora_config, with the default training params (no training_params.json):

• lora/llama-3.2-3B-rank32-velora: counterpart of lora/llama-3.2-3B-rank32 (q_proj/v_proj)
• lora/llama-3.2-3B-rank10-target-mlp-velora: counterpart of lora/llama-3.2-3B-rank10-target-mlp (the paper finds the down projection to be the most effective placement, Table 6)

Both use VeloraConfig(num_groups=32, init_type="batch_average_once", scale=1.0), the paper's best sub-token size (D/32, Table 5b; the PEFT default of 64 was less accurate there) and initialization (Table 5c). I did not tune anything beyond that.

Results (VeLoRA on an RTX 5090 32 GB, baselines are the official L40S results):

Config	Test acc	Peak memory (accelerator_memory_max)	Forgetting	Trainable params
LoRA r=32 (q,v), official	0.4784	21260 MiB	0.050	9,175,040
+ VeLoRA	0.4412 (-3.7 pp)	19346 MiB (-1914, -9.0%)	0.100	9,175,040
LoRA r=10 MLP, official	0.4943	23554 MiB	0.032	9,461,760
+ VeLoRA	0.4701 (-2.4 pp)	19462 MiB (-4092, -17.4%)	0.045	9,461,760

Honest read: the memory savings are real and in line with what I expected from the saved activations (VeLoRA replaces the inputs stored for the LoRA A projections with their compressed version), and they are much larger for the MLP config because down_proj has an 8192-dim input. Accuracy goes down, though: with 1319 test samples the standard error of the difference between two runs is about 1.9 pp, so the q/v drop is probably real and the MLP drop is borderline. Forgetting also increases. One caveat is that I compared against the L40S results rather than running the LoRA baselines on the same GPU, so the exact deltas should be taken with a grain of salt.

If you think this is useful, I can open a config-only PR with the two ⁠adapter_config.json⁠ files, or put either one in the PR and the other in the graveyard.

[BenjaminBossan]

Thanks for running the VeLoRA experiments. I agree they would be useful to have. Since the config targeting MLP layers shows more convincing results, I would propose to add that one and the other one can go to the graveyard.