Contributing to the Image Generation Benchmark

[BenjaminBossan]

A place to discuss community contributions to the image generation benchmark. Before starting your experiments, read the contribution guideline and discuss your ideas with the maintainers here.

The results can be seen in this Gradio Space (on the top left, select the image-gen task). Failed experiments (i.e. ones that didn't lead to an improvement) go to the benchmark graveyard -- check it to see past experiments and upload your failed experiments there.

[Sergasgr]

Hi @BenjaminBossan,

I'd like to propose adding a BEFT experiment to the Image Generation Benchmark. I already ran it locally, so I'm sharing the full picture — reasoning and results, including a comparison against the existing LoRA/OFT runs — in one message.

Why BEFT
It wasn't covered in experiments/ yet. Since the DreamBooth "cat pillow" dataset is a few-shot scenario, BEFT feels like a natural gap to fill. The paper specifically reports strong results in low-data regimes, and it also showed a solid memory/accuracy trade-off in the MetaMathQA benchmark recently.

(Side note: This work used AI assistance for configuration review and result analysis. I personally reviewed it, ran the experiment myself, and everything below comes from my own run.)

Setup

• Model: black-forest-labs/FLUX.2-klein-base-4B
• target_modules: matched exactly to the existing LoRA/OFT configs for a fair comparison — to_q, to_k, to_v, to_out.0, add_q_proj, add_k_proj, add_v_proj, to_add_out, to_qkv_mlp_proj, linear_in, linear_out
• Hardware: RTX 5070 Ti, 16GB (Note: existing LoRA/OFT results were run on an L40S, so runtime below isn't a direct head-to-head)

Minor bug report: While copying the target_modules, I noticed "to_add_out" is listed twice in lora/flux2-klein-default/adapter_config.json. I know duplicate entries are a no-op, but I wanted to flag it for a potential quick cleanup. I removed the duplicate for my BEFT run.

Results & Comparison
Here is how BEFT compares against the existing LoRA (r=32) and OFT runs:

Metric	LoRA (r=32)	OFT	BEFT
DINOv2 similarity (test)	0.6972	0.7078	0.5227
Drift	0.2620	0.2619	0.0732
Max memory reserved	10212 MB	9222 MB	9458 MB
Adapter checkpoint size	146.3 MB	18.2 MB	3.4 MB
Trainable params	38,338,560 (0.98%)	4,761,600 (0.12%)	890,880 (0.023%)
Runtime (train)	759.2s (L40S)	524.0s (L40S)	989.8s (RTX 5070 Ti)

An honest read on the trade-offs:
BEFT trades some subject fidelity for a massive drop in trainable parameters (43x smaller than LoRA, 5x smaller than OFT). Interestingly, it shows notably lower drift than both (0.073 vs ~0.262), meaning the base "cat" concept stays much more intact. Memory doesn't scale down proportionally, but that's expected since the frozen 4B base model's activations dominate the VRAM.

Qualitatively, the subject was still very recognizable (I'm attaching a photo and a color drawing sample). I also noticed that validation similarity climbed to 0.530 around step 400 and oscillated in the 0.50–0.52 range through step 700. Since it seemed to plateau early, pushing max_steps further didn't feel necessary for this first pass.

Next Steps
Since this is my first time contributing to this benchmark, I'd love to get your perspective on these numbers. Does my reading of the trade-offs make sense to you? Let me know if you think this profile makes BEFT a worthwhile addition to the current experiments.

[BenjaminBossan]

Thanks for running these further experiments. Please also keep the result files for those experiments that don't yield good results, so that they can be added to the benchmark graveyard.

About the tunable parameters: It's good that you left certain parameters alone, like the number of training steps, since changing this would make results hard to compare. But others like learning rate are absolutely fair game, some PEFT methods will require higher or lower lrs than others. For instance, it seems like HiRA could benefit from a higher learning rate. As for the targeted modules, I think it's also worth exploring, though this should most likely be done with a PEFT method that runs faster as a separate exercise.

• use_asa requires "integration with a training callback" per the config's docstring, which I haven't verified is wired into this benchmark's run.py.

For the image-gen benchmark, it's a simple to this line, which would need to be updated the same as what we have in the MetaMath benchmark. Such a change can be part of the PR with the results.

The AdaMSS results themselves look promising, but even for HiRA, it is what it is, so feel free to submit the results. If you want to run a few more experiments after my feedback, that would also be welcome.

Regarding the OOM warnings, I think this is also related to the PyTorch version, I get them too but never saw them with older versions. Are you running the latest one? Anyway, I would ignore those, the final run will be on other hardware anyway.

[Sergasgr]

Thank you for your feedback @BenjaminBossan,

Both follow-ups are up now — PRs open for HiRA (#3640) and AdaMSS (#3641), same reasoning and results as posted above in this thread.

Ran three more experiments based on your suggestions:

HiRA with a tuned learning rate. Ran a 20-trial Optuna sweep within the paper's suggested [1e-4, 2e-4] range (best: 0.000153), then pushed further to 5e-4 to see if the trend held:

Metric	lr=1e-4 (default)	lr=1.53e-4	lr=5e-4	LoRA (same GPU)
DINOv2 similarity (test)	0.5550	0.6333	0.7075	0.6815
Drift	0.1454	0.2285	0.2626	0.2721

The trend is monotonic across all three LRs, and at 5e-4 HiRA edges past LoRA on DINOv2 (+3.8%) with essentially matching drift. The validation curve at 5e-4 also looks meaningfully more converged than the lower-LR runs — climbs steadily instead of plateauing then jumping late. So this looks less like "HiRA underperforms on this task" and more like "the shared default LR just wasn't right for this method" — exactly what you flagged. It's genuinely encouraging to see HiRA close the gap with LoRA once tuned — though I'd stop short of calling it an outright win yet, since LoRA hasn't had the same LR search applied, and the 0.026 gap is within the noise I've seen step-to-step on this metric within a single run.

BEFT with expanded target_modules, as a separate, faster-method exercise per your suggestion. While inspecting the model I noticed to_out in the single-stream blocks is a bare Linear, unlike the double-stream blocks where it's a to_out.0 inside a ModuleList — so the shared target_modules list (matching on to_out.0) was silently missing the single-stream output projection across all 20 of those blocks, in every method tested so far. Added it (as a regex, since list-mode suffix matching can't distinguish it from the double-stream ModuleList container):

Metric	BEFT (original)	BEFT (+ single-stream to_out)
DINOv2 similarity (test)	0.5227	0.6015
Drift	0.0732	0.1082
Trainable params	890,880	952,320
Runtime	989.8s	989.5s

+15% DINOv2 for +61,440 params and essentially no runtime cost. Seems like BEFT did have room to "target more parameters" after all — just via covering more layers rather than a per-layer capacity knob.

A couple of process items:

• Confirmed I'm on torch 2.13.0+cu130 across all runs, in case that's useful for the OOM-warning question.
• I haven't used the benchmark graveyard before and want to make sure I do it right — could you point me to how and what experiments I can append. I'm keeping every result file, including the ones that don't pan out, so nothing's been discarded.

Happy to keep contributing in running more experiments or other suggestions: more hyperparameter variants, the adaptive AdaMSS (use_asa=True) now that the training-callback support is wired in via my PR, or a different method or direction altogether.

[BenjaminBossan]

Thanks for your further research. I agree that it is a bit dangerous to tune too much on the test set results, as this means we're overfitting on it. Ideally, we would only tune on the validation data, but since the whole dataset is quite small, the validation data is very noisy. For HiRA, I'd still say feel free to update the experiment with the higher learning rate that you found (no need to add this as a second experiment) but let's not tune it any further.

As for BEFT, since its capacity is very constrained to begin with, I think it's fine to submit the updated experiment with more target layers too.

Regarding AdaMSS, feel free to give use_asa=True a try. Let's ensure that the tuning budget is roughly what you had for HiRA (in terms of number of experiments, not runtime) to keep things roughly fair.

• I haven't used the benchmark graveyard before and want to make sure I do it right — could you point me to how and what experiments I can append. I'm keeping every result file, including the ones that don't pan out, so nothing's been discarded.

There is not much to it. The idea is to create a PR to https://huggingface.co/datasets/peft-internal-testing/benchmark-graveyard and just upload your result files. They contain the config from the experiment, so should stand on their own. Follow the existing structure.

[Sergasgr]

Hi @BenjaminBossan,

Opened the two update PRs: HiRA with the tuned LR (#3661) and BEFT with the expanded target_modules (#3663).

AdaMSS with use_asa=True. Ran two configs to resolve an open question first: the default final_warmup=1000 sits above the shared 750-step budget, so I also tried final_warmup=600 to guarantee the pruning schedule can actually finish within it.

Metric	use_asa=False (accepted)	use_asa=True, final_warmup=600	use_asa=True, final_warmup=1000
DINOv2 similarity (test)	0.6790	0.6823	0.6784
Drift	0.2519	0.2416	0.2548

Honest read: all three are within 0.004 of each other on DINOv2 — not a meaningful difference given the run-to-run noise I've seen on this metric elsewhere. I can't tell from what this benchmark logs whether the pruning schedule actually engaged or settled in either ASA run — num_trainable_params is identical across all three configs (2,905,054), which looks like it's captured once at init rather than re-measured after training, so it doesn't help confirm anything either way.

Given the near-identical results, I'd propose keeping final_warmup=600 as the submitted entry — not because it scored better, but because it's the only one that's guaranteed by construction to complete within the shared budget, which seems like the more defensible choice given we can't verify convergence directly.

Benchmark graveyard. Before opening that PR, wanted to confirm the list looks right to you. Four results now superseded or redundant:

• BEFT original (before the to_out target_modules fix)
• HiRA with the shared default lr=1e-4
• HiRA with lr=1.53e-4 (the intermediate Optuna result)
• AdaMSS with use_asa=True, final_warmup=1000 (redundant given the 600 result above)

Does that list look right, or is there anything you'd rather I leave out (or add)?

[BenjaminBossan]

Thank you for running these further experiments and for the updates to HiRA and BEFT. Regarding AdaMSS, I'd say the difference is small enough, well within the range of noise, that we can keep the current experiment as is.

As for the graveyard, please go ahead and submit your result files.