Add Nemotron 3 to tests via tiny model

[sergiopaniego]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@qgallouedec @albertvillanova

---

Note

Medium Risk
Medium risk because it introduces a new hybrid Mamba-Attention tiny model and adds test-specific workarounds that patch transformers internals and alter training args (disabling gradient checkpointing) for that model.

Overview
Adds generation of a new tiny-NemotronHForCausalLM (Nemotron 3 hybrid Mamba+Attention) model in generate_tiny_models.py, including CPU-friendly config defaults and a float32 override for specific mixer parameters before pushing to the Hub.

Extends DPOTrainer and SFTTrainer parametrized training tests to include this tiny NemotronH model (guarded on transformers>=5.3.0) and adds a targeted workaround: patch kernel lazy-loading and disable gradient checkpointing when running NemotronH tests.

^{Written by Cursor Bugbot for commit 6ba7b76. This will update automatically on new commits. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

Thanks!

The CI is red:

  FAILED tests/test_dpo_trainer.py::TestDPOTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM] - RuntimeError: causal_conv1d with channel last layout requires strides (x.stride(0) and x.stride(2)) to be multiples of 8
  FAILED tests/test_sft_trainer.py::TestSFTTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM] - RuntimeError: causal_conv1d with channel last layout requires strides (x.stride(0) and x.stride(2)) to be multiples of 8

[qgallouedec]

thanks!! just a few comments

[qgallouedec]

Can you cast backbone.layers.[N].mixer.D and backbone.layers.[N].mixer.A_log to fp32?: it seems like these two layers are in fp32, and we want to be as close as possible to the reference model

check how we do here for Qwen3.5 https://github.com/huggingface/trl/pull/5278/changes#diff-dd3349f840a26de373fc88378e6fcded0b75423da8a34f7cfa6ac573b7398b8bL404

[qgallouedec]

really not sure about this. we don't train on cpu, so why testing it + we wouldn't know it a gpu-specific issue is introduced

[qgallouedec]

Thanks!

The CI is red:

  FAILED tests/test_dpo_trainer.py::TestDPOTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM] - RuntimeError: causal_conv1d with channel last layout requires strides (x.stride(0) and x.stride(2)) to be multiples of 8
  FAILED tests/test_sft_trainer.py::TestSFTTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM] - RuntimeError: causal_conv1d with channel last layout requires strides (x.stride(0) and x.stride(2)) to be multiples of 8

is it possible that this error originates from what params are used to build the model?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[sergiopaniego]

Fixes applied. There is an issue with some dependencies that needs to be addressed in transformers.
I've created the PR there: huggingface/transformers#44853.