Add Apertus model support with XIeLU activation

[sinievanderben]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

Comments

This PR adds support for the novel Extended Inverse ELU (xIELU) activation and makes the Apertus‑to‑TransformerLens weight converter aware of its trainable parameters.

Key changes:

This PR adds support for the Apertus model.

New trainable xIELU activation.

• XIELU class in utils.py with learnable α₊, α₋, β.
• Original xielu function kept for static use.
• Added "xielu" to ACTIVATION_FN_DICT.

Activation wiring

• can_be_used_as_mlp.select_activation_function() now instantiates XIELU when cfg.act_fn == "xielu".

Apertus weight converter improvements

• Extracts trainable activation parameters (alpha_p, alpha_n, beta).
• Handles different attribute locations (mlp.act_fn, mlp.act, or top‑level attrs).
• Falls back to default constants if missing (back‑compatible with old checkpoints).

[jlarson4]

Hello @sinievanderben!

In order to get this PR merged and included in a future release there are a couple things I will need from you:

1. Please make sure your code passes all CI checks. MyPy is currently flagging your changes.
2. Please add tests for the new XIELU feature.

Thank you!

[sinievanderben]

sorry, I am not 100% certain but the error message of the failure looks like something I did not introduce? I can still be wrong!

[jlarson4]

I believe you are correct, I will look into it!