fine_tune to a text classification task

[WilliamHoo]

I am trying to get mamba working for a text classification task by adding a classification head after the model.

For transformer models, people usually the last_hidden_state as the input to the classification head, any suggestions for mamba?

[WilliamHoo]

also, any recommendations on tokenizers?

[albertfgu]

I don't know much about tokenizers for fine tuning. For the classification head, many variations are possible. You could grab the final recurrent state of the model (although that might be unsupported with the current released version); you could grab the output at the last timestep; you can average the outputs at all timesteps.

[turian]

@WilliamHoo If you do figure out how to do this, I would be curious

[jmunozmendiFunditec]

I would also be interested. Any news?

[maksymdolgikh]

See my comment here #163 for a suggestion.

[getorca]

I've put this, https://github.com/getorca/mamba_for_sequence_classification together to be compatible with HF to use mamba for sequence classification.