Repository navigation
Pooling and padding for frozen Mamba features with closed-form heads #1067
AntGalasso
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hello,
I am developing Mamba-Jev, an open-source toolkit that fits ridge, nearest-class-mean (NCM), and linear discriminant analysis (LDA) heads on frozen features using sufficient statistics.
I am preparing an evaluation on intent-classification datasets such as CLINC150, Banking77, and HWU64, starting with a pretrained Mamba checkpoint.
I would appreciate feedback on two implementation questions:
Is the final-normalized hidden state of the last non-padding token a sensible sentence representation, or would you recommend mean or state-based pooling?
Are there known pitfalls involving right-padding and recurrent-state handling when extracting features through Hugging Face versus mamba-ssm?
The Mamba-specific evaluation is still pending, so I am looking for implementation guidance rather than claiming results.
Repository: (https://github.com/AntGalasso/jev-mamba)
Technical notes: attached
Thank you!
[mamba_jev_research_proposal_v031 (1).pdf]
(https://github.com/user-attachments/files/33919855/mamba_jev_research_proposal_v031.1.pdf)
All reactions