New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding Bert Prediction Head #1721

Open

peridotml opened this issue Feb 16, 2024 · 0 comments

peridotml commented Feb 16, 2024

I ported SPLADE (https://huggingface.co/naver/efficient-splade-VI-BT-large-query) to candle and I had to add a prediction head to the BertModel. I would be happy to contribute it, but I would need some help.

My implementations basically copies the transformers library's BertOnlyMLMHead.

Let me know if you you are interested and I will do my best in a PR.

The tricky part was managing the tied layer with the VarBuild, but I'm sure with a bit of help it wouldn't be too challenging.

peridotml mentioned this issue

Add Splade #1747

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment