Construct SGPT

[tsamarahrana]

Hello, I am currently working on my project and I'm interested on your paper.

First of all, is it correct that SGPT is based on GPT-Neo? If yes, it is possible to construct SGPT that is based on GPT-2?
How to construct it from scratch?

Thank you

[Muennighoff]

Hey, yes that's correct.

If you want to create SGPT as a Bi-Encoder for embeddings (SGPT-BE), then you need to finetune GPT-2 using the code in this repo.

If you want to create SGPT as a Cross-Encoder for reranking, then you just need to swap out GPT-Neo for GPT-2 in the inference script.

[tsamarahrana]

Thank you for the fast reply.

I want to create SGPT as Bi-Encoder. Would you please tell me which code to be exact?

[Muennighoff]

Thank you for the fast reply.

I want to create SGPT as Bi-Encoder. Would you please tell me which code to be exact?

Sure, there are insturctions in the README: https://github.com/Muennighoff/sgpt/tree/main/biencoder/nli_msmarco

Let me know if something is unclear!

[tsamarahrana]

Okay I’ll see how it goes, thank you for the guide!