What content is encoded when Llama13B encoded a sentence

[cgpeter96]

Only prompt? or prompt+response?

[VPeterV]

We have integrated the prompt within our code. If you want to use our scorers, you only need to give your query sentence for scoring complexity or query + response for scoring quality.

Our code will convert your input (i.e. query or query + response) to our designed format for scorers to score samples.

[edbeeching]

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

[VPeterV]

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

Hi @edbeeching ! We encode the sentence by using the hidden states of the last token for one sentence (i.e. the last token in query + response)