clarification on annotation entries from alpaca_eval

[xwinxu]

I wanted to clarify the notation in the alpaca_eval command. Let's say I passed in --model_outputs my_model_A.json and
--reference_outputs my_model_B.json. The resulting annotations.json will contain keys output_1, output_2, and rankings under raw_completions:

    "raw_completion":{
      "concise_explanation":"....",
      "ordered_models":[
        {
          "model":"M",
          "rank":1
        },
        {
          "model":"m",
          "rank":2
        }
      ]
    }

According to this template, m corresponds to output_1 and M corresponds to output_2, correct? Then would this mean the model_outputs corresponds to m as well? I'm not sure which one takes precedence.

Thanks!

[rtaori]

Do you see a field referenced_models alongside? It should look something like this:
"referenced_models": { "M": "output_1", "m": "output_2" }
which should be easier to understand / hopefully resolves the confusion

[YannDubs]

@xwinxu check out the readme, the rank tells you which model was preferred so if preference = 2 (i.e. output_2 is preferred) and M has rank 1 (i.e. it's preferred) then you would know that M corresponds to output_2. The reason why all of this is a bit complicated is because we randomize the order of the outputs when evaluating.

I'll add the field referenced_models that Rohan is referring to, which currently is only added for sample sheets so you won't see it locally.

Note that your reference model will always be output_1.

[YannDubs]

Done @xwinxu , to add referenced_models you need to (1) update alpaca_eval, and (2) rerun the parsing by using the flag --is_reapply_parsing True. Note that this won't rerun the actual OAI annotations, which are cached.

[xwinxu]

Thanks Yann!