Evaluation metrics weights and normalization

[fohrloop]

The evaluation metrics, for example config/evaluation/default.yml has a metrics block. This block has a list of different metrics. For example:

  # Each finger's load shall be relative to the specified weights
  finger_balance:
    enabled: true
    weight: 50.0
    normalization:
      type: fixed
      value: 1.0
    params:
      # Intended factors for finger load. Thumb values are ignored.
      intended_loads:
        [Left, Pinky]: 1.0
        [Left, Ring]: 1.6
        [Left, Middle]: 2.0
        [Left, Index]: 2.0
        [Left, Thumb]: 2.0
        [Right, Thumb]: 2.0
        [Right, Index]: 2.0
        [Right, Middle]: 2.0
        [Right, Ring]: 1.6
        [Right, Pinky]: 1.0

  # Each hand's load shall be close to 50%
  hand_disbalance:
    enabled: true
    weight: 40.0
    normalization:
      type: fixed
      value: 1.0
    params:
      null: null

  # Each keystroke incurs a cost (defined in the keyboard's layout config)
  key_costs:
    enabled: true
    weight: 20.0
    normalization:
      type: weight_found
      value: 1.0
    params:
      null: null

Could you explain how the final "cost" score is calculated from these? My first assumption is:

final_cost = weight1*metric1 + weight2*metric2 + .... weightN*metricN

is that correct? The second question is about the normalization block. It has two keys

• type: fixed | weight_found | weight_all
• value: float (seems to be 1.0 or 0.01)

How do the normalization parameters affect the final score?

[dariogoetz]

Your assumption is correct. The final score is the weighted sum of the individual metric scores just as in your formula.

The cost for most metrics is generated by summing up all costs associated with individual ngrams. In that case, one tries to factor out the total weight in the corpus (the total number of occurrences of all ngrams) and normalize the result to have a value "per ngram". If you chose weight_all or weight_found, the original metric value is divided by the total weight of all ngrams / all ngrams that can be typed with the layout (i.e. do not contain unknown symbols). If fixed is chosen, the total score is left untouched.

In your example above, the hand_dispalance metric itself already computes percentages, so it would not make sense to divide by the number of ngrams (again). This is basically already accounted for in the metric implementation itself.

The normalization type value is just an additional factor. It could be left out, because the weight basically does the same thing. It could be used to first normalize to something like percentages from fractions (or the other way around) before using the metric weight or so.

[fohrloop]

Perfect, thanks a lot! So to summarize, the final cost is calculated as:

final_cost = weight1*normalized_metric1 + weight2*normalized_metric2 + .... weightN*normalized_metricN

each metric is calculated as:

normalized_metric =  value*(metric/N)

where:

• value is the value from the normalization dictionary from the evaluation configuration
• metric is the calculated metric, as output by the chosen metric function (usually from the same name .rs file as the metric name)
• N depends on the normalization.type, and is
  • 1 if type is fixed
  • total number of ngrams (1-grams, 2-grams or 3-grams depending on which type of metric it is), if type is weight_all
  • the number of ngrams which may be typed with the layout, if the type is weight_found

Is that correct?

[dariogoetz]

Yes 👍

[fohrloop]

Thank you again! I marked my comment as the answer as it summarizes the logic, although you did provide the answer (just to make it easier to find. I hope you don't mind).