Success makers: telemetry

[TyDunn]

Definition of done

• telemetry decided
• telemetry implemented

[usc-m]

Some ideas on metrics:

• command executed (extract/stats/both)
• number of markers defined
• some measure of marker complexity (minimum/maximum conditions per marker? some sort of bucketing by number of conditions per marker?)
• what strategies are used to pull trackers
• if people use the seed option if they're using the sample strategy

[alwx]

Some additions to ideas:

• number of markers per operator (might give an idea of which operators are used the most, but can also be included to the "marker complexity" metric proposed above)
• marker type (diagnostic/a partial success marker/a full success marker/...) — however, might be not always easy to determine

[alwx]

My suggestion for what needs to be tracked:

• rasa evaluate markers CLI command executed
  • if it's just extract, produce stats or both
  • the strategy that will beused to select trackers for marker extraction (all/first_n/sample)
  • the number of trackers to extract markers from
  • seed to use if selecting trackers by 'sample'
• the output of marker extraction has been saved
  • number of markers extracted
• the stats has been produced
  • number of markers read