chore: Support custom scoring summary functions

[tssweeney]

Note to Reviews: Yes, this PR has many lines of change, but functionality-wise it is quite minimal. The vast majority of the changes are the result of changing the underlying data model to support different key structures for summaries and scores. I also took the opportunity to cleanup a lot of code from the first MVP, add A LOT of comments, and re-organize code decouple unrelated files. I also took this chance to rename various symbols more aptly.

Problem: When building the evaluation comparison page, I made an invalid assumption about the shape of scoring data and summary data. Namely: i thought that the shape of summary data was DEFINED as the shape of the score data, where each score "leaf" is split into a dictionary in the "summary" - and that dictionary had a regular form for boolean or floating values respectively. However this is ONLY true for metrics that are "autosummarized". It is perfectly possible for the user to define their own summary function. Jason did this here: https://wandb.ai/jzhao/resume-bot-eval/weave/compare-evaluations?evaluationCallIds=%5B%22fecc2462-10c1-4a7b-a0b9-4e1726e5618d%22%2C%224bc16671-95b6-4044-94d4-b34417b44868%22%5D where he defined a score as {"correct": boolean}, but summarized as {"f1: float, "precision": float, "recall" float}. This is perfectly possible with our Evaluation framework, but was not supported in the UI.

Solution: The major change here is to decouple our metric definitions. Specifically, our data model currently has scorerMetricDimensions: {[metricDimensionId: string]: ScorerMetricDimension};, but this is not sufficient to describe the world. Now we have two fields:

  // ScoreMetrics define the metrics that are associated on each individual prediction
  scoreMetrics: MetricDefinitionMap;

  // SummaryMetrics define the metrics that are associated with the evaluation as a whole
  // often aggregated from the scoreMetrics.
  summaryMetrics: MetricDefinitionMap;

Pretty much all the remaining changes are supporting the model change and making the score handling more robust and centralized. I am also being a lot more strict with "Metric" referring to a general value, "SummaryMetric" being a metric summarizing the eval, and "ScoreMetric" being a metric that relates to a specific input/output pair.

A few Before and Afters:

**Tests: ** I added a big unit test to assert the correct shapes of data assumed by the UI to protect against breakage in the future.

Here are a bunch of good examples for all sorts of eval situations for manual testing:

• Unit Test

  • https://app.wandb.test/timssweeney/dev_testing/weave/compare-evaluations?evaluationCallIds=%5B%22b473c999-0468-4692-a03c-4e5120112a3b%22%2C%222c8b8f41-b249-44b4-b2dc-ada94987dd7c%22%5D

• Anish

  • https://app.wandb.test/a-sh0ts/arxiv-papers-anthropic-testv6-8/weave/compare-evaluations?evaluationCallIds=%5B%224d26e233-37ab-4134-8985-f3ed58bb5c73%22%2C%22c99bca2c-3798-4b24-b5e6-75b245f3a0c8%22%5D

• Lavanya

  • https://app.wandb.test/lavanyashukla/nims_rag_2/weave/compare-evaluations?evaluationCallIds=%5B%227ca7836f-e055-42eb-94c5-1625b73af789%22%2C%221024b6c8-6707-4e61-9474-463ad424a067%22%5D
  • https://app.wandb.test/lavanyashukla/nims_rag_2/weave/compare-evaluations?evaluationCallIds=%5B%22fad0e96b-e9e3-4a84-8ce0-ad61dce5987e%22%2C%221024b6c8-6707-4e61-9474-463ad424a067%22%2C%227ca7836f-e055-42eb-94c5-1625b73af789%22%5D
  • https://app.wandb.test/lavanyashukla/nims_rag_2/weave/compare-evaluations?evaluationCallIds=%5B%223fd22e88-84a6-4f66-ba71-2f789d44aded%22%2C%227ca7836f-e055-42eb-94c5-1625b73af789%22%2C%221024b6c8-6707-4e61-9474-463ad424a067%22%2C%22d6358605-5c1d-4561-819d-179bb433a7a7%22%5D

• Jason

  • https://app.wandb.test/jzhao/resume-bot-eval/weave/compare-evaluations?evaluationCallIds=%5B%22fecc2462-10c1-4a7b-a0b9-4e1726e5618d%22%2C%224bc16671-95b6-4044-94d4-b34417b44868%22%5D

• Shawn

  • https://app.wandb.test/shawn/humaneval6/weave/compare-evaluations?evaluationCallIds=%5B%2258c9db2c-c1f8-4643-a79d-7a13c55fbc72%22%5D
  • https://app.wandb.test/shawn/humaneval6/weave/compare-evaluations?evaluationCallIds=%5B%2258c9db2c-c1f8-4643-a79d-7a13c55fbc72%22%2C%228563f89b-07e8-4042-9417-e22b4257bf95%22%5D
  • https://app.wandb.test/shawn/humaneval6/weave/compare-evaluations?evaluationCallIds=%5B%2258c9db2c-c1f8-4643-a79d-7a13c55fbc72%22%2C%228563f89b-07e8-4042-9417-e22b4257bf95%22%2C%2232f3e6bc-5488-4dd4-b9c4-801929f2c541%22%2C%2234c0a20f-657f-407e-bb33-277abbb9997f%22%5D

• Adam

  • https://app.wandb.test/wandb-designers/signal-maven/weave/compare-evaluations?evaluationCallIds=%5B%22eb4a3bed-5e67-4caf-a911-db4705f5254e%22%5D
  • https://app.wandb.test/wandb-designers/signal-maven/weave/compare-evaluations?evaluationCallIds=%5B%22eb4a3bed-5e67-4caf-a911-db4705f5254e%22%2C%22c4345512-ef52-4112-a941-83fd6dac779f%22%5D
  • https://app.wandb.test/wandb-designers/signal-maven/weave/compare-evaluations?evaluationCallIds=%5B%22eb4a3bed-5e67-4caf-a911-db4705f5254e%22%2C%22c4345512-ef52-4112-a941-83fd6dac779f%22%2C%22bf5188ba-48cd-4c6d-91ea-e25464570c13%22%2C%228d76bb68-837e-4fff-8159-489958fd1f65%22%5D

• Good Manual Tests:

  • Loads
  • Charts look good
  • Scorecard looks good
  • Scorecard links work
  • Filters are changeable and selectable
  • Filtered data is correct
  • SummaryScorers link correctly (both single and mixed)
  • MetricScorers link correctly (both single and mixed)
  • MetricValues link to scoring call correctly

[jamie-rasmussen]

I put a moveToFront in EmojiDetails, maybe they can join forces.