Support RunId or Experimentation API in a cross-process manner

[skylarbpayne]

Description

When setting up evals for an existing system, the reality is that the "AI pipeline" is often not "pure". It depends on a lot of external resources in the middle. This makes it hard to simply "extract" it out and run it as part of an experiment.

As such, it is common to have an external tool/script which triggers an endpoint to start the AI pipeline. Generally, this is the same endpoint that real users would trigger in the product. This script can then be pointed at a local development or even production environment to generate logs.

However, it is possible to run the same set of inputs against the same versioned function multiple times. It would be helpful to be able to annotate and compare these separately (i.e. analogous to an A/A test).

[willbakst]

I think there are two key pieces here:

1. Every span should have an optional Experiment or Run ID
2. There needs to be a way to then view / compare batches of spans by ID

This should cover future work on Experiments since it's ultimately the same underlying concept.

I imagine a context manager would be ideal here so the ID inclusion isn't part of the version of the traced function, something like:

with lilypad.Run():
    output = my_fn()

where my_fn's trace will automatically have a RunID attached.

If you need to pipe the ID through, you could do something like:

run_id: RunID = lilypad.Run.generate_id()
with lilypad.Run(id=run_id):
    output = my_fn()

What do we think?

[skylarbpayne]

Sounds good to me, with the one caveat that when I'm piping through the ID, there willl be cases where it's missing, so the behavior of with lilypad.Run(id=run_id) when run_id=None I would expect would have the same behavior of not using the context manager at all?

[willbakst]

Sounds good to me, with the one caveat that when I'm piping through the ID, there willl be cases where it's missing, so the behavior of with lilypad.Run(id=run_id) when run_id=None I would expect would have the same behavior of not using the context manager at all?

Yeah we should engineer it to work like this.

The context manager just sets the run context, and internally we would then grab that run context. So if the ID is None there's no ID to attach and we should handle that as though there's no run context.

Also realizing I mixed up my capitalization and lowercasing.

Should be:

• lilypad.Run class that encapsulates all the run information
• lilypad.run context manager that results in a lilypad.Run (i.e. with lilypad.run() as run: ... where run: Run)
• lilypad.Run.generate_id staticmethod on the Run class (distinctly separate from the context manager)