Laminar evaluations

[Rainhunter13]

This PR integrates Laminar evaluations into Openhands Benchmarks, enabling dynamic creation of Laminar Evaluations and Datapoints per every run.

Core changes:

• Added singleton LaminarService managing Laminar client, evaluations, and datapoints
• For every benchmark run, create a Laminar evaluation.
• For every instance run, create an evaluation datapoint, create execution span and propagate it to isolated workspaces via environment variable LMNR_SPAN_CONTEXT
• Save Laminar metadata (eval/datapoints ids) to be able to update scores upon inference evaluation.
• Update evaluation datapoints with scores based on inference evaluation output.

Usage:

export LMNR_PROJECT_API_KEY=your_key_here
uv run swebench-infer ...

[simonrosenberg]

Can you solve the 2 mutables?
Then there is still an issue because the eval output format is really suboptimal but this can be solved in the standardization tickef!

[simonrosenberg]

Actually can we not merge this until I standardize the evaluation output format? sorry! Should be done by tomorrow

Issue here #177

[Rainhunter13]

@simonrosenberg addressed the comments!

sure, let me know once you standardise the format, I'll adjust laminar service as needed, thank you

[simonrosenberg]

@OpenHands please resolve the conflicts

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

I've resolved the merge conflicts in PR #175 by merging origin/main into the laminar_evaluations branch. Here's a summary of the conflict resolutions:

Conflicts Resolved

1. benchmarks/commit0/run_infer.py:

   • Kept the remote workspace support from main (adds APIRemoteWorkspace support alongside DockerDevWorkspace)
   • Preserved the forward_env parameter for DockerDevWorkspace from the laminar_evaluations branch
   • Note: APIRemoteWorkspace doesn't support forward_env parameter, so it was only added to DockerDevWorkspace

2. pyproject.toml:

   • Combined dependencies from both branches
   • Kept lmnr>=0.7.24 (the higher version from laminar_evaluations)
   • Added all new dependencies from main (deprecation, fastmcp, httpx, litellm, pydantic, etc.)
   • Added new script entrypoints from main (multiswebench, swebenchmultimodal)

3. uv.lock:

   • Regenerated the lock file using uv lock to reflect the merged dependencies

Important Note

The branch has been pushed to origin/laminar_evaluations on the OpenHands/benchmarks repository. However, PR #175 is from a fork (lmnr-ai/benchmarks), and I don't have write access to that fork.

The PR author (@Rainhunter13) will need to either:

1. Pull the changes from origin/laminar_evaluations into their fork's branch, or
2. The PR can be closed and a new one created from origin/laminar_evaluations

The resolved branch is available at: https://github.com/OpenHands/benchmarks/tree/laminar_evaluations

_{View full conversation}

[simonrosenberg]

@Rainhunter13 approved, thanks! and sorry for the delayed process

[Rainhunter13]

@simonrosenberg thanks! Just fixed the tests, do you think it will possible to merge this soon?