Skip to content

feat: add ExperimentContext and e2e examples#59

Merged
Quentin Brosse (QuentinBrosse) merged 11 commits intonextfrom
quentin/rel-562-examples-experiments
Nov 14, 2025
Merged

feat: add ExperimentContext and e2e examples#59
Quentin Brosse (QuentinBrosse) merged 11 commits intonextfrom
quentin/rel-562-examples-experiments

Conversation

@QuentinBrosse
Copy link
Copy Markdown
Collaborator

@QuentinBrosse Quentin Brosse (QuentinBrosse) commented Nov 12, 2025

  • RecordExperimentExample -> repro of langsmith-rest-experiments repo
  • E2eEvalExample -> e2e (real OpenAI agent -> dataset -> examples -> experiments -> LLM-as-a-juge)
  • Some improvements in wappers

Todo:

  • Rework ExperimentContext?
  • Full review

Close REL-562 REL-563

@QuentinBrosse Quentin Brosse (QuentinBrosse) changed the base branch from main to next November 12, 2025 15:39
@QuentinBrosse Quentin Brosse (QuentinBrosse) force-pushed the quentin/rel-562-examples-experiments branch from c8a65a9 to f7d307c Compare November 13, 2025 13:38
@QuentinBrosse Quentin Brosse (QuentinBrosse) marked this pull request as ready for review November 13, 2025 13:54
*/
public final class ExperimentContext {

private static final ThreadLocal<Context> CONTEXT = ThreadLocal.withInitial(Context::new);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make this class a singleton, rather than static.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with static because it follows the same pattern as OpenTelemetry's Context:

// OpenTelemetry does this (static):
Context.current().with(key, value).makeCurrent()

// Not this (singleton):
Context.getInstance().current().with(key, value).makeCurrent()

Do you see a specific benefit to the singleton pattern here that I'm missing?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but still refactored to match OpenTelemetry Context pattern

@QuentinBrosse Quentin Brosse (QuentinBrosse) changed the title doc: improve/fix DatasetExample feat: add ExperimentContext and e2e examples Nov 14, 2025
Refactor OpenTelemetry configuration and logging to improve usability and follow industry standards.

API Changes:
  - Remove `configureForLangSmith()` static methods in favor of Builder pattern
  - Remove `configureFromEnv()` - use `builder().build()` instead
  - Add required field validation in `Builder.build()` with clear error messages
  - Builder constructor auto-reads environment variables (LANGSMITH_API_KEY, etc.)
  - Builder methods override environment variable values
@QuentinBrosse Quentin Brosse (QuentinBrosse) merged commit ee23b97 into next Nov 14, 2025
6 checks passed
@ericdong-langchain ericdong-langchain deleted the quentin/rel-562-examples-experiments branch November 20, 2025 16:57
John Kennedy (jkennedyvz) added a commit that referenced this pull request Mar 31, 2026
…ansitive deps

Bumps Jackson 2.18.2→2.18.6 in published api deps and adds version
constraints for vulnerable transitive dependencies in test and example
scopes (Tomcat 9.0.115, Jetty 9.4.57, logback 1.2.13,
commons-fileupload 1.6.0, commons-io 2.14.0, json-smart 2.4.9,
snakeyaml 1.31, spring-web 5.3.34).

Addresses alerts: #75 (jackson-core), #82/#81/#80/#79/#78/#72/#70/#69
/#68/#66/#65/#60/#59/#54/#52/#51/#48/#46/#45/#44/#43/#35/#34/#29/#31
(tomcat/jetty/logback/commons).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Jacob Lee (jacoblee93) pushed a commit that referenced this pull request Mar 31, 2026
…ansitive deps (#119)

Bumps Jackson 2.18.2→2.18.6 in published api deps and adds version
constraints for vulnerable transitive dependencies in test and example
scopes (Tomcat 9.0.115, Jetty 9.4.57, logback 1.2.13,
commons-fileupload 1.6.0, commons-io 2.14.0, json-smart 2.4.9,
snakeyaml 1.31, spring-web 5.3.34).

Addresses alerts: #75 (jackson-core), #82/#81/#80/#79/#78/#72/#70/#69
/#68/#66/#65/#60/#59/#54/#52/#51/#48/#46/#45/#44/#43/#35/#34/#29/#31
(tomcat/jetty/logback/commons).

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Jacob Lee (jacoblee93) added a commit that referenced this pull request Mar 31, 2026
* Revert "main to next (#108)" (#110)

This reverts commit 5a8988d.

* Fix/lint and remove method count (#111)

* release: 0.1.0-alpha.24

* fix: lint and remove outdated method count ci test

Made-with: Cursor

---------

Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>

* feat: Adds package version resource to build (#100)

* codegen metadata

* codegen metadata

* chore: make `Properties` more resilient to `null`

* chore: drop apache dependency

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): expand imports

* feat(api): manual updates

* codegen metadata

* feat(api): manual updates

* codegen metadata

* feat(api): manual updates

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* feat(api): manual updates

removed endpoints not in openAPI spec

* codegen metadata

* codegen metadata

* codegen metadata

* chore: align user agent format (#96)

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* chore(internal): codegen related update

* chore(internal): bump palantir-java-format

* chore(ci): skip uploading artifacts on stainless-internal branches

* chore: update placeholder string

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* fix(client): incorrect `Retry-After` parsing

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* chore(internal): tweak CI branches

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): update retry delay tests

* fix(client): allow updating header/query affecting fields in `toBuilder()`

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* chore(internal): bump ktfmt

* chore: remove old test (#97)

* codegen metadata

* feat(api): manual updates

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): update gitignore

* codegen metadata

* codegen metadata

* codegen metadata

* chore(ci): skip lint on metadata-only changes

Note that we still want to run tests, as these depend on the metadata.

* Fix error messages (#102)

* codegen metadata

* Merge

* Merge

* Lint

* Add debug log

* main to next (#108)

* codegen metadata

* codegen metadata

* chore: make `Properties` more resilient to `null`

* chore: drop apache dependency

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): expand imports

* feat(api): manual updates

* codegen metadata

* feat(api): manual updates

* codegen metadata

* feat(api): manual updates

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* feat(api): manual updates

removed endpoints not in openAPI spec

* codegen metadata

* codegen metadata

* codegen metadata

* chore: align user agent format (#96)

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* chore(internal): codegen related update

* chore(internal): bump palantir-java-format

* chore(ci): skip uploading artifacts on stainless-internal branches

* chore: update placeholder string

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* fix(client): incorrect `Retry-After` parsing

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* chore(internal): tweak CI branches

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): update retry delay tests

* fix(client): allow updating header/query affecting fields in `toBuilder()`

* codegen metadata

* codegen metadata

* feat(api): api update

* feat(api): api update

* codegen metadata

* chore(internal): bump ktfmt

* chore: remove old test (#97)

* codegen metadata

* feat(api): manual updates

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* chore(internal): update gitignore

* codegen metadata

* codegen metadata

* codegen metadata

* chore(ci): skip lint on metadata-only changes

Note that we still want to run tests, as these depend on the metadata.

* Fix error messages (#102)

* codegen metadata

* release: 0.1.0-alpha.24

* fix: lint and remove outdated method count ci test (#103)

Made-with: Cursor

* chore(deps): bump gradle/actions from 5 to 6 in the all-actions group (#99)

Bumps the all-actions group with 1 update: [gradle/actions](https://github.com/gradle/actions).


Updates `gradle/actions` from 5 to 6
- [Release notes](https://github.com/gradle/actions/releases)
- [Commits](gradle/actions@v5...v6)

---
updated-dependencies:
- dependency-name: gradle/actions
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>
Co-authored-by: Jacob Lee <jacoblee93@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Change to warning and add a test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>
Co-authored-by: ericdong-langchain <ericdong@langchain.dev>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Adds contributing guide (#112)

* ci: add minimum workflow permissions (#106)

- Add top-level `permissions: contents: read` to all 4 workflow files
- Change release-doctor.yml trigger from `pull_request` to `pull_request_target`
  to prevent secret exfiltration via PR-controlled workflow modifications

Co-authored-by: Posture Fix <posture-fix@langchain.ai>

* ci: SHA-pin third-party Gradle actions (#107)

Pin gradle/actions/setup-gradle and gradle/gradle-build-action to full
commit SHAs to prevent supply chain attacks via tag hijacking.

- gradle/actions/setup-gradle@v6  → @205054a...  (ci.yml ×2, codeql.yml)
- gradle/gradle-build-action@v3   → @12318b0...  (ci.yml, publish-sonatype.yml)

Co-authored-by: Posture Fix <posture-fix@langchain.ai>

* chore: fix dependabot.yml posture issues (#105)

* chore: update dependabot.yml to comply with posture checks

- Change schedule from weekly to monthly for all ecosystems
- Add update-types split (major vs minor-and-patch) for gradle and github-actions
- Add docker ecosystem entry for .devcontainer/Dockerfile

* chore: add target-branch next and fix docker group split for Stainless posture compliance

---------

Co-authored-by: Posture Fix <posture-fix@langchain.ai>

* feat(api): api update

* feat(api): api update

* feat: Add `traceable` function wrapper for LangSmith tracing (#101)

* Adds versioning resource to build

* Adds initial version of traceable

* Lint

* Deflake

* Progress

* Progress

* Fixes

* Fixes

* More refactor

* Small bug

* Refactor

* Fix

* Devin feedback

* Tests and feedback

* Remove redundant comment

* Docstring

* codegen metadata

* codegen metadata

* feat: Adds processInputs and processOutputs to traceable (#113)

* Adds processInputs and processOutputs to traceable

* Move generics into TraceProcessIO to avoid having them top level

* Fix docstring, nit

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* codegen metadata

* codegen metadata

* codegen metadata

* feat(api): api update

* codegen metadata

* feat(api): manual updates

* codegen metadata

* fix: Fix format detection when pulling legacy prompts (#115)

* Fix format detection when pulling legacy prompts

* Feedback

* Update AGENTS.md (#116)

* feat: Rework wrapOpenAI (#114)

* Update wrapOpenAI

* Remove

* Fixes

* nits

* Feedback

* Cache

* Remove comment

* Fix

* feat(api): api update

* feat(api): api update

* fix: patch security alerts — bump Jackson and constrain vulnerable transitive deps (#119)

Bumps Jackson 2.18.2→2.18.6 in published api deps and adds version
constraints for vulnerable transitive dependencies in test and example
scopes (Tomcat 9.0.115, Jetty 9.4.57, logback 1.2.13,
commons-fileupload 1.6.0, commons-io 2.14.0, json-smart 2.4.9,
snakeyaml 1.31, spring-web 5.3.34).

Addresses alerts: #75 (jackson-core), #82/#81/#80/#79/#78/#72/#70/#69
/#68/#66/#65/#60/#59/#54/#52/#51/#48/#46/#45/#44/#43/#35/#34/#29/#31
(tomcat/jetty/logback/commons).

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* codegen metadata

* codegen metadata

* feat: Adds support for tracing streams with traceable (#117)

* Adds support for tracing streams with traceable

* Polish

* Make stream tracing opt-in

* Rework to use a passthrough instead of a proxy

* Record stream cancellations as errors

* Feedback

* Format and add to AGENTS.md

* feat: Adds streaming support for wrapOpenAI (#118)

* Adds support for tracing streams with traceable

* Polish

* Make stream tracing opt-in

* Rework to use a passthrough instead of a proxy

* Adds streaming support for wrapOpenAI

* Record stream cancellations as errors

* Adds streaming example

* Allow empty config default, add example

* Fix

* release: 0.1.0-alpha.25

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: ericdong-langchain <ericdong@langchain.dev>
Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>
Co-authored-by: Jacob Lee <jacoblee93@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: John Kennedy <65985482+jkennedyvz@users.noreply.github.com>
Co-authored-by: Posture Fix <posture-fix@langchain.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants