implement plugin model for ResultSink#142
Merged
Merged
Conversation
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
There was a problem hiding this comment.
Pull request overview
Implements a plugin model for ResultSink so third-party sink kinds can be installed alongside agentevals and resolved at runtime via setuptools entry points (with an additional programmatic override registry).
Changes:
- Added sink factory registry + entry-point discovery to
build_sinks()(with precedence: built-ins → entry points (non-shadowing) → programmatic overrides). - Added diagnostics helpers (
registered_sink_kinds,log_registered_sinks) and logs available sinks on API startup. - Added an example installable sink plugin (
examples/custom_sink) and expanded sink-related tests/docs.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/agentevals/run/sinks.py |
Adds plugin entry-point loading, programmatic factory registration, and sink-kind diagnostics. |
tests/run/test_sinks.py |
Adds coverage for registry behavior, entry-point resolution, collision rules, and failure handling. |
src/agentevals/api/app.py |
Logs registered sink kinds during API lifespan startup. |
src/agentevals/run/__init__.py |
Updates module docstring to reflect plugin-capable sinks. |
examples/README.md |
Documents custom result sinks and points to the example plugin. |
examples/custom_sink/README.md |
Provides instructions and configuration examples for a sink plugin. |
examples/custom_sink/pyproject.toml |
Defines an installable example package with agentevals.sinks entry point. |
examples/custom_sink/agentevals_example_custom_sink/sink.py |
Implements the example demo_ndjson sink and its entry-point factory. |
examples/custom_sink/agentevals_example_custom_sink/__init__.py |
Adds package initializer for the example plugin. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+253
to
+257
| def _merge_sink_factories() -> dict[str, SinkFactory]: | ||
| """Built-ins, then entry points (no built-in shadowing), then programmatic overrides.""" | ||
| merged: dict[str, SinkFactory] = dict(_builtin_factories()) | ||
| eps = entry_points(group=SINK_ENTRY_POINT_GROUP) | ||
| for ep in eps: |
Comment on lines
+254
to
+266
| """Built-ins, then entry points (no built-in shadowing), then programmatic overrides.""" | ||
| merged: dict[str, SinkFactory] = dict(_builtin_factories()) | ||
| eps = entry_points(group=SINK_ENTRY_POINT_GROUP) | ||
| for ep in eps: | ||
| if ep.name in merged: | ||
| logger.debug("skipping sink entry point %r; built-in kind takes precedence", ep.name) | ||
| continue | ||
| try: | ||
| loaded = ep.load() | ||
| if not callable(loaded): | ||
| logger.warning("sink entry point %r is not callable; skipping", ep.name) | ||
| continue | ||
| merged[ep.name] = cast(SinkFactory, loaded) |
krisztianfekete
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
implement the plugin model for ResultSinks -- you can install the sink together with the agentevals CLI and then reference it when doing runs.