EventRecorder changes to support evaluators and reasoning strings

[nv-alicheng]

What does this PR do?

• Moves output logging to EventDB from output JSONL file
• Extends output logging functionality to allow tracking reasoning sequences for Issuers+Endpoints that support it
• Adds STOP_PERFORMANCE_TRACKING event to mark a timestamp which removes any samples issued after it from performance metrics calculation

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Summary of Changes

Hello @nv-alicheng, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the event recording and reporting system to improve the robustness and utility of performance and accuracy metrics. By transitioning from a simple JSONL file to a SQLite database for event storage, the system can now capture more detailed information, including reasoning sequences. Furthermore, the introduction of a dedicated event to stop performance tracking allows for more accurate and controlled measurement of performance, ensuring that only relevant samples contribute to the final metrics.

Highlights

• Event Logging Refactor: Output logging has been moved from a flat JSONL file to a more structured SQLite EventDB, enhancing data integrity and query capabilities.
• Reasoning Sequence Tracking: The event logging functionality is extended to support tracking reasoning sequences for Issuers and Endpoints that provide this information, allowing for richer evaluation data.
• Performance Tracking Control: A new STOP_PERFORMANCE_TRACKING event has been introduced. This event marks a timestamp after which any issued samples will be excluded from performance metrics calculations, providing more precise control over performance evaluation windows.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors the event recording system to use a SQLite database (EventDB) instead of a JSONL file for logging outputs. This change supports more structured data, including 'reasoning' sequences for evaluators. A new STOP_PERFORMANCE_TRACKING event is introduced to allow for more precise performance measurement by excluding samples issued after a certain point. The changes are extensive, touching event generation, session management, and metrics reporting. My review focuses on the correctness and maintainability of these new implementations. I've found a critical bug in the session logic and a couple of medium-severity issues related to type hints and code duplication that should be addressed.

[arekay-nv]

Partial review. Will followup later.

[arekay-nv]

LGTM.