Add HTTP recorder for evals; introduce --http-run flag

[nstankov-bg]

Change name

Introducing HTTP Recorder to OAIEvals Framework

Change description

This pull request introduces the HttpRecorder class to the evals framework, extending the current Eval mechanism.

HTTP Recorder

A new class HttpRecorder has been added to the evals.record module. This new recorder sends evaluation events directly to a specified HTTP endpoint using POST requests. The URL for this endpoint can be specified using the --http-run-url command-line argument when running the evaluations. In addition to the local and dry run modes, we now have an HTTP run mode that can be triggered using the --http-run flag.

Motivation

This change was largely motivated by the development of the OAIEvals Collector. As the creator of this Go application designed specifically for collecting and storing raw evaluation metrics, I saw the need for an HTTP endpoint in our evaluation mechanism. The OAIEvals Collector provides an HTTP handler designed for evals, thus making it an ideal recipient for the data recorded by the new HttpRecorder.

Allowing for more types of exporters, such as the new HttpRecorder, will likely increase the adoption of testing. The flexibility and ease-of-use of the HTTP Recorder allows for more developers to easily integrate testing into their workflows. This could ultimately lead to higher quality code, faster debugging, and an overall more efficient development process.

In addition, the integration of the HttpRecorder and visualization tools like Grafana dramatically lowers the barrier of entry for developers seeking to visualize their evaluation. By providing a streamlined process for collecting and visualizing metrics, I aim to make data-driven development more accessible, leading to more informed decision-making and improved model quality.

Demo:

Elasticsearch x Kabana:

Elasticsearch Test
Basic Visualization via Kibana

InfluxDB:

InfluxDB Test 1	InfluxDB Test 2

Grafana:

Grafana Test
Basic Visualization via InfluxDB & TimeScaleDB
!

Kafka:

Kafka Test
Basic Visualization via Kafka UI
!

Testing

The new HTTP Recorder feature was thoroughly tested with the following commands:

• python3 oaieval.py gpt-3.5-turbo abstract-causal-reasoning-text.dev.v0 --max_samples=50 --dry-run
• python3 oaieval.py gpt-3.5-turbo abstract-causal-reasoning-text.dev.v0 --max_samples=50
• python3 oaieval.py gpt-3.5-turbo abstract-causal-reasoning-text.dev.v0 --max_samples=20 --http-run --http-run-url=http://localhost:8081/events --http-batch-size=10
• python3 oaieval.py gpt-3.5-turbo abstract-causal-reasoning-text.dev.v0 --max_samples=20 --http-run --http-run-url=http://localhost:8081/events

Each command provided expected results, demonstrating the feature's proper functionality.

Criteria for a good eval ✅

The introduced changes enhance the project's functionality and do not break any existing features. The changes pass all the existing tests and follow the project's contribution guidelines.

Eval structure 🏗️

• The changes are located in evals/cli/oaieval.py for the new HTTP run mode and evals.record for the new HttpRecorder.
• I ensure I have the right to use the data I submit via this eval.

Final checklist 👀

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.
• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.
• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.
• I have filled out all required fields of this form.
• I have used Git LFS for the Eval JSON data.

Eval JSON data

Due to the nature of this PR, there are no new eval samples.

[bclarkicc]

This is a ridiculously helpful change for a project our org is currently working on, and I sincerely hope that it gets merged!

[dbergunder]

This will prevent us from maintaining a fork. 👍🏻

[jwang47]

One scenario we might also want to handle is if there's an uncaught error that happens during the eval run. We might want to send an event indicating that the eval failed in that situation.

[nstankov-bg]

Hey @jwang47,

One scenario we might also want to handle is if there's an uncaught error that happens during the eval run. We might want to send an event indicating that the eval failed in that situation.

Regarding the handling of errors during the evaluation run, we can represent a failed eval event with a structure similar to a successful event. The Type field would differ, and the Data field would carry additional error-specific information.

For example, an event for a failed evaluation could look like this:

{
  "run_id": "2307080128125Q6U7IFP",
  "event_id": 4,
  "sample_id": "abstract-causal-reasoning-text.dev.484",
  "type": "eval_failed",
  "data": {
    "error": {
      "type": "service_unavailable",
      "code": 503,
      "message": "Service Unavailable",
      "details": "The upstream service was overloaded."
    }
  },
  "created_by": "",
  "created_at": "2023-07-08 01:28:13.704853+00:00"
}

Considering that a successful one looks like this:

{
  "run_id": "2307080128125Q6U7IFP",
  "event_id": 3,
  "sample_id": "abstract-causal-reasoning-text.dev.484",
  "type": "match",
  "data": {
    "correct": true,
    "expected": "ANSWER: off",
    "picked": "ANSWER: off",
    "sampled": "undetermined."
  },
  "created_by": "",
  "created_at": "2023-07-08 01:28:13.704853+00:00"
}

In this structure, the Type field is set to "eval_failed" to indicate that this is an error event. Within the Data field, we have an error object that contains detailed information about the error, such as its type, code, message, and additional details. This allows us to provide comprehensive information about any errors that occur, facilitating easier troubleshooting.

Can this be added in another PR, as the complexities of error handling on such level can get rather complex, and I would love to tackle this with the focused attention it needs.

[jwang47]

Can this be added in another PR, as the complexities of error handling on such level can get rather complex, and I would love to tackle this with the focused attention it needs.

Sounds good, let's add in a later PR.

[jwang47]

Looks good, thanks for the contribution!