Our CI pipeline should trigger the benchmark pipeline

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

The ci.yml file runs our tests without api keys. If someone changed the prompt and an API key is needed, we want to run our benchmark pipeline.

Step 1: mark all the tests that require an API key (with the pytest mark decorator)

Step 2: add a pytest plugin that will inform us if any of these marked tests has failed:

Step 3: update the workflow.yaml in order to use the plugin when running pytest. If tests marked as requiring an api key succeed, it means no prompt change happened. The workflow passes. Otherwise, trigger the benchmark workflow and make the current workflow fail.

Step 4: create a new workflow called benchmarks. This benchmark will systematically fail for now.

Examples 🌈

No response

Motivation 🔦

No response