Commit 1ecde1c
feat: implement baml-cli test (#1458)
NOT READY FOR MERGE YET: projected land date Feb 21
Design notes:
```bash
# past work
baml-cli test run --include "MyFunction::" --include "AnyMatch" --exclude "::LongTest"
# syntax: <part>::<part>
# syntax: "*\w\d_" for function/test filter clauses
baml-cli test list # this is the default action because it costs $$$
baml-cli test run # to run tests
baml-cli test --output-formats [pretty | github | junit=path/to/file.xml]
```
Product requirements:
- allow users to enforce that baml tests pass in CI
- users may only want to run a subset of tests in CI
- filter based on file & name (in the future, maybe some kind of “test
tag” attribute)
- `--include`
- `--exclude`
- tests for a given function
- regex
- `--verbose` - TBD
- report timing stats - TBD
- tests may be flakey and need re-running - TBD
- for github actions: we can use [[output
groups](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines)](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines)
- for non-github aka generic integration (TBD) junit xml is a must
- cf [gitlab unit test
reports](https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html)
- debug loop: what are users going to do when tests fail?
- test failures need to show logs
- re-run a specific test
- can go into the wasm playground for this
- happy-ish path
- show # of tests that passed, timing info, # of tests skipped - take
`cargo test` as inspiration (maybe pytest/mocha too)
- test execution will be concurrent
- do we need concurrency limits?
- need to be able to set concurrency down to 1 (claude basic tier),
default concurrency should be... ???
- definitely need timeouts too
Fixes #1345
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> Adds `baml-cli test` command for running BAML tests with filtering,
output formats, and concurrency options, along with `.env` file support
and test execution integration with GitHub Actions and JUnit XML
reporting.
>
> - **CLI**:
> - Adds `baml-cli test` command in `commands.rs` to run BAML tests with
options for filtering (`--include`, `--exclude`), output formats
(`--output-format`), and concurrency (`--parallel`).
> - Integrates test execution with GitHub Actions and JUnit XML
reporting in `output_github.rs` and `output_junit.rs`.
> - **Environment**:
> - Implements `.env` file loading in `dotenv/mod.rs` with support for
multiline strings, variable interpolation, and escape sequences.
> - Adds tests for `.env` loading in `dotenv/tests.rs`.
> - **Test Execution**:
> - Introduces `TestExecutor` trait in `test_executor/mod.rs` for
running tests with status tracking and rendering.
> - Implements `PrettyTestExecutionStatusRenderer`,
`GithubTestExecutionStatusRenderer`, and `JUnitXMLRenderer` for
different output formats.
> - Adds `TestFilter` in `test_execution_args.rs` for filtering tests
based on patterns.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup>
for 01e78d6. It will automatically
update as commits are pushed.</sup>
<!-- ELLIPSIS_HIDDEN -->
---------
Co-authored-by: Vaibhav Gupta <vbv@boundaryml.com>1 parent 8ea9ed8 commit 1ecde1c
File tree
23 files changed
+2307
-126
lines changed- .github/workflows
- engine
- baml-lib/baml-core/src/ir
- baml-runtime
- src
- cli
- dotenv
- test_executor
- types
- cli
- src
- language_client_python
- python_src/baml_py
- src
- language_client_ruby/ext/ruby_ffi/src
23 files changed
+2307
-126
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
0 commit comments