Skip to content

Commit 1ecde1c

Browse files
sxlijinhellovai
andauthored
feat: implement baml-cli test (#1458)
NOT READY FOR MERGE YET: projected land date Feb 21 Design notes: ```bash # past work baml-cli test run --include "MyFunction::" --include "AnyMatch" --exclude "::LongTest" # syntax: <part>::<part> # syntax: "*\w\d_" for function/test filter clauses baml-cli test list # this is the default action because it costs $$$ baml-cli test run # to run tests baml-cli test --output-formats [pretty | github | junit=path/to/file.xml] ``` Product requirements: - allow users to enforce that baml tests pass in CI - users may only want to run a subset of tests in CI - filter based on file & name (in the future, maybe some kind of “test tag” attribute) - `--include` - `--exclude` - tests for a given function - regex - `--verbose` - TBD - report timing stats - TBD - tests may be flakey and need re-running - TBD - for github actions: we can use [[output groups](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines)](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines) - for non-github aka generic integration (TBD) junit xml is a must - cf [gitlab unit test reports](https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html) - debug loop: what are users going to do when tests fail? - test failures need to show logs - re-run a specific test - can go into the wasm playground for this - happy-ish path - show # of tests that passed, timing info, # of tests skipped - take `cargo test` as inspiration (maybe pytest/mocha too) - test execution will be concurrent - do we need concurrency limits? - need to be able to set concurrency down to 1 (claude basic tier), default concurrency should be... ??? - definitely need timeouts too Fixes #1345 <!-- ELLIPSIS_HIDDEN --> ---- > [!IMPORTANT] > Adds `baml-cli test` command for running BAML tests with filtering, output formats, and concurrency options, along with `.env` file support and test execution integration with GitHub Actions and JUnit XML reporting. > > - **CLI**: > - Adds `baml-cli test` command in `commands.rs` to run BAML tests with options for filtering (`--include`, `--exclude`), output formats (`--output-format`), and concurrency (`--parallel`). > - Integrates test execution with GitHub Actions and JUnit XML reporting in `output_github.rs` and `output_junit.rs`. > - **Environment**: > - Implements `.env` file loading in `dotenv/mod.rs` with support for multiline strings, variable interpolation, and escape sequences. > - Adds tests for `.env` loading in `dotenv/tests.rs`. > - **Test Execution**: > - Introduces `TestExecutor` trait in `test_executor/mod.rs` for running tests with status tracking and rendering. > - Implements `PrettyTestExecutionStatusRenderer`, `GithubTestExecutionStatusRenderer`, and `JUnitXMLRenderer` for different output formats. > - Adds `TestFilter` in `test_execution_args.rs` for filtering tests based on patterns. > > <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup> for 01e78d6. It will automatically update as commits are pushed.</sup> <!-- ELLIPSIS_HIDDEN --> --------- Co-authored-by: Vaibhav Gupta <vbv@boundaryml.com>
1 parent 8ea9ed8 commit 1ecde1c

File tree

23 files changed

+2307
-126
lines changed

23 files changed

+2307
-126
lines changed

.github/workflows/sandbox.yaml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: sandbox
2+
3+
on:
4+
push:
5+
branches:
6+
- sam/baml-test
7+
8+
concurrency:
9+
group: ${{ github.workflow }}-${{ github.ref }}
10+
cancel-in-progress: true
11+
12+
jobs:
13+
test:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: bash output
17+
run: |
18+
# Start a log group.
19+
echo "::group::Colored Log Group"
20+
21+
# Output text in various colors.
22+
echo -e "::group::\033[31mThis is red text\033[0m"
23+
echo -e "\033[32mThis is green text\033[0m"
24+
echo "::endgroup::"
25+
echo -e "\033[34mThis is blue text\033[0m"
26+
27+
# End the log group.
28+
echo "::endgroup::"
29+
30+
echo -e "::group::\033[31mThis is red text\033[0m"
31+
echo -e "\033[32mThis is green text\033[0m"
32+
sleep 120
33+
echo "::endgroup::"
34+
35+
# New log groups with pass/fail emojis
36+
echo -e "::group::\033[32m✅ This is a pass message\033[0m"
37+
echo "::endgroup::"
38+
39+
echo -e "::group::\033[31m❌ This is a fail message\033[0m"
40+
echo "::endgroup::"
41+
42+
echo -e "::group::\033[32m✅ Another pass message\033[0m"
43+
echo "::endgroup::"
44+
45+
echo -e "::group::\033[32m✅ Yet Another pass message\033[0m"
46+
echo "sleep 120 start"
47+
sleep 120
48+
echo "sleep 120 done"
49+
echo "::endgroup::"
50+
51+
# Output some colored text outside the group as well.
52+
echo "Ungrouped colored output:"
53+
echo -e "\033[35mThis is magenta text\033[0m"

0 commit comments

Comments
 (0)