Skip to content

Commit 854cdbd

Browse files
aaronvgcursoragent
andauthored
Add baml-cli test documentation (#2021)
A new reference document for the `baml test` command was created at `fern/03-reference/baml-cli/test.mdx`. This document provides comprehensive details on the command's usage, options (e.g., `--include`, `--exclude`, `--parallel`), test filtering patterns, parallel execution, environment variable handling, and exit codes for CI/CD integration. It includes various examples for different workflows and explains test definition and output. Additionally, a "Command Line Testing" section was added to `fern/01-guide/04-baml-basics/testing-functions.mdx`. This section introduces `baml test` as an alternative to the VSCode playground, providing quick usage examples and highlighting its utility for CI/CD pipelines and batch testing. It also includes a cross-reference to the newly created detailed CLI test reference. These changes ensure `baml test` is fully documented and discoverable within the existing documentation structure. <!-- ELLIPSIS_HIDDEN --> ---- > [!IMPORTANT] > Adds comprehensive documentation for the `baml-cli test` command, including usage, options, and examples, and integrates it into the existing documentation structure. > > - **Documentation**: > - Adds `baml-cli test` command reference in `test.mdx`, covering usage, options, filtering, parallel execution, environment variables, and exit codes. > - Includes examples for different workflows, test definition, and output. > - **Guide Update**: > - Adds "Command Line Testing" section in `testing-functions.mdx`, introducing `baml test` as an alternative to VSCode playground. > - Provides quick usage examples and highlights utility for CI/CD pipelines. > - Cross-references the detailed CLI test reference. > - **Navigation**: > - Updates `docs.yml` to include the new `test.mdx` in the documentation navigation. > > <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup> for a2dc1f9. You can [customize](https://app.ellipsis.dev/BoundaryML/settings/summaries) this summary. It will automatically update as commits are pushed.</sup> <!-- ELLIPSIS_HIDDEN --> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
1 parent e9303ae commit 854cdbd

3 files changed

Lines changed: 277 additions & 49 deletions

File tree

fern/01-guide/04-baml-basics/testing-functions.mdx

Lines changed: 78 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
slug: /guide/baml-basics/testing-functions
33
---
44

5-
65
You can test your BAML functions in the VSCode Playground by adding a `test` snippet into a BAML file:
76

87
```baml
@@ -33,33 +32,35 @@ test Test1 {
3332
```
3433

3534
### Try it! Press 'Run Test' below!
36-
<div class="resizer">
37-
<iframe
38-
class="resized"
39-
src="https://promptfiddle.com/embed?id=testing_functions"
40-
41-
height="640"
42-
style="border: none;"
43-
resize="both"
44-
overflow="auto"
45-
msallowfullscreen
46-
></iframe>
47-
</div>
48-
4935

36+
{" "}
37+
38+
<div class="resizer">
39+
<iframe
40+
class="resized"
41+
src="https://promptfiddle.com/embed?id=testing_functions"
42+
height="640"
43+
style="border: none;"
44+
resize="both"
45+
overflow="auto"
46+
msallowfullscreen
47+
></iframe>
48+
</div>
5049

5150
See more [interactive examples](https://promptfiddle.com)
5251

5352
The BAML playground will give you a starting snippet to copy that will match your function signature.
5453

5554
<Warning>
56-
BAML doesn't use colons `:` between key-value pairs except in function parameters.
55+
BAML doesn't use colons `:` between key-value pairs except in function
56+
parameters.
5757
</Warning>
5858

5959
<hr />
6060
## Complex object inputs
6161

6262
Objects are injected as dictionaries
63+
6364
```rust
6465
class Message {
6566
user string
@@ -87,6 +88,7 @@ test Test1 {
8788
}
8889
}
8990
```
91+
9092
<hr />
9193
## Test Image Inputs in the Playground
9294

@@ -131,7 +133,8 @@ test Test1 {
131133
<ParamField path="file" type="string" required="true">
132134
The path to the image file, relative to the directory containing the current BAML file.
133135

134-
Image files must be somewhere in `baml_src/`.
136+
Image files must be somewhere in `baml_src/`.
137+
135138
</ParamField>
136139

137140
<ParamField path="media_type" type="string">
@@ -185,7 +188,8 @@ test Test1 {
185188
The mime-type of the image. If not set, and the provider expects a mime-type
186189
to be provided, BAML will try to infer it based on the contents of the file.
187190

188-
If `base64` is a data URL, this field will be ignored.
191+
If `base64` is a data URL, this field will be ignored.
192+
189193
</ParamField>
190194

191195
</Tab>
@@ -212,10 +216,10 @@ You can define test cases using audio files, URLs, or base64 strings.
212216
<Tab title="File">
213217

214218
<Warning>
215-
Committing a lot of audio files into your repository can make it slow to clone and
216-
pull your repository. If you expect to commit >500MiB of audio, please read
217-
[GitHub's size limit documentation][github-large-files] and consider setting
218-
up [large file storage][github-lfs].
219+
Committing a lot of audio files into your repository can make it slow to clone
220+
and pull your repository. If you expect to commit >500MiB of audio, please
221+
read [GitHub's size limit documentation][github-large-files] and consider
222+
setting up [large file storage][github-lfs].
219223
</Warning>
220224

221225
```baml
@@ -232,7 +236,8 @@ test Test1 {
232236
<ParamField path="file" type="string" required="true">
233237
The path to the audio file, relative to the directory containing the current BAML file.
234238

235-
audio files must be somewhere in `baml_src/`.
239+
audio files must be somewhere in `baml_src/`.
240+
236241
</ParamField>
237242

238243
<ParamField path="media_type" type="string">
@@ -286,7 +291,8 @@ test Test1 {
286291
The mime-type of the audio. If not set, and the provider expects a mime-type
287292
to be provided, BAML will try to infer it based on the contents of the file.
288293

289-
If `base64` is a data URL, this field will be ignored.
294+
If `base64` is a data URL, this field will be ignored.
295+
290296
</ParamField>
291297
</Tab>
292298
</Tabs>
@@ -295,16 +301,16 @@ test Test1 {
295301

296302
Test blocks in BAML code may contain checks and asserts. These attributes
297303
behave similarly to value-level [Checks and Asserts](/guide/baml-advanced/checks-and-asserts),
298-
with several additional variables available in the context of the jinja
304+
with several additional variables available in the context of the jinja
299305
expressions you can write in a test:
300306

301-
- The `_` variable contains fields `result`, `checks` and `latency_ms`.
302-
- The `this` variable refers to the value computed by the test, and is
303-
shorthand for `_.result`.
304-
- In a given check or assert, `_.checks.$NAME` can refer to the NAME of any earlier
305-
check that was run in the same test block. By referring to prior checks,
306-
you can build compound checks and asserts, for example asserting that all
307-
checks of a certain type passed.
307+
- The `_` variable contains fields `result`, `checks` and `latency_ms`.
308+
- The `this` variable refers to the value computed by the test, and is
309+
shorthand for `_.result`.
310+
- In a given check or assert, `_.checks.$NAME` can refer to the NAME of any earlier
311+
check that was run in the same test block. By referring to prior checks,
312+
you can build compound checks and asserts, for example asserting that all
313+
checks of a certain type passed.
308314

309315
The following example illustrates how each of these features can be used to
310316
validate a test result.
@@ -323,12 +329,13 @@ test MyTest {
323329
```
324330

325331
`@@check` and `@@assert` behave differently:
326-
- A `@@check` represents a property
327-
of the test result that should either be manually checked or checked by a
328-
subsequent stage in the test. Multiple `@@check` predicates can fail
329-
without causing a hard failure of the test.
330-
- An `@@assert` represents a hard guarantee. The first failing assert will halt
331-
the remainder of the checks and asserts in this particular test.
332+
333+
- A `@@check` represents a property
334+
of the test result that should either be manually checked or checked by a
335+
subsequent stage in the test. Multiple `@@check` predicates can fail
336+
without causing a hard failure of the test.
337+
- An `@@assert` represents a hard guarantee. The first failing assert will halt
338+
the remainder of the checks and asserts in this particular test.
332339

333340
For more information about the syntax used inside `@@check` and `@@assert`
334341
attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts)
@@ -345,15 +352,37 @@ The `type_builder` block can contain new types scoped to the parent `test` block
345352
and also `dynamic` blocks that act as modifiers for dynamic classes or enums.
346353

347354
### Try it! Press 'Run Test' below!
348-
<div class="resizer">
349-
<iframe
350-
class="resized"
351-
src="https://promptfiddle.com/embed?id=dynamic_types"
352-
353-
height="640"
354-
style="border: none;"
355-
resize="both"
356-
overflow="auto"
357-
msallowfullscreen
358-
></iframe>
359-
</div>
355+
356+
{" "}
357+
358+
<div class="resizer">
359+
<iframe
360+
class="resized"
361+
src="https://promptfiddle.com/embed?id=dynamic_types"
362+
height="640"
363+
style="border: none;"
364+
resize="both"
365+
overflow="auto"
366+
msallowfullscreen
367+
></iframe>
368+
</div>
369+
370+
## Command Line Testing
371+
372+
While the VSCode playground is excellent for interactive development and debugging, you can also run your tests from the command line using the BAML CLI:
373+
374+
```bash
375+
# Run all tests
376+
baml-cli test
377+
378+
# Run tests for a specific function
379+
baml-cli test -i "ClassifyMessage::"
380+
381+
# Run tests in parallel with custom concurrency
382+
baml-cli test --parallel 5
383+
384+
# List available tests without running them
385+
baml-cli test --list
386+
```
387+
388+
See the [CLI Test Reference](/ref/baml-cli/test) for complete documentation of all available options, filtering capabilities, and output formats.

0 commit comments

Comments
 (0)