Repository navigation
Commit f741e43
Fix validation to reject field-level assertions in test blocks (#2259)
## Summary
Fixes #1252 by adding validation to reject field-level `@assert` and
`@check` attributes on test blocks. These attributes were previously
accepted by the parser but silently ignored at runtime, causing
confusion.
## Problem
- BAML tests incorrectly accept `@assert` (single @) syntax without
warnings
- Field-level assertions on test fields are silently ignored at runtime
- Only block-level `@@assert` (double @) assertions actually work for
tests
- This leads to users writing non-functional tests that appear to work
## Solution
Added semantic validation in the test validator to:
1. Check all fields in test blocks for `@assert` or `@check` attributes
2. Emit clear error messages directing users to use `@@assert` or
`@@check` at the block level instead
3. Point to the exact location of the invalid attribute
## Changes
- **Modified**:
`engine/baml-lib/baml-core/src/validate/validation_pipeline/validations/tests.rs`
- Added validation logic to detect field-level assertion attributes
- Provides helpful error messages with the correct syntax
- **Added**: Test files to verify the validation works correctly
- `field_level_assertions_v2.baml` - validates @Assert rejection
- `field_level_check.baml` - validates @check rejection
## Testing
- ✅ All existing tests pass
- ✅ New validation tests pass with expected error messages
- ✅ No clippy warnings
- ✅ Error messages point to correct locations
## Example Error
```
error: @Assert is not allowed on test fields. Use @@Assert at the test block level instead.
--> functions_v2/tests/field_level_assertions_v2.baml:9
|
8 | test MyTest {
9 | functions [TestFunction] @Assert({{ true }})
|
```
🤖 Generated with [Claude Code](https://claude.ai/code)
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> Adds validation to reject field-level `@assert` and `@check`
attributes in test blocks, providing clear error messages and directing
users to use block-level assertions.
>
> - **Behavior**:
> - Adds validation to reject field-level `@assert` and `@check`
attributes in test blocks, directing users to use `@@assert` or
`@@check` at the block level.
> - Provides error messages pointing to the exact location of invalid
attributes.
> - **Validation Logic**:
> - Modified `validate()` in `tests.rs` to include logic for detecting
field-level assertion attributes.
> - **Testing**:
> - Added `field_level_assertions_v2.baml` and `field_level_check.baml`
to test validation of `@assert` and `@check` rejection.
> - All existing and new tests pass, ensuring no regression.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup>
for 86029cc. You can
[customize](https://app.ellipsis.dev/BoundaryML/settings/summaries) this
summary. It will automatically update as commits are pushed.</sup>
<!-- ELLIPSIS_HIDDEN -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aaronvg <aaron@boundaryml.com>1 parent b4ddb6f commit f741e43
3 files changed
Lines changed: 58 additions & 0 deletions
File tree
- engine/baml-lib
- baml-core/src/validate/validation_pipeline/validations
- baml/tests/validation_files/functions_v2/tests
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
10 | 28 | | |
11 | 29 | | |
12 | 30 | | |
| |||
Lines changed: 20 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
Lines changed: 20 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
0 commit comments