Skip to content

Commit f741e43

Browse files
dexhorthyclaudeaaronvg
authored
Fix validation to reject field-level assertions in test blocks (#2259)
## Summary Fixes #1252 by adding validation to reject field-level `@assert` and `@check` attributes on test blocks. These attributes were previously accepted by the parser but silently ignored at runtime, causing confusion. ## Problem - BAML tests incorrectly accept `@assert` (single @) syntax without warnings - Field-level assertions on test fields are silently ignored at runtime - Only block-level `@@assert` (double @) assertions actually work for tests - This leads to users writing non-functional tests that appear to work ## Solution Added semantic validation in the test validator to: 1. Check all fields in test blocks for `@assert` or `@check` attributes 2. Emit clear error messages directing users to use `@@assert` or `@@check` at the block level instead 3. Point to the exact location of the invalid attribute ## Changes - **Modified**: `engine/baml-lib/baml-core/src/validate/validation_pipeline/validations/tests.rs` - Added validation logic to detect field-level assertion attributes - Provides helpful error messages with the correct syntax - **Added**: Test files to verify the validation works correctly - `field_level_assertions_v2.baml` - validates @Assert rejection - `field_level_check.baml` - validates @check rejection ## Testing - ✅ All existing tests pass - ✅ New validation tests pass with expected error messages - ✅ No clippy warnings - ✅ Error messages point to correct locations ## Example Error ``` error: @Assert is not allowed on test fields. Use @@Assert at the test block level instead. --> functions_v2/tests/field_level_assertions_v2.baml:9 | 8 | test MyTest { 9 | functions [TestFunction] @Assert({{ true }}) | ``` 🤖 Generated with [Claude Code](https://claude.ai/code) <!-- ELLIPSIS_HIDDEN --> ---- > [!IMPORTANT] > Adds validation to reject field-level `@assert` and `@check` attributes in test blocks, providing clear error messages and directing users to use block-level assertions. > > - **Behavior**: > - Adds validation to reject field-level `@assert` and `@check` attributes in test blocks, directing users to use `@@assert` or `@@check` at the block level. > - Provides error messages pointing to the exact location of invalid attributes. > - **Validation Logic**: > - Modified `validate()` in `tests.rs` to include logic for detecting field-level assertion attributes. > - **Testing**: > - Added `field_level_assertions_v2.baml` and `field_level_check.baml` to test validation of `@assert` and `@check` rejection. > - All existing and new tests pass, ensuring no regression. > > <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup> for 86029cc. You can [customize](https://app.ellipsis.dev/BoundaryML/settings/summaries) this summary. It will automatically update as commits are pushed.</sup> <!-- ELLIPSIS_HIDDEN --> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: aaronvg <aaron@boundaryml.com>
1 parent b4ddb6f commit f741e43

3 files changed

Lines changed: 58 additions & 0 deletions

File tree

‎engine/baml-lib/baml-core/src/validate/validation_pipeline/validations/tests.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
use baml_types::{Constraint, ConstraintLevel};
2+
use internal_baml_ast::ast::WithName;
23
use internal_baml_diagnostics::{DatamodelError, DatamodelWarning, Span};
34
use internal_baml_jinja_types::{validate_expression, JinjaContext, PredefinedTypes, Type};
45

@@ -7,6 +8,23 @@ use crate::validate::validation_pipeline::context::Context;
78
pub(super) fn validate(ctx: &mut Context<'_>) {
89
let tests = ctx.db.walk_test_cases().collect::<Vec<_>>();
910
tests.iter().for_each(|walker| {
11+
// Validate that test fields don't have @assert or @check attributes
12+
let test_ast = walker.ast_node();
13+
for (_field_id, field) in test_ast.iter_fields() {
14+
for attr in &field.attributes {
15+
if attr.name.name() == "assert" || attr.name.name() == "check" {
16+
ctx.push_error(DatamodelError::new_validation_error(
17+
&format!(
18+
"@{} is not allowed on test fields. Use @@{} at the test block level instead.",
19+
attr.name.name(),
20+
attr.name.name()
21+
),
22+
attr.span.clone(),
23+
));
24+
}
25+
}
26+
}
27+
1028
let constraints = &walker.test_case().constraints;
1129
let args = &walker.test_case().args;
1230
let mut check_names: Vec<String> = Vec::new();
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Test that field-level assertions are not allowed on test block fields
2+
3+
function TestFunction(input: string) -> string {
4+
client "openai/gpt-4"
5+
prompt #"Test function"#
6+
}
7+
8+
test MyTest {
9+
functions [TestFunction] @assert({{ true }})
10+
args {
11+
input "hello"
12+
}
13+
}
14+
15+
// error: Error validating: @assert is not allowed on test fields. Use @@assert at the test block level instead.
16+
// --> functions_v2/tests/field_level_assertions_v2.baml:9
17+
// |
18+
// 8 | test MyTest {
19+
// 9 | functions [TestFunction] @assert({{ true }})
20+
// |
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Test that field-level @check is not allowed on test block fields
2+
3+
function TestFunction(input: string) -> string {
4+
client "openai/gpt-4"
5+
prompt #"Test function"#
6+
}
7+
8+
test MyTest {
9+
functions [TestFunction]
10+
args {
11+
input "hello"
12+
} @check(args_check, {{ true }})
13+
}
14+
15+
// error: Error validating: @check is not allowed on test fields. Use @@check at the test block level instead.
16+
// --> functions_v2/tests/field_level_check.baml:12
17+
// |
18+
// 11 | input "hello"
19+
// 12 | } @check(args_check, {{ true }})
20+
// |

0 commit comments

Comments
 (0)