[feat] Support `test.args.{ file }` for `string` types

[bennbollay]

Product

BAML

Problem Statement / Use Case

Currently even the "helper" AI on the documentation things that the following is legal:

function Extract(input: string) -> Meaning

test sample_1_data {
  functions [Extract]
  args {
    input {
      file "./full_of_meaning.txt"
    }
  }
}

The { file } approach is currently valid for image and audio parameter types, but being able to hoist string types as well from filesystem files makes a ton of sense. Not being able to do so violated my principle of least surprise when I was writing test cases.

Proposed Solution

Support { file } when loading string arguments in test sections.

Alternative Solutions

There's a variety of middling approaches or external scripts that can insert the data in, but none of them are "conformant" with the expectations around BAML.

Additional Context

Tagging @sxlijin and @Elijas regarding the thread in Discord on this topic.

[linear]

BAML-401 [feat] Support `test.{ file }` for `string` types

[Elijas]

There's a variety of middling approaches or external scripts that can insert the data in

I'll add these just in case someone needs a solution right now, even if it's not as elegant

Workaround 1: Wrap the external file as a baml file

template_string TestArticle1() #" 
  My name is john and i have a dog.
"#

template_string GetArticle(article: string) #"
  {% if article == "TestArticle1" %}{{ TestArticle1() }}
  {% else %}{{ article }}
  {% endif %}
"#

function Summarize(article: string) -> string {
client Default
prompt #"
  Summarize

  {{ GetArticle(article) }}
  
  {{ ctx.output_format }}
"#
}

test test1 {
    functions [Summarize]
    args {
      article TestArticle1
    }
}

Workaround 2: Use a 3rd-party script to create a new test which has the path replaced with contents

Disclaimer: I am the author. It's not thoroughly tested yet, bug reports and feedback welcome!

https://github.com/Elijas/baml-agents/tree/main/baml_agents/devtools#2-inline-baml-tests-inline-baml-test-file-arguments

[sxlijin]

Hey, just wanted to update you. We have some language work in the pipeline happening here around being able to load variables from files (demo) that we're deferring to on this.

Unfortunately it turned out that adding file loading had some frustrating technical implications around the underlying wiring (specifically it raised questions about what we allow as a string literal in the language, and we're nervous about allowing test-only constructs like this, or more generally the semantics of when we do the file reading). Sorry for the delay on this!

[smorin]

I would love this capability. It will help with testing a lot, for example, for files that violate escaping rules.

[sathish331977]

the documents we use are large and cannot be packed in a function as string, would be very helpful if we have this functionality

[nicolasvasquez]

Is there any update on this?

[sxlijin]

No update yet, sorry 😞