Expectations

[hinthornw]

• change default dataset creation (should be unique per file)

• change env vars to use LANGSMITH_ prefix
  Want feedback on:

• ergonomics: is this a general UX we want to support

• imports: should i re-implement string, embedding distance, etc.?

• Do we want default implementations?

• Any other we ought to include at the outset?

• I could also do a general expect(value).is(...) or something: want anything super generic like that?

Example:

@unit(inputs=x, outputs=y)
def test_output_semantically_close():
    response = oai_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Say hello!"},
        ],
    )
    # The embedding_distance call logs the embedding distance to LangSmith
    expect.embedding_distance(
        prediction=response.choices[0].message.content,
        reference="Hello!",
        # The following optional assertion logs a
        # pass/fail score to LangSmith
        # and raises an AssertionError if the assertion fails.
    ).to_be_less_than(0.5)
    # Compute damerau_levenshtein distance
    expect.string_distance(
        prediction=response.choices[0].message.content,
        reference="Hello!",
        # And then log a pass/fail score to LangSmith
    ).to_be_less_than(0.5)

The idea is it's still an easy onramp for developers to quickly write some scoring functions and get it running regularly in CI

[hinthornw]

It's playing funky with doctest - looking into it more tonight