feat: Add first party eve instrumentation

[Luca Forstner (lforst)]

Ok this instrumentation is a bit interesting:

• Eve already has docs for braintrust but the traces it generates are not great. They're otel traces with wild disconnection issues.
• Eve is most likely deployed to Vercel workflows very often which means each step runs in it's own "function invocation", meaning we cannot rely on ALS.
• To combat the "can't use ALS" situation, we use the session/turn-IDs that eve gives us to produce hashes that we then use for our trace and span IDs.
• Also, because the turns are detached runtime-wise, we have a hard time creating a trace for each turn, which is arguably not what we want in any case (?) - so instead we attach all turns, tools, subagents, llm calls to a singular trace as they happen.

The way someone instruments their app is via:

// agent/instrumentation.ts
import { braintrustEveInstrumentation, initLogger } from "braintrust";
import { defineState } from "eve/context";
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation(
  braintrustEveInstrumentation({
    defineState,
    setup: ({ agentName }) => {
      initLogger({
        projectName: 'project',
      });
    },
  }),
);

plus

// agent/hooks/braintrust.ts
import { braintrustEveHook } from "braintrust";
import { defineHook } from "eve/hooks";

export default defineHook(braintrustEveHook());

[Abhijeet Prasad (AbhiPrasad)]

no orchestrion config? Or iitm setup?

[Abhijeet Prasad (AbhiPrasad)]

why are these false? Should we make them user configurable?

[Luca Forstner (lforst)]

These are options that defineInstrumentation() uses. When they are enabled, the otel data when turned on contains input and output data. Since awe are grabbing the input and output data from hook calls instead of instrumentation it doesn't matter.

[Abhijeet Prasad (AbhiPrasad)]

should we bound? Like use an LRU to manage memory?

[Abhijeet Prasad (AbhiPrasad)]

should we use current here? Ditto with some other places where we reference state instead of current.

[Abhijeet Prasad (AbhiPrasad)]

should we only call flushInstrumentation for events we handle?

[Abhijeet Prasad (AbhiPrasad)]

should be a minor!

[Abhijeet Prasad (AbhiPrasad)]

kind feels weird there is no createTurn and endTurn methods on the bridge here.

[Luca Forstner (lforst)]

We have handleTurnStarted and handleTurnCompleted and handleTurnFailed. Is that sufficient?

[Abhijeet Prasad (AbhiPrasad)]

Hmm in general the lifecycle here felt hard for me to follow. I didn't think too deeply about it though.

[Abhijeet Prasad (AbhiPrasad)]

I think we can have

type EveEntityKind = "session" | "step" | "subagent" | "tool" | "turn";

async function generateEveIds(
  kind: EveEntityKind,
  ...parts: string[]
): Promise<{ rowId: string; spanId: string }> {
  const [rowId, spanId] = await Promise.all([
    deterministicEveId(`eve:row:${kind}`, ...parts),
    deterministicEveId(`eve:${kind}`, ...parts),
  ]);
  return { rowId, spanId };
}

and take care of most of the use cases, and then we just need rootSpanIdForSession and spanIdForSubagent?

[Abhijeet Prasad (AbhiPrasad)]

clanker tells me ToolState.kind and ToolState.stepIndex were written at every creation site but never read.