llm logs should show information on tool calls

[simonw]

Follows:

• Persist tools and tool responses to the database #1003

[simonw]

I was thinking about truncating this, then I realized that for any LLM prompt that included a tool call the full details of the tool call are the most interesting thing about it, so no truncation!

[simonw]

Strikes me that the llm logs command should have a filter that lets you see just the logs that used a specific tool... but we don't actually have a user-facing unique ID for tools yet since the name of the tool may not be particularly interesting, especially for tools provided using the --tools PYTHON_CODE option.

I guess maybe we should us the sha256 hash of the name and description and arguments? Closest we have to a useful identifier.

Note that llm fragments and llm schemas commands reveal things stashed in the database, but llm tools currently does something different: it shows tools registered via plugins.

Maybe that's a bad design? It's inconsistent at least.

[simonw]

For the moment just showing tool information in the logs is enough to close the issue. I'll open another post-alpha one for filtering options.

[simonw]

With some help from Claude I came up with this:

SELECT
  responses.id,
  -- Tools related to this response
  COALESCE(
    (SELECT json_group_array(json_object(
        'id', t.id,
        'hash', t.hash,
        'name', t.name,
        'description', t.description,
        'input_schema', json(t.input_schema)
      ))
      FROM tools t
      JOIN tool_responses tr ON t.id = tr.tool_id
      WHERE tr.response_id = responses.id
    ), 
    '[]'
  ) AS tools,
  -- Tool calls for this response
  COALESCE(
    (SELECT json_group_array(json_object(
        'id', tc.id,
        'tool_id', tc.tool_id,
        'name', tc.name,
        'arguments', tc.arguments,
        'tool_call_id', tc.tool_call_id
      ))
      FROM tool_calls tc
      WHERE tc.response_id = responses.id
    ),
    '[]'
  ) AS tool_calls,
  -- Tool results for this response
  COALESCE(
    (SELECT json_group_array(json_object(
        'id', tr.id,
        'tool_id', tr.tool_id,
        'name', tr.name,
        'output', tr.output,
        'tool_call_id', tr.tool_call_id
      ))
      FROM tool_results tr
      WHERE tr.response_id = responses.id
    ),
    '[]'
  ) AS tool_results
FROM responses
  where id in (:one, :two, :three);

[simonw]

id	tools	tool_calls	tool_results
01jv3xct4k39aeq7andqftc82b	[]	[]	[]
01jv3xfrrk97nfexqtm9njbfcs	[{"id":4,"hash":"57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0","name":"now","description":"Time right now","input_schema":{"properties":{},"type":"object"}}]	[{"id":9,"tool_id":4,"name":"now","arguments":"{}","tool_call_id":"toolu_01SeBftjFg6N1ZW5wqca6T3Q"}]	[]
01jv3xfsjpkkvnsy64sk0726cs	[{"id":4,"hash":"57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0","name":"now","description":"Time right now","input_schema":{"properties":{},"type":"object"}}]	[]	[{"id":9,"tool_id":4,"name":"now","output":"2025-05-12 21:12:17.366068","tool_call_id":"toolu_01SeBftjFg6N1ZW5wqca6T3Q"}]

[simonw]

Short output looks like this:

- model: anthropic/claude-3-haiku-20240307
  datetime: '2025-05-13T04:12:16'
  conversation: 01jv3xfrrhy7dyfddv49pp05db
  tool_calls:
  - now({})
  prompt: what time is it?
  prompt_fragments: []
  system_fragments: []
- model: anthropic/claude-3-haiku-20240307
  datetime: '2025-05-13T04:12:17'
  conversation: 01jv3xfrrhy7dyfddv49pp05db
  tool_results:
  - 'now: 2025-05-12 21:12:17.366068'
  prompt_fragments: []
  system_fragments: []

JSON output looks like this:

[
  {
    "id": "01jv3xfrrk97nfexqtm9njbfcs",
    "model": "anthropic/claude-3-haiku-20240307",
    "prompt": "what time is it?",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": "",
    "response_json": {},
    "conversation_id": "01jv3xfrrhy7dyfddv49pp05db",
    "duration_ms": 834,
    "datetime_utc": "2025-05-13T04:12:16.531047+00:00",
    "input_tokens": 311,
    "output_tokens": 34,
    "token_details": null,
    "conversation_name": "what time is it?",
    "conversation_model": "anthropic/claude-3-haiku-20240307",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "tools": [
      {
        "id": 4,
        "hash": "57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0",
        "name": "now",
        "description": "Time right now",
        "input_schema": {
          "properties": {},
          "type": "object"
        }
      }
    ],
    "tool_calls": [
      {
        "id": 9,
        "tool_id": 4,
        "name": "now",
        "arguments": {},
        "tool_call_id": "toolu_01SeBftjFg6N1ZW5wqca6T3Q"
      }
    ],
    "tool_results": [],
    "attachments": []
  },
  {
    "id": "01jv3xfsjpkkvnsy64sk0726cs",
    "model": "anthropic/claude-3-haiku-20240307",
    "prompt": "",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": "The current time is 21:12:17.",
    "response_json": {},
    "conversation_id": "01jv3xfrrhy7dyfddv49pp05db",
    "duration_ms": 844,
    "datetime_utc": "2025-05-13T04:12:17.366282+00:00",
    "input_tokens": 372,
    "output_tokens": 15,
    "token_details": null,
    "conversation_name": "what time is it?",
    "conversation_model": "anthropic/claude-3-haiku-20240307",
    "schema_json": null,
    "prompt_fragments": [],
    "system_fragments": [],
    "tools": [
      {
        "id": 4,
        "hash": "57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0",
        "name": "now",
        "description": "Time right now",
        "input_schema": {
          "properties": {},
          "type": "object"
        }
      }
    ],
    "tool_calls": [],
    "tool_results": [
      {
        "id": 9,
        "tool_id": 4,
        "name": "now",
        "output": "2025-05-12 21:12:17.366068",
        "tool_call_id": "toolu_01SeBftjFg6N1ZW5wqca6T3Q"
      }
    ],
    "attachments": []
  }
]

[simonw]

Looks like this:

2025-05-13T04:12:16 conversation: 01jv3xfrrhy7dyfddv49pp05db id: 01jv3xfrrk97nfexqtm9njbfcs

Model: anthropic/claude-3-haiku-20240307

Prompt

what time is it?

Tools

• now: 57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0
  
  Time right now
  
  Arguments: {}

Response

Tool calls

• now: toolu_01SeBftjFg6N1ZW5wqca6T3Q
  
  Arguments: {}

2025-05-13T04:12:17

Prompt

-- none --

Tools

• now: 57b370ea2a2ad98e2b152df7e1714dafea6112991c5b72736fe83cc2adc8ffa0
  
  Time right now
  
  Arguments: {}

Tool results

• now: toolu_01SeBftjFg6N1ZW5wqca6T3Q
  
  2025-05-12 21:12:17.366068

Response

The current time is 21:12:17.