-
Notifications
You must be signed in to change notification settings - Fork 19
Fact shards experiment results
VScode stores chat files along with a variety of prompt metrics at ~/Application Support/Code/User/workspaceStorage/<uuid>/chatSessions/<chat_session_uuid>.jsonl. With AI assistance a small tool was written to extract various token related metrics from a pair of chats then compute the delta between them to determine which session performed better.
Prompt tokens
- Tokens consumed by the input to the model (user message + system instructions + loaded facts/context).
- Higher prompt tokens = more context/instructions were given to the agent.
Completion tokens
- Tokens generated by the model as output (code, explanations, reasoning).
- Lower completion tokens = agent was more direct and less exploratory.
Total tokens
- Sum of prompt tokens + completion tokens.
- Reflects overall API cost for the task.
Duration to complete
- Time in seconds from task start to finish.
- Includes agent thinking time, tool execution time, and model response latency.
- A prompt is given to the old
api-endpoint-builder.mdagent and the chat session id is noted - In a new chat, the exact same prompt (and model) is given to the new
api-endpoint-builder.mdthat references the new "fact shards" - The two chats are fed into the analysis tool
make token-benchmark-analyze-two-sessions OLD=<session_uuid> NEW=<session_uuid> - Deltas are generated and a report is created
We're looking mainly for negative median deltas as a rough signal that the "fact shard" version of the agent is consuming fewer tokens than the agent that contains all patterns and code examples regardless of relevance.
Even with identical prompts, reasoning paths can differ between runs (and obviously models) and simply cannot be accounted for when benchmarking.
- Tools may be called differently, or in different orders that affect output or next steps
- Codebase exploration depth (and order of exploration) varies between runs.
Ideally each test should have been repeated several times and averaged before comparison. Since that's a lot of tokens I opted not to do it that way - i.e. take these measurements with a grain of salt
In this test, we see some pretty good results, fewer tokens were used and it completed significantly faster using the shard approach.
Model: GPT-5-3-Codex
Prompt:
In app/service/service_data_retention_schema.py add bulk_update_service_data_retention_request requiring integer days_of_retention and notification_types as a non-empty, unique array of enum [sms, letter, email]. In app/dao/service_data_retention_dao.py add fetch_service_data_retention_by_notification_types(service_id, notification_types) ordered by notification_type and bulk_update_service_data_retention(service_id, notification_types, days_of_retention) that transactionally updates matching rows for that service (set days_of_retention and updated_at=datetime.utcnow(), use synchronize_session=False). Do not modify routes, models, migrations, or unrelated files.
| Metric | Median Delta |
|---|---|
| Prompt tokens | -54.69% |
| Total tokens | -53.03% |
| Completion tokens | +14.59% |
| Duration | -32.98% |
| Turns | 0.00% |
This test was not good at all, the shard approach was abysmal compared to the non-shard approach. The AI performed a lot of exploratory steps before it began implementing. It looked at a ton of file for examples of patterns, some of which were documented in the shards already.
Model: GPT-5-3-Codex
Prompt:
Add GET endpoint /service/<service_id>/template-usage-by-category that returns notification usage aggregated by template category. Response schema includes array of category objects with category_name (string), total_notifications (integer), by_status (object with sent/delivered/failed counts), and by_notification_type (object with sms/email/letter counts). In app/dao/notification_dao.py add get_notification_usage_by_template_category(service_id, start_date, end_date) that joins Notification, TemplateHistory, and TemplateCategory models, groups by category, and calculates counts. Add schema validation for date parameters and unit tests.
| Metric | Median Delta |
|---|---|
| Prompt tokens | +83.47% |
| Total tokens | +76.49% |
| Completion tokens | +10.41% |
| Duration | +12.07% |
| Turns | -3.33% |
Opus 4.6 did quite a bit better, using fewer tokens but the agent still got stuck in several exploratory loops, dug through the codebase for patterns and examples - which again were already documented in the linked facts files (e.g. how to register a blueprint).
Model: Claude Opus 4.6
Prompt:
Add GET endpoint /service/<service_id>/template-usage-by-category that returns notification usage aggregated by template category. Response schema includes array of category objects with category_name (string), total_notifications (integer), by_status (object with sent/delivered/failed counts), and by_notification_type (object with sms/email/letter counts). In app/dao/notification_dao.py add get_notification_usage_by_template_category(service_id, start_date, end_date) that joins Notification, TemplateHistory, and TemplateCategory models, groups by category, and calculates counts. Add schema validation for date parameters and unit tests.
| Metric | Median Delta |
|---|---|
| Prompt tokens | +12.15% |
| Total tokens | +6.14% |
| Completion tokens | -41.26% |
| Duration | -37.98% |
| Turns | -32.56% |
Two new fact shards were added at this point:
- One detailing query aggregation patterns
- One restricting how deeply the agent can explore the codebase for existing patterns
This time around the model "wandered" around the codebase less when thinking and planning the task and completed it much faster, suspiciously faster than expected which makes me think this is more of a "lucky run" than an actual improvement.
Model: Claude Opus 4.6
Prompt:
Add GET endpoint /service/<service_id>/template-usage-by-category that returns notification usage aggregated by template category. Response schema includes array of category objects with category_name (string), total_notifications (integer), by_status (object with sent/delivered/failed counts), and by_notification_type (object with sms/email/letter counts). In app/dao/notification_dao.py add get_notification_usage_by_template_category(service_id, start_date, end_date) that joins Notification, TemplateHistory, and TemplateCategory models, groups by category, and calculates counts. Add schema validation for date parameters and unit tests.
| Metric | Median Delta |
|---|---|
| Prompt tokens | -4.74% |
| Total tokens | -3.18% |
| Completion tokens | +9.12% |
| Duration | +14.88% |
| Turns | +18.60% |
While it seems there are some gains with the shard approach, it's difficult to say with certainty given the variance in reasoning paths between runs. When the prompt given mapped well to the documented fact shards like in test 1 the results look promising. Conversely when exploration + shard context is utilized (Tests 2-4) the results seem unreliable. Lastly maintaining a well mapped catalogue of repo-specific facts seems like a high level of effort for a gain that isn't guaranteed and can even be detrimental.