feat(activity): add activity tracking and LLM cost visibility#131
Merged
danielnaab merged 9 commits intomainfrom May 7, 2026
Merged
feat(activity): add activity tracking and LLM cost visibility#131danielnaab merged 9 commits intomainfrom
danielnaab merged 9 commits intomainfrom
Conversation
Foundation service for story #119 (activity tracking and LLM cost visibility). SQLite-backed store records events with timestamps, supports filtered queries, and computes usage summaries with estimated costs from Bedrock pricing.
Fire-and-forget helper that records LLM usage (model, input/output tokens, duration) to the activity store. Designed to be called after each generateText() invocation without affecting the hot path.
Wrap generateText() calls in extraction, shaping, and filling with timing and token tracking. The activityStore is created in server.tsx and threaded through registries and agent constructors. Instrumented operations: - extraction (Step 1 PDF analysis) - extraction-formspec (Step 2 FormSpec generation) - extraction-fieldmapping (Step 3 AcroForm field mapping) - shaping (form structure editing) - filling (conversational form filling) All tracking is fire-and-forget — failures log to stderr without affecting the user-facing operation.
Add activity tracking calls for non-LLM events: - sign_in: tracked in auth callback after successful authentication - project_created: tracked for PDF upload, fixture, and corpus project creation - pdf_uploaded: tracked when a user uploads a PDF file Thread activityStore to createAuthRoutes via an optional options parameter.
Adds the `activity` CLI command with three subcommands: - `summary` — shows usage summary with cost breakdowns by user/operation/project - `events` — lists raw activity events with filtering options - `digest` — posts daily usage digest to Slack via the notify service
…ntation - Pass userId to extraction options in project-service fireAndForgetExtraction - Add userId/projectId to shaping request in edit route intent handler - Add userId/projectId to filling context in both chat view and chat message handlers - Instrument tool-use extraction variant with trackLlmCall timing - Pass activityStore through extraction registry to tool-use variant - Complete pricing table with all model IDs used in the codebase
danielnaab
force-pushed
the
story-119/activity-tracking
branch
from
e111224 to
c89f55a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/services/activity/service that logs platform events (LLM calls, project creation, sign-in, PDF upload) with full user/project/operation attributionbun run cli activity summary|events|digest) for querying activity and posting daily Slack digestsStory
Closes #119
Acceptance Criteria
Test Plan
bun run checkpasses (1367 tests, 0 failures)tsc --noEmit)bun run cli activity digest)Review Notes
test/layout-tokens.test.tsfails withoutdist/styles.cssbuild artifact — unrelated to this branch.