[DONE] feature/tokensa — implement token usage tracking and fix queue-bootstrap schema mismatches

[bharatsachya]

What changed

• LLM Token Tracking:
  • Updated @bb/llm clients (jsonClient, ollama, openrouter) to return inputTokens and outputTokens in analysis responses.
  • Refactored analysis strategies (file, folder, repo summary) to bubble up token usage through the pipeline.
  • Enhanced GithubKnowledgeSource and KnowledgeInfo types to support CommitHashRecord with token metrics.
• Queue Bootstrap Fixes (Enterprise):
  • Lint: Resolved all no-explicit-any and curly errors in packages/knowledge-server/src/queue-bootstrap.ts.
  • Bug Fix: Migrated from record.source to record.info.githubInfo to align with the enterprise schema and ensure ingestion stats are correctly synced.
  • Type Safety: Mapped OSS status.processedFiles to enterprise info.githubInfo.noOfFilesProcessed to resolve TypeScript compilation errors.
• Persistence Layer: Updated persistIngestionRun in packages/progress to accurately record token usage by commit and increment user usage metrics in the users collection.

Why

• Usage Granularity: We need to track LLM costs at a per-commit and per-run level for better visibility into ingestion overhead.
• Schema Alignment: The enterprise knowledge-server was failing to sync stats because it was looking for commit IDs in the OSS-specific source field instead of the enterprise-standard info.githubInfo.
• System Stability: Resolving these type mismatches and lint errors prevents runtime crashes during job completion.

How to test

1. Verify Builds & Lint:
   • Run bun run lint in the root.
   • Run bun run typecheck in the root.
2. End-to-End Ingestion:
   • Trigger a GitHub ingestion job for a repository.
   • Once completed, verify the knowledge record in MongoDB:
     • Check info.githubInfo.tokenUsageByCommit for the current commit hash.
     • Check info.githubInfo.commitHashes for the new inputTokens and outputTokens fields.
3. Usage Metrics:
   • Verify that the user's usage field in the users collection has been incremented by the total token count (input + output) of the ingestion run.