feat(llm): anonymous per-feature token-usage tracking

[Ar1anit]

Summary

Track LLM token usage per AI feature, fully anonymized, so we understand how users interact with our AI features and build a sellable / ML-ready dataset ahead of a potential acquisition.

Hard requirement: everything must be anonymous. No prompt/response content is stored, and usage rows are not linked to a User record. Each row is keyed by a stable salted SHA-256 actor hash so we keep per-actor behavioral sequences (useful for ML) while being irreversible and surviving account deletion.

This issue covers the foundational tracking layer only. The ML model and an analytics dashboard are explicitly out of scope and tracked as follow-ups (see "Out of scope").

Motivation

• Understand which features burn the most tokens (cover letters vs. resumes vs. interviews vs. parsing) and how that correlates with tier and language.
• Cost visibility per feature/model to protect margins.
• Build an anonymized behavioral dataset that strengthens the company's valuation in a future sale.

Why this is cheap to add

• Single chokepoint: every AI call already passes through LLMService.callProvider() (wrapped by the opossum circuit breaker) in apps/api/src/llm/llm.service.ts.
• Tokens are already returned: the Azure OpenAI response includes usage { prompt_tokens, completion_tokens, total_tokens } — we currently discard it in apps/api/src/llm/providers/azure-openai.provider.ts.
• userId, tier, and detected language are already available at call time.

Anonymity model

• actorHash = sha256(userId + LLM_USAGE_HASH_SALT) — new secret env var.
• No User foreign key on the usage table. Stable per user, irreversible without the salt, and survives account deletion → genuinely anonymous analytics.
• Never persist prompt or completion text — only counts and metadata.

Scope (this issue)

1. Data model — new LlmUsageEvent (Prisma, no User relation)

Fields:

• actorHash (indexed)
• feature (one of: application-cover-letter, application-resume, application-profile-tailor, application-ats-keywords, application-skill-categorization, application-translation, keywords-extraction, keywords-profile, interview-questions, interview-feedback, interview-analysis, resume-parser)
• provider (azure-openai | azure-ai-foundry | mock)
• model / deployment name
• promptTokens, completionTokens, totalTokens
• tier (FREE | PRO | PREMIUM at time of call)
• language (DE | EN)
• latencyMs
• success (bool) + circuitState (closed/open/halfOpen)
• estimatedCostUsd (tokens × per-model price map)
• createdAt
• Indexes on (feature, createdAt) and (actorHash, createdAt)

Open question for implementation: store feature as a Prisma enum (type-safe, refactor-friendly) vs. a plain String (more flexible). Recommend enum.

2. Plumbing

• Extend LLMProvider.generateText() to return { content: string; usage?: { promptTokens; completionTokens; totalTokens } } in apps/api/src/llm/llm.interface.ts.
• Capture response.data.usage in azure-openai.provider.ts; update azure-ai-foundry.provider.ts and mock.provider.ts to the new contract (mock synthesizes plausible counts).
• Add a lightweight LlmCallContext (feature, userId, tier, language) param to callText / callJson; record one LlmUsageEvent inside callProvider() (success and failure paths).
• Add LLM_USAGE_HASH_SALT to the Zod env schema and apps/api/.env.example.
• Per-model price map for estimatedCostUsd.

3. Call-site tagging

Tag each existing LLM call with its feature label:

• apps/api/src/applications/applications.service.ts
• apps/api/src/keywords/keywords.service.ts
• apps/api/src/interviews/services/*.ts
• apps/api/src/resume-parser/resume-parser.service.ts

4. Docs

• Update ARCHITECTURE.md, README.md, and .github/copilot-instructions.md (new model + env var) per the repo's mandatory documentation-sync rule.

Out of scope (follow-up issues)

• ML model trained on the dataset.
• Admin analytics dashboard / aggregation endpoints.
• Anonymized dataset export pipeline.

Privacy / GDPR notes

• DB is Neon Postgres in EU/Frankfurt.
• No PII, no prompt/response content, no User FK — rows are anonymous and do not need to be deleted on account erasure.
• Salt stored only as a Fly secret (flyctl secrets set LLM_USAGE_HASH_SALT=...).

Acceptance criteria

• Every LLM call (all providers, success + failure) writes exactly one LlmUsageEvent with no User FK.
• Azure OpenAI token counts are captured from the real usage field.
• No prompt/response text is ever persisted.
• actorHash is a salted SHA-256 and stable per user.
• Migration generated forward-only; docs updated in the same PR.