feat(tokens): unify counting via ports.Tokenizer port

[pocky]

Summary

• Unified token counting for all CLI-based agent providers (Claude, Gemini, Codex, Copilot, OpenCode) behind an injected ports.Tokenizer interface, replacing scattered estimateTokens/estimateInputTokens inline helpers
• Each provider now exposes an extractTokenUsage hook that pulls real token counts from its JSON output; a fallbackTokenizer (len/4) is used only when the provider does not emit token data, with TokensEstimated=true to signal approximation
• Adds TokensInput, TokensOutput, and TokensEstimated fields to step state, making them accessible as interpolation variables in workflow YAML
• Removes the tiktoken-go dependency entirely and deletes the tiktoken_tokenizer implementation; the ports.Tokenizer port is the single injection point for future real tokenizer swaps

Changes

Domain

• internal/domain/workflow/context.go: Add TokensInput, TokensOutput, TokensEstimated fields to StepState
• internal/domain/workflow/reference.go: Register new token fields in ValidStateProperties and lowercaseToUppercase alias map

Application

• internal/application/execution_service.go: Propagate TokensInput, TokensOutput, TokensEstimated from conversation and single-turn results into step state
• internal/application/interpolation_helpers.go: Include new token fields when building interpolation context from step state

Infrastructure — base provider

• internal/infrastructure/agents/base_cli_provider.go: Add tokenizer ports.Tokenizer field to baseCLIProvider; default to fallbackTokenizer; add extractTokenUsage hook to cliProviderHooks; replace estimateTokens/estimateInputTokens calls with tokenizer calls in both execute and executeConversation
• internal/infrastructure/agents/base_cli_provider_tokenizer_test.go: New — 390-line test suite covering tokenizer injection, IsEstimate propagation, CountTurnsTokens slicing, no-mutation guarantee on prior turns, and error-path guard

Infrastructure — per-provider hooks

• internal/infrastructure/agents/claude_provider.go: Add tokenizer field; wire extractClaudeTokenUsage hook (parses result event usage, including cache tokens and total_cost_usd)
• internal/infrastructure/agents/gemini_provider.go: Add tokenizer field; wire extractGeminiTokenUsage hook (parses result event stats)
• internal/infrastructure/agents/codex_provider.go: Add tokenizer field; wire extractCodexTokenUsage hook (parses turn.completed event usage)
• internal/infrastructure/agents/copilot_provider.go: Add tokenizer field; wire extractCopilotTokenUsage hook (parses assistant.message event outputTokens)
• internal/infrastructure/agents/opencode_provider.go: Add tokenizer field; wire extractOpenCodeTokenUsage hook (parses step_finish event part.tokens)
• internal/infrastructure/agents/helpers.go: Remove dead estimateTokens and estimateInputTokens helpers; add intFromMap utility used by extraction hooks
• internal/infrastructure/agents/helpers_test.go: Remove tests for deleted helpers
• internal/infrastructure/agents/options.go: Add tokenizer-related provider option constants
• internal/infrastructure/agents/provider_options_test.go: Expand option tests to cover tokenizer injection

Removed

• internal/infrastructure/tokenizer/tiktoken_tokenizer.go: Deleted — tiktoken implementation removed; ports.Tokenizer is now the extension point
• internal/infrastructure/tokenizer/tiktoken_tokenizer_test.go: Deleted — accompanying tests

Interpolation

• pkg/interpolation/reference.go: Register TokensInput, TokensOutput, TokensEstimated in ValidStateProperties
• pkg/interpolation/resolver.go: Map new token fields into StepStateData during resolution
• pkg/interpolation/reference_json_field_test.go: Update fixture expectations for new fields
• pkg/interpolation/reference_test.go: Update reference validation tests

Dependencies

• go.mod: Remove tiktoken-go, glamour, and several transitive dependencies pulled in by tiktoken
• go.sum: Remove corresponding checksums

Docs

• docs/development/creating-agent-provider.md: New — comprehensive guide for implementing a new agent provider (hooks, token extraction, testing)
• docs/reference/interpolation.md: Document TokensInput, TokensOutput, TokensEstimated variables; add per-provider source table
• docs/user-guide/agent-steps.md: Update token tracking section with new fields, TokensEstimated semantics, and per-provider source table
• docs/development/architecture.md: Update tokenizer package description
• docs/development/project-structure.md: Update tokenizer package annotation

Project config

• CHANGELOG.md: Add F094 entry under Unreleased
• CLAUDE.md: Add nolint:errcheck replication rule; remove stale pitfall entry

Test plan

• make build — binary compiles with tiktoken dependency removed
• make test — all unit and integration tests pass, including new tokenizer tests in base_cli_provider_tokenizer_test.go
• make lint — zero violations; nolint:errcheck directives present with matching comments across all providers
• Run a workflow with a Claude agent step and verify {{.states.step.TokensInput}}, {{.states.step.TokensOutput}}, and {{.states.step.TokensEstimated}} interpolate correctly in a downstream step

Closes #339

---

Generated with awf commit workflow