Skip to content

feat: add token counting (OpenAI tiktoken + Claude API) #20

@kiki830621

Description

@kiki830621

Summary

Add macdoc convert --to tokens <file> that outputs exact token counts.

Depends on #27 (CLI migration to convert --to pattern).

CLI Interface

macdoc convert --to tokens file.md                        # both models
macdoc convert --to tokens --model gpt-4o file.md         # OpenAI only
macdoc convert --to tokens --model claude-sonnet file.md   # Claude only

Output Format

# Default (tabular)
Model              Tokens
─────────────────────────
gpt-4o             1,234
claude-sonnet-4-6  1,198

# Single model (just the number for piping)
1234

Token Counting Methods

Provider Method Cost
OpenAI swift-tiktoken (offline BPE) Free, offline
Claude Anthropic /v1/messages/count_tokens API Free, online (RPM-limited)

Architecture

  • New package: token-counter-swift under packages/
  • Layer 2 utility (measurement tool, not format converter)
  • Depends on: swift-tiktoken, Foundation (URLSession for Anthropic API)
  • Needs ANTHROPIC_API_KEY env var for Claude counting

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions