Token Management

Token Management

Understanding and managing tokens in Flutter AI SDK.

What are Tokens?

Tokens are pieces of words that AI models process. Roughly:

• 1 token ≈ 4 characters in English
• 100 tokens ≈ 75 words

Token Usage

Accessing Usage

final response = await ai.chat('Hello!');

if (response.usage != null) {
  print('Prompt tokens: ${response.usage!.promptTokens}');
  print('Completion tokens: ${response.usage!.completionTokens}');
  print('Total tokens: ${response.usage!.totalTokens}');
}

Usage Object

class Usage {
  final int promptTokens;      // Tokens in your input
  final int completionTokens;  // Tokens in the response
  final int totalTokens;       // Sum of both
}

Token Limits by Model

OpenAI

Model	Context Window	Max Output
gpt-4-turbo	128,000	4,096
gpt-4	8,192	8,192
gpt-4-32k	32,768	32,768
gpt-3.5-turbo	16,385	4,096
gpt-4o	128,000	4,096

Anthropic

Model	Context Window	Max Output
claude-3-opus	200,000	4,096
claude-3-sonnet	200,000	4,096
claude-3-haiku	200,000	4,096
claude-3.5-sonnet	200,000	8,192

Google AI

Model	Context Window	Max Output
gemini-1.5-pro	2,000,000	8,192
gemini-1.5-flash	1,000,000	8,192
gemini-1.0-pro	32,768	8,192

Estimating Tokens

Before Requests

// Access token counter
final tokenCounter = ai.context.estimatedTokens;
print('Current context tokens: $tokenCounter');

// Check available tokens
final available = ai.context.availableTokens;
print('Available for response: $available');

Rough Estimation

int estimateTokens(String text) {
  // Rough estimate: 1 token ≈ 4 characters
  return (text.length / 4).ceil();
}

// Usage
final estimate = estimateTokens('Hello, how are you today?');
print('Estimated tokens: $estimate'); // ~7 tokens

Setting Max Tokens

final ai = FlutterAI(
  provider: AIProvider.openai,
  config: AIConfig(
    apiKey: 'your-key',
    maxTokens: 500, // Limit response length
  ),
);

Context Window Management

ContextManager Settings

final contextManager = ContextManager(
  maxTokens: 8000,       // Maximum for context
  reservedTokens: 1000,  // Reserved for response
);

Monitoring Context Size

// Before making requests
if (ai.context.estimatedTokens > 6000) {
  print('Warning: Context is getting large');
}

if (ai.context.availableTokens < 500) {
  print('Warning: Limited room for response');
  ai.clearContext();
}

Handling Context Overflow

try {
  await ai.chat(message);
} on AIContextLengthError catch (e) {
  print('Context too long!');
  print('Max: ${e.maxTokens}');
  print('Requested: ${e.requestedTokens}');
  
  // Clear and retry
  ai.clearContext();
  await ai.chat(message);
}

Cost Management

Tracking Usage

class TokenTracker {
  int totalInputTokens = 0;
  int totalOutputTokens = 0;
  
  void track(Usage? usage) {
    if (usage != null) {
      totalInputTokens += usage.promptTokens;
      totalOutputTokens += usage.completionTokens;
    }
  }
  
  double estimateCost(double inputPrice, double outputPrice) {
    return (totalInputTokens / 1000 * inputPrice) +
           (totalOutputTokens / 1000 * outputPrice);
  }
}

// Usage
final tracker = TokenTracker();

final response = await ai.chat('Hello');
tracker.track(response.usage);

// GPT-4-turbo pricing example
final cost = tracker.estimateCost(0.01, 0.03); // $/1K tokens
print('Estimated cost: \$${cost.toStringAsFixed(4)}');

Streaming Token Usage

await for (final chunk in ai.streamChat(message)) {
  if (chunk.isDone && chunk.usage != null) {
    print('Stream used ${chunk.usage!.totalTokens} tokens');
  }
}

Best Practices

1. Set Appropriate Limits

// For concise responses
final config = AIConfig(
  apiKey: 'key',
  maxTokens: 300,
);

// For detailed responses
final config = AIConfig(
  apiKey: 'key',
  maxTokens: 2000,
);

2. Monitor Usage

class AICostMonitor {
  int sessionTokens = 0;
  final int budget = 10000; // Token budget
  
  Future<AIResponse> chat(FlutterAI ai, String message) async {
    if (sessionTokens >= budget) {
      throw Exception('Token budget exceeded');
    }
    
    final response = await ai.chat(message);
    sessionTokens += response.usage?.totalTokens ?? 0;
    
    print('Session tokens: $sessionTokens / $budget');
    return response;
  }
}

3. Optimize Prompts

// Verbose - uses more tokens
final response = await ai.chat('''
Hello! I hope you are doing well today.
I have a question for you that I would like you to answer.
Can you please tell me what the capital of France is?
Thank you very much in advance for your help!
''');

// Concise - uses fewer tokens
final response = await ai.chat('What is the capital of France?');

4. Clear Context Strategically

// After completing a task
ai.clearContext();

// Or periodically
if (ai.conversation.length > 20) {
  // Summarize first
  final summary = await ai.chat(
    'Summarize our conversation briefly',
    addToContext: false,
  );
  
  ai.clearContext();
  ai.context.addUserMessage('Previous context: ${summary.text}');
}

Related

• Context Management - Managing conversation history
• Configuration - Setting maxTokens
• Error Handling - Handling context length errors