Add LLM token estimation for file selection output

[Copilot]

Print estimated token count for each file added to context and total tokens used.

Implementation

• Token estimation: ~4 chars per token heuristic using utf8.RuneCountInString()
• Per-file output: Display tokens alongside existing file inclusion messages
• Total output: Print aggregate count after all files processed
• Template expansion: Count tokens after variable substitution

Output Format

Using persona file: .prompts/personas/expert.md (~35 tokens)
Including memory file: .prompts/memories/setup.md (~30 tokens)
Using task file: .prompts/tasks/implement.md (~46 tokens)
Total estimated tokens: 111

Files

• token_counter.go: Estimation function
• token_counter_test.go: Unit tests covering edge cases
• main.go: Integration at persona/memory/task processing points
• integration_test.go: End-to-end validation

Original prompt

Each time a file is selected and added. Print an estimate of the number of LLM tokens used. Also print the total number of tokens used.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[alexec]

@copilot Please change

[alexec]

Should say "task" not "prompt"

[Copilot]

Changed to "Using task file:" in commit ad6de93.

[alexec]

Remove the zero check. This will never happen.

[Copilot]

Removed the zero check in commit ad6de93.

[alexec]

@copilot Make the requested changes

[Copilot]

@copilot Make the requested changes

I've already addressed the two review comments in commit ad6de93:

1. Changed "Using prompt file:" to "Using task file:"
2. Removed the unnecessary zero check from estimateTokens()

Could you clarify what additional changes you'd like me to make?

[Copilot]

Pull Request Overview

This PR adds token counting functionality to estimate the number of LLM tokens in processed files. The implementation uses a simple heuristic of approximately 4 characters per token.

• Adds estimateTokens() function with UTF-8 character counting
• Updates output messages to display token counts for persona, memory, and task files
• Adds total token count tracking and display

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
token_counter.go	Implements the token estimation function using a 4-char-per-token heuristic
token_counter_test.go	Adds unit tests for the token estimation function
main.go	Integrates token counting into file processing workflow with per-file and total counts
integration_test.go	Adds end-to-end test verifying token counting output

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Lines 1356-1358 already verify that totalLine contains 'Total estimated tokens:' (since it's found by searching for that string in lines 1352-1354). The check on lines 1361-1363 is redundant and can be removed.

Suggested change

	if !strings.Contains(totalLine, "Total estimated tokens:") {
	t.Errorf("Expected 'Total estimated tokens:' in output, got: %s", totalLine)
	}