Repository map token limit not respected

[hzappen]

Issue

I am using aider to work on a "larger" git repository.
I noticed very high input token usage when sending queries to claude-3.5-sonnet.
Further investigations lead me to believe that aider is currently not respecting the repository map token limit.
Consider the below snippet, in which i call aider explicitly with "--map-tokens 1024", however the "/tokens" command still reports it to use over 16.000 tokens for the repository map.
The value reported is consistent with the API usage reports i get from the Anthropic API console.
I tried deleting the ".aider.tags.cache.v3" directory within the repository folder.
This lets aider regenerate the repository map cache, however the token usage stays the same.
Setting the map-tokens config key in the ".aider.conf.yml" file in my homedir also does not change the behavior.
Another symptom possibly related is the very slow autocomplete when adding files to the chat. Sometimes it takes several seconds before the list with the suggestions is generated.

PS C:\Data\repos\MyLargeProject> aider --map-tokens 1024
Aider v0.40.6
Models: claude-3-5-sonnet-20240620 with diff edit format, weak model claude-3-haiku-20240307
Git repo: .git with 552 files
Repo-map: using 1024 tokens
Use /help to see in-chat commands, run with --help to see cmd line args
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> /tokens

Approximate context window usage, in tokens:

$ 0.0034    1,133 system messages
$ 0.0493   16,419 repository map  use --map-tokens to resize
==================
$ 0.0527   17,552 tokens total
          182,448 tokens remaining in context window
          200,000 tokens max context window size
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

Version and model info

Aider v0.40.6
Models: claude-3-5-sonnet-20240620 with diff edit format, weak model claude-3-haiku-20240307
Git repo: .git with 552 files

[hzappen]

I did some further testing and found that the "map-tokens" parameter indeed does do something. The repository map size reported by the tokens command approximately halves/doubles when i set "map-tokens" to 512 or 2048:

PS C:\Data\repos\MyLargeProject> aider --map-tokens 512
Aider v0.40.6
Models: claude-3-5-sonnet-20240620 with diff edit format, weak model claude-3-haiku-20240307
Git repo: .git with 552 files
Repo-map: using 512 tokens
Use /help to see in-chat commands, run with --help to see cmd line args
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> /tokens

Approximate context window usage, in tokens:

$ 0.0034    1,133 system messages
$ 0.0246    8,210 repository map  use --map-tokens to resize
==================
$ 0.0280    9,343 tokens total
          190,657 tokens remaining in context window
          200,000 tokens max context window size
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

PS C:\Data\repos\MyLargeProject> aider --map-tokens 2048
Aider v0.40.6
Models: claude-3-5-sonnet-20240620 with diff edit format, weak model claude-3-haiku-20240307
Git repo: .git with 552 files
Repo-map: using 2048 tokens
Use /help to see in-chat commands, run with --help to see cmd line args
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> /tokens

Approximate context window usage, in tokens:

$ 0.0034    1,133 system messages
$ 0.0984   32,791 repository map  use --map-tokens to resize
==================
$ 0.1018   33,924 tokens total
          166,076 tokens remaining in context window
          200,000 tokens max context window size
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

[paul-gauthier]

Thanks for trying aider and filing this issue.

The --map-tokens option does not specify a hard limit. Aider adjusts the size of the repo map dynamically based on the state of the chat. It will usually stay within that setting's value. But it does expand it significantly at times, especially when no files have been added to the chat and aider needs to understand the entire repo as best as possible.

[paul-gauthier]

FYI, the version in the main branch (not yet released) uses somewhat less tokens for repomap.

[hzappen]

First, thank you very much for providing this awesome project! And also for the very quick response.

I just tried out the current main branch, and indeed it roughly halves the repo map token usage in my case.

May i suggest updating the documentation to reflect that the value set by --map-tokens is not a hard limit?
Currently, it states (https://aider.chat/docs/repomap.html#optimizing-the-map):

Aider optimizes the repo map by selecting the most important parts of the codebase which will fit into the token budget assigned by the user (via the --map-tokens switch, which defaults to 1k tokens).

[paul-gauthier]

Thanks for pointing out that misleading section of the docs. They've been updated and are rebuilding now.

[paul-gauthier]

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.

[capybarahero]

Hi, @paul-gauthier, apologies for reopening this issue.

qq: Have you considered having a flag to set a hard limit for the repository map?

I've also been experiencing high token usage on a large repo and I'd prefer a way to limit my costs even if it impacts performance.

Thanks!