Impose Limit on input tokens given to the LLM

[duaraghav8]

• check the size of Dockerfile + package.json + directory tree being sent as initial user prompt
• check size of files being sent back as read_files tool response

Impose token limits on these to make sure the end user's costs are more predictable.

[aryabharat]

Hi @duaraghav8 what should be the limit for tokens? should it be user configurable ?
I can pick this.

[duaraghav8]

@aryabharat a default of 3K input tokens is a reasonable limit IMO.
And yes, it should be configurable by the user
Let me know in case you need any clarifications from me. Looking forward to your PR!

[aryabharat]

Hi @duaraghav8
Thanks for your response!

I have a few questions—should we set the token limit per file, or should there be a limit on the total tokens?

dirTreeTokens := countTokens(dirTree)
packageJSONTokens := countTokens(packageJSON)
totalTokens := dirTreeTokens + packageJSONTokens

[duaraghav8]

@aryabharat A limit on total tokens should be good for now.
Normally, we do expect that code files will be much larger compared to Dockerfile or package.json. But we don't want to be restrictive about any particular files contents.

[aryabharat]

@aryabharat A limit on total tokens should be good for now. Normally, we do expect that code files will be much larger compared to Dockerfile or package.json. But we don't want to be restrictive about any particular files contents.

ok.