Hitting Anthropic API Rate Limits

[NeutronicMC]

Are there any plans to add a context compacting feature or something like it?

I have tier 3 Anthropic api rate limits, and on small to mid-size projects I'm hitting the max input tokens per minute pretty regularly (at a total context size of around 30-50k tokens). Ultimately for me this means that it'll only run for around 10 minutes before I need to start a new session. I've considered Claude Max, as it seems my Claude pro account doesn't have this same limitation until it hits the hard cap, but I don't use $100 in api credits per month yet so it's a hard sell.

[adamdotdevin]

we already auto-compact, but i need to add the explicit command, will do so today

[luuk00101]

This is WAY worse on the API, it really pushes me towards Claude Pro, but when you say you're hitting limits even there... Can't justify spending THAT much just on AI tools just yet...

[adamdotdevin]

@NeutronicMC what version of opencode are you on? we made a ton of improvements to token efficiency yesterday

[NeutronicMC]

@NeutronicMC what version of opencode are you on? we made a ton of improvements to token efficiency yesterday

v0.1.76

[thdxr]

can you hit the limit and then send me a /share url

[NeutronicMC]

Here you go: https://dev.opencode.ai/s/501e5869
This also occurs fairly often with 3.5 Sonnet on the 160k input tokens per minute limit, but it would've taken a little longer to reproduce.

As a side note, it'd be nice if /share copied the link to your clipboard.

[thdxr]

yeah so looking at this it's because of the effect mcp doc tool call

it's quite expensive to load into context so you're hitting rate limits. although you said you have tier 3 this looks more like the 20K a minute rate limit

[NeutronicMC]

Yeah, I used the Effect MCP to fill up the context fast, but in pre-existing code bases (without effect) when it reads a lot of files it'll do the same thing. It was very difficult to work around on tier 2 though. You could get it to hit the rate limit in this codebase pretty quickly I think.

[NeutronicMC]

yeah so looking at this it's because of the effect mcp doc tool callit's quite expensive to load into context so you're hitting rate limits. although you said you have tier 3 this looks more like the 20K a minute rate limit

Yeah, I'm not entirely sure what's going on under the hood but I'm 100% certain that my rate limit is 80k input a minute, and I'm using an API key connected to that account. Here's what it looks like from the dashboard:

That specific chat that I shared had 40k total tokens in the context, but I can hit the rate limit of 160k on 3.5 sonnet the same way with only 40k - 50k tokens.

Edit: The new compact command does help quite a bit, although it'd still be nice if that could be managed automatically somehow.

[adamdotdevin]

yeah so looking at this it's because of the effect mcp doc tool callit's quite expensive to load into context so you're hitting rate limits. although you said you have tier 3 this looks more like the 20K a minute rate limit

Yeah, I'm not entirely sure what's going on under the hood but I'm 100% certain that my rate limit is 80k input a minute, and I'm using an API key connected to that account. Here's what it looks like from the dashboard:

That specific chat that I shared had 40k total tokens in the context, but I can hit the rate limit of 160k on 3.5 sonnet the same way with only 40k - 50k tokens.

Edit: The new compact command does help quite a bit, although it'd still be nice if that could be managed automatically somehow.

compact is called automatically, but not until you hit a threshold that's close to the context window limit

@thdxr we should make the threshold configurable

[mhamza15]

This session hit the limit pretty quick: https://opencode.ai/s/ece77cea

I haven't really ever hit the limit with Claude Code when using it for similar usecases

[thdxr]

try v0.1.117 - did some large optimizations

[NeutronicMC]

try v0.1.117 - did some large optimizations

This is great, ran it with the previous example with the same prompt and it was able to get all the way through, and managed to hit $20 which is a new record for me! I ended up switching to Claude Max anyway, but this will be nice if I ever manage to hit that rate limit.