High token usage (~500k tokens) during initial pipeline/scan sequence on Claude Pro plan #1089
|
Hi everyone, Newbie here, looking for some advice on token optimization with career-ops. I set up the project last night and focused my configuration around Project Manager and Product Manager roles in Australia. My target sources include major Australian job boards, defence roles, and a substantial list of ASX-listed companies. I'm running this via Claude Code in the terminal. This morning, I walked through the following sequence in a single session:
The Issue: A few questions for the community: Is the agent over-scanning? Could Claude Code be reading the entire DOM/history of the targeted Australian job boards or ASX career portals during the Scan or Pipeline steps? Configuration size: Did I potentially add too many targets to my portals.yml / configuration file for an active terminal session to handle efficiently? Best practice workflow: Is there a more token-efficient way to handle daily scans and single-URL evaluations? Should I be separating the Scan commands from the individual evaluation (oferta) sessions to prevent context window bloating? |
Replies: 5 comments 6 replies
|
The 500k is almost certainly the Scan step, not the single-URL eval. When an agent fetches a job board or careers portal, it pulls the whole rendered DOM into context, and a "scan all targets" pass does that per source in one session. With a big ASX target list, that's dozens of full-page reads stacked into one context window, and it compounds because each later step still carries the earlier pages. Three things that help:
The framing that fixed this for me is treating context as a budget you spend per source, with a cap, not something that's free until it isn't. I maintain a reference architecture that documents this pattern: https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#9-context-is-a-budget-not-a-constant |
|
Thanks @jimy-r, appreciate the time to respond, and for the advice. |
|
@jimy-r nailed the diagnosis — the Scan step is almost always where the tokens go, not the single-URL eval. When career-ops scans a board the agent pulls the rendered page into context, and a big target list (your ASX set + defence + boards) means dozens of full-page reads stacked into one session. Running Pipeline → Oferta → Scan → Pipeline in a single session compounds it, because each step also re-reads the context that has already accumulated. A few concrete levers, roughly in order of impact: 1. Don't run the whole sequence in one session. Each phase re-reads the growing context, so a long chain pays for the same tokens repeatedly. Run scan, then start a fresh session (or 2. Use batch mode for evaluation. 3. Offload the heavy scoring to the free Gemini tier. 4. Scan in smaller passes. A huge ASX list scanned in one go is the worst case for context size. Split your 5. Filter before you evaluate. Let scan plus the min-score gate prune the list, so you only spend evaluation tokens on the roles worth a full read, not every listing. Token efficiency is the single thing people raise most, so this kind of detailed report genuinely helps — thank you for it. For your setup the biggest win is probably #1 + #2: scan separately, then batch-evaluate in a fresh session. |
|
Thanks @santifer, amazing idea youve created here. |
|
Hey @WindHaze20 — how is Claude Code scaling on your career-ops repo? Are you hitting those weird context walls where it starts losing track of files, or is it staying snappy? |
|
I've been having some trouble with bulk processing, example (running pdf/cover with multiple agents on 15 jobs). When i run bulk teh context falls apart. I get mixed up job titles, non-cronological dates adn format reverts back to American english and includes em-dashes. <y token usage was also huge on that request 2 x 5hr blocks in about 15min. When i run a single job, everythign works as expected. |
|
@WindHaze20 The bulk failure you're describing is context bleed, and the tells in your report line up exactly. Mixed-up job titles mean the 15 jobs are sharing one accumulating context, so late items pick up fragments of earlier ones. The format drift is the same failure seen from the other side. Your spelling and punctuation rules are a fixed-size instruction in a context that keeps growing, and once the job data swamps them the model reverts to its defaults, which are American English and em-dashes. Single-job runs work because the instruction-to-content ratio stays healthy. The fix is the batch mode santifer described above. One isolated worker per job, where each worker sees only its own posting, your resume and the template, and none of the other 14 jobs. That removes the cross-contamination and the drift at the same time, and the token bill should drop from compounding to roughly linear, since no worker re-reads the pile the previous ones built up. |
|
Late follow-up, and worth one since this thread gets found by people hitting the same wall. @Amar07Singh — on context walls specifically: the honest answer is that the wall is real and it is not really about repo size, it is about how much rendered page text a single session accumulates. career-ops itself is prompts plus scripts; what fills a context window is a scan pulling dozens of full pages into it. So the fix is not a bigger model, it is keeping the page-heavy work out of the agent. Three things exist today that did not when this thread started, all of them free:
The habit that helps most is still the boring one @jimy-r pointed at: do not chain Pipeline → Oferta → Scan → Pipeline in one session. Each step keeps carrying the earlier pages. Separate sessions cost nothing and reset the accumulation. @WindHaze20 — if you are still running the ASX list, I would like to know whether the economy tier plus the script-based scan brought it under your Pro limit. That is the number I cannot generate myself. |
|
@santifer I I gave up on scan entirely, most results were expired or stale links. Instead I switched to pasting in URLs from job ads I'd found and liked. If a role scored well, I'd run it through pdf then cover. That workflow works really well, and usage limits never became an issue. One thing worth flagging: I evaluated 40+ jobs in a single session one evening and hit context bleed again, unreliable evaluations, broken CV PDFs, wrong dates and titles. The fix is simple: keep runs to 10 roles max, then reset the session and start fresh. |
|
@WindHaze20 Good data point, and the 10-role cap works for a reason worth naming. Context bleed starts when the accumulated job text swamps your fixed instructions, so the thing you are actually capping is tokens per session, not roles. Ten is a solid default for typical ad lengths. A batch of long JDs will hit the wall sooner and short ones later, so if failures ever come back at ten, halve the batch before assuming the workflow broke. If you note the rough length of the ads in a failing batch once, the safe number becomes obvious. |
|
"High token usage (~500k tokens) during initial pipeline/scan sequence on Claude Pro plan" is useful as a cost surface because it turns model spend into a workflow-level measurement problem, not just a monthly invoice. A useful shape would be one usage event per model call, with provider/model, feature name, token fields, retry count, latency, and billing bucket all attached. Then cost can be traced back to product behavior instead of guessed from invoices. I am testing a multi-model OpenAI-compatible API layer around official Chinese models, so the routing and usage-accounting parts of this are directly relevant to me. In this case, is the main pain attribution by user/workspace, or just understanding which agent steps are consuming the most tokens? |
@jimy-r nailed the diagnosis — the Scan step is almost always where the tokens go, not the single-URL eval. When career-ops scans a board the agent pulls the rendered page into context, and a big target list (your ASX set + defence + boards) means dozens of full-page reads stacked into one session. Running Pipeline → Oferta → Scan → Pipeline in a single session compounds it, because each step also re-reads the context that has already accumulated.
A few concrete levers, roughly in order of impact:
1. Don't run the whole sequence in one session. Each phase re-reads the growing context, so a long chain pays for the same tokens repeatedly. Run scan, then start a fresh session (or
/clear) bef…