[pull] main from Alishahryar1:main - #187
Merged
Merged
Conversation
## Problem OpenAI-compatible Chat providers count cached prompt tokens inside `prompt_tokens`, but FCC reported that inclusive total as ordinary Anthropic input and omitted the cache-read usage field. ## Changes | Before | After | | --- | --- | | Anthropic usage counted standard Chat cache hits as ordinary input. | Anthropic usage separates validated standard Chat cache hits into `cache_read_input_tokens`. | | Missing or inconsistent upstream cache metrics had no generic partition. | Missing or inconsistent metrics retain the existing unsplit usage without guessing. | | Streaming and non-streaming Messages inherited the same incorrect final usage. | Streaming and non-streaming Messages receive the same corrected terminal usage. | <!-- greptile_comment --> <details open><summary><h3>Greptile Summary</h3></summary> This change reports validated OpenAI-compatible cached prompt tokens separately from ordinary input tokens in Anthropic-compatible usage responses, and updates the package version to 5.15.3. The potential for cached tokens to be double-counted, ordinary prompt tokens to be lost, or malformed cache metrics to corrupt final usage was exercised and disproved. Streamed and aggregated non-stream responses with 22 prompt tokens and 15 cached tokens reported 7 input tokens, 15 cache-read input tokens, and 4 output tokens. An invalid cache count of 23 against 22 prompt tokens retained ordinary input usage of 22 without a cache-read field. The focused OpenAI Chat usage tests passed. </details> <h3>Confidence Score: 5/5</h3> The updated usage reporting is safe to merge based on the exercised streaming, non-stream aggregation, and invalid-metric paths. No defects remain in the final review. The key accounting behavior was executed with valid cached usage and an inconsistent cache-token count, and the focused provider test suite passed. **Files Needing Attention:** No files require follow-up attention. <details><summary><h3><a href="https://www.greptile.com/trex"><img alt="T-Rex" src="https://greptile-static-assets.s3.amazonaws.com/trex/trex_green.svg" height="20" align="absmiddle"></a> T-Rex Logs</h3></summary> **What T-Rex did** - Ran an executable mocked-provider validation to exercise valid streamed usage, valid aggregated non-stream usage, and a malformed cached-token scenario. - Validated that valid usage with prompt\_tokens=\[REDACTED\] and cached\_tokens=\[REDACTED\] produced input\_tokens=\[REDACTED\], cache\_read\_input\_tokens=\[REDACTED\], and output\_tokens=\[REDACTED\] in both delivery modes, and that an invalid cached\_tokens=\[REDACTED\] value against prompt\_tokens yielded ordinary input\_tokens=\[REDACTED\] without a cache field. - Ran the focused OpenAI Chat usage suite, which completed with 31 tests passed. - Collected and reviewed artifacts, including the mocked-provider validation source, the cache-usage validation output, and the focused OpenAI Chat suite output. <a href="https://app.greptile.com/trex/runs/21161145/artifacts"><picture><source media="(prefers-color-scheme: dark)" srcset="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifactsDark.svg?v=4"><source media="(prefers-color-scheme: light)" srcset="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"><img alt="View all artifacts" src="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"></picture></a> <sub><a href="https://www.greptile.com/trex"><img alt="T-Rex" src="https://greptile-static-assets.s3.amazonaws.com/trex/trex_green.svg" height="14" align="absmiddle"></a> Ran code and verified through T-Rex</sub> </details> <sub>Reviews (1): Last reviewed commit: ["fix: report chat cache hits in Anthropic..."](b680f05) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=57758076)</sub> <!-- /greptile_comment -->
![https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifactsDark.svg?v=4"><source](https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifactsDark.svg?v=4"><source){kind=link}
![https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"><img](https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"><img){kind=link}
![https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"></picture></a](https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"></picture></a){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )