fix: track OpenAI chat completions parse calls#618
Open
marandaneto wants to merge 2 commits into
Open
Conversation
Member
Author
|
@PostHog/team-ai-observability pls double check this since i dont know much about those wrappers, i just let an agent figure it out but i think it does what's supposed to do |
Contributor
Prompt To Fix All With AIFix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
posthog/test/ai/openai/test_openai.py:1196-1204
**Non-parameterized fallback warning test**
The change adds `warn_on_fallback` to eight wrapper classes (`WrappedResponses`, `WrappedChat`, `WrappedCompletions`, `WrappedEmbeddings`, `WrappedBeta`, `WrappedBetaChat`, `WrappedBetaCompletions` — in both sync and async variants), but the test only exercises `WrappedChat`. Per the team's preference for parameterised tests, this should cover a representative set of wrapper classes so a typo or missing call in any one of them is caught.
### Issue 2 of 2
posthog/ai/openai/wrapper_utils.py:4-5
**Module-level mutable set leaks across test runs**
`_fallback_warnings` is initialised once at import time and never cleared. `test_fallback_logs_warning` passes today because `"posthog_unwrapped_test_attribute"` is a unique key, but if the same test is re-executed in the same process (e.g. `pytest-repeat`, `pytest-xdist` with `--looponfail`, or a future test that happens to reuse the same combination), the set already contains the key, the `log.warning` call is skipped, and the caplog assertion fails. Exposing a way to reset the set (or keying it per-client instance) would make the deduplication safe to test.
Reviews (1): Last reviewed commit: "fix openai chat completions parse tracki..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
posthog-python Compliance ReportDate: 2026-05-22 12:08:39 UTC
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 518ms |
| Format Validation.Event Has Uuid | ✅ | 1507ms |
| Format Validation.Event Has Lib Properties | ✅ | 1507ms |
| Format Validation.Distinct Id Is String | ✅ | 1508ms |
| Format Validation.Token Is Present | ✅ | 1506ms |
| Format Validation.Custom Properties Preserved | ✅ | 1507ms |
| Format Validation.Event Has Timestamp | ✅ | 1507ms |
| Retry Behavior.Retries On 503 | ✅ | 9515ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 3510ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 3507ms |
| Retry Behavior.Respects Retry After Header | ✅ | 9510ms |
| Retry Behavior.Implements Backoff | ✅ | 23525ms |
| Retry Behavior.Retries On 500 | ✅ | 7509ms |
| Retry Behavior.Retries On 502 | ✅ | 7512ms |
| Retry Behavior.Retries On 504 | ✅ | 7512ms |
| Retry Behavior.Max Retries Respected | ✅ | 23533ms |
| Deduplication.Generates Unique Uuids | ✅ | 1497ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 7513ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 14515ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 7512ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 1508ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 1507ms |
| Compression.Sends Gzip When Enabled | ✅ | 1508ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 1507ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 1005ms |
| Batch Format.Multiple Events Batched Together | ✅ | 1506ms |
| Error Handling.Does Not Retry On 403 | ✅ | 3509ms |
| Error Handling.Does Not Retry On 413 | ✅ | 3506ms |
| Error Handling.Retries On 408 | ✅ | 7511ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ❌ | 509ms |
| Request Payload.Flags Request Uses V2 Query Param | ❌ | 300872ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ❌ | 301031ms |
| Request Payload.Flags Request Omits Authorization Header | ❌ | 300945ms |
| Request Payload.Token In Flags Body Matches Init | ❌ | 301024ms |
| Request Payload.Groups Round Trip | ❌ | 300998ms |
| Request Payload.Groups Default To Empty Object | ❌ | 301026ms |
| Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It | ❌ | 300990ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ❌ | 300962ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 301024ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ❌ | 300992ms |
| Request Lifecycle.No Flags Request On Init Alone | ❌ | 301019ms |
| Request Lifecycle.No Flags Request On Normal Capture | ❌ | 301020ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ❌ | 300997ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ❌ | 300926ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ❌ | 301062ms |
Failures
request_payload.request_with_person_properties_device_id
Field 'token' not found in /flags request body at path 'token'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'geoip_disable', 'device_id', 'flag_keys_to_evaluate', 'sentAt', 'api_key']
request_payload.flags_request_uses_v2_query_param
No error message
request_payload.flags_request_hits_flags_path_not_decide
No error message
request_payload.flags_request_omits_authorization_header
No error message
request_payload.token_in_flags_body_matches_init
No error message
request_payload.groups_round_trip
No error message
request_payload.groups_default_to_empty_object
No error message
request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it
No error message
request_payload.disable_geoip_false_propagates_as_geoip_disable_false
No error message
request_payload.disable_geoip_omitted_defaults_to_false
No error message
request_payload.flag_keys_to_evaluate_contains_only_requested_key
No error message
request_lifecycle.no_flags_request_on_init_alone
No error message
request_lifecycle.no_flags_request_on_normal_capture
No error message
request_lifecycle.two_flag_calls_produce_two_remote_requests
No error message
request_lifecycle.mock_response_value_is_returned_to_caller
No error message
side_effect_events.get_feature_flag_captures_feature_flag_called_event
No error message
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
Fixes #614.
OpenAI
chat.completions.parsecalls were falling through to the raw OpenAI SDK object, so PostHog tracking was skipped andposthog_*arguments could be forwarded to OpenAI and crash. Unsupported fallback wrapper access now also logs a warning when tracking is bypassed.💚 How did you test it?
uv run ruff check posthog/ai/openai/openai.py posthog/ai/openai/openai_async.py posthog/ai/openai/wrapper_utils.py posthog/test/ai/openai/test_openai.pyuv run --extra test pytest posthog/test/ai/openai/test_openai.py -q📝 Checklist
If releasing new changes
sampo addto generate a changeset file