Skip to content

test(flags): make wrong-key load_feature_flags deterministic#432

Merged
andehen merged 6 commits intomasterfrom
fix-broken-CI-test
Feb 13, 2026
Merged

test(flags): make wrong-key load_feature_flags deterministic#432
andehen merged 6 commits intomasterfrom
fix-broken-CI-test

Conversation

@andehen
Copy link
Contributor

@andehen andehen commented Feb 13, 2026

Summary

Stabilize feature-flag tests in CI and remove runtime-dependent behavior in capture cache handling.

Example failure run where 3.10 and 11 passes, but 13 and 14 fails https://github.com/PostHog/posthog-python/actions/runs/21986040963/job/63520352185?pr=431

Changes

  • Make test_load_feature_flags_wrong_key deterministic by mocking posthog.client.get to raise APIError(401, "Unauthorized") and mocking posthog.client.Poller.
  • Make test_capture_multiple_users_doesnt_out_of_memory deterministic by setting client.distinct_ids_feature_flags_reported.max_size = 100 directly on the instance (with an explanatory comment).
  • Update Client._capture_feature_flag_called to avoid implicit defaultdict missing-key insertion and use explicit get/assign before adding reported flag keys.

Why

CI reproduced a non-deterministic failure in capture-cache size assertions (101 != 1) that did not consistently reproduce locally. These changes remove reliance on runtime-specific insertion behavior and network-dependent test setup.

Validation

  • pytest -q posthog/test/test_feature_flags.py -k "test_load_feature_flags_wrong_key"
  • pytest -q posthog/test/test_feature_flags.py -k "test_capture_multiple_users_doesnt_out_of_memory"
  • pytest -q posthog/test/test_feature_flags.py -k "TestCaptureCalls"

@github-actions
Copy link
Contributor

github-actions bot commented Feb 13, 2026

posthog-python Compliance Report

Date: 2026-02-13 13:43:18 UTC
Duration: 159369ms

✅ All Tests Passed!

29/29 tests passed


Capture Tests

29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields 517ms
Format Validation.Event Has Uuid 1507ms
Format Validation.Event Has Lib Properties 1507ms
Format Validation.Distinct Id Is String 1506ms
Format Validation.Token Is Present 1507ms
Format Validation.Custom Properties Preserved 1507ms
Format Validation.Event Has Timestamp 1507ms
Retry Behavior.Retries On 503 9514ms
Retry Behavior.Does Not Retry On 400 3510ms
Retry Behavior.Does Not Retry On 401 3507ms
Retry Behavior.Respects Retry After Header 9514ms
Retry Behavior.Implements Backoff 23520ms
Retry Behavior.Retries On 500 7509ms
Retry Behavior.Retries On 502 7514ms
Retry Behavior.Retries On 504 7513ms
Retry Behavior.Max Retries Respected 23528ms
Deduplication.Generates Unique Uuids 1497ms
Deduplication.Preserves Uuid On Retry 7514ms
Deduplication.Preserves Uuid And Timestamp On Retry 14513ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 7516ms
Deduplication.No Duplicate Events In Batch 1503ms
Deduplication.Different Events Have Different Uuids 1506ms
Compression.Sends Gzip When Enabled 1507ms
Batch Format.Uses Proper Batch Structure 1507ms
Batch Format.Flush With No Events Sends Nothing 1005ms
Batch Format.Multiple Events Batched Together 1505ms
Error Handling.Does Not Retry On 403 3508ms
Error Handling.Does Not Retry On 413 3508ms
Error Handling.Retries On 408 7514ms

@andehen andehen requested review from a team February 13, 2026 13:41
@posthog-project-board-bot posthog-project-board-bot bot moved this to In Review in Feature Flags Feb 13, 2026
@andehen
Copy link
Contributor Author

andehen commented Feb 13, 2026

@greptile-apps please review

@github-project-automation github-project-automation bot moved this from In Review to Approved in Feature Flags Feb 13, 2026
@andehen andehen merged commit bb0c7b4 into master Feb 13, 2026
23 checks passed
@andehen andehen deleted the fix-broken-CI-test branch February 13, 2026 14:53
@github-project-automation github-project-automation bot moved this from Approved to Done in Feature Flags Feb 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants