Skip to content

fix(tests): environment-agnostic cassette normalization and sorting#1521

Merged
jaceksan merged 2 commits intomasterfrom
fix/cassette-normalization-and-sorting
Apr 9, 2026
Merged

fix(tests): environment-agnostic cassette normalization and sorting#1521
jaceksan merged 2 commits intomasterfrom
fix/cassette-normalization-and-sorting

Conversation

@tychtjan
Copy link
Copy Markdown
Contributor

@tychtjan tychtjan commented Apr 9, 2026

Problem

When VCR cassettes are recorded locally (Docker Compose) vs on staging, the resulting YAML files differ in ways that have nothing to do with actual API changes — causing noisy, hard-to-review PRs.

Root causes identified:

  1. The API returns certain arrays (referenceProperties, edges, workspaceDataFilterReferences, etc.) in non-deterministic order
  2. XLIFF/XML localization groups come back in different order
  3. Export hash placeholders get session-global numeric indices that shift with test execution order
  4. The demo user has profile fields (email, name) locally but not on staging

Changes

Targeted JSON array sorting

Sorts only arrays under 5 specific keys known to be non-deterministic: referenceProperties, edges, workspaceDataFilterColumns, workspaceDataFilterReferences, userGroups. Applied only during YAML serialization — does not affect VCR replay behavior or test assertions.

XML group sorting fix

_sort_xml_groups() was only called for str response bodies. Compressed responses arrive as bytes and were skipped entirely. Now applied in both branches of custom_before_request() and custom_before_response().

Export hash stabilization

Replaces session-global numeric indices (EXECUTION_NORMALIZED_3) with format-based names (EXECUTION_NORMALIZED_CSV, EXPORT_NORMALIZED_XLSX). Each cassette gets self-contained, deterministic placeholders regardless of test execution order.

Demo user profile fields

Added firstname, lastname, email to demo_user.json so make load-staging produces the same user profile as the local Docker bootstrap.

Test plan

  • Recorded all cassettes locally with OVERWRITE — all tests pass
  • Replayed without OVERWRITE — all tests pass (190 SDK + 66 pandas)
  • Verified make clean-staging + make load-staging succeeds and demo user has profile fields
  • Verified targeted sorting only touches safe keys; user/workspace/data-source ordering is preserved

risk: low

tychtjan added 2 commits April 9, 2026 11:17
VCR cassette normalization improvements to eliminate noisy diffs
between local Docker and staging recordings:

- Add _sort_known_arrays() that sorts only specific non-deterministic
  API response arrays (referenceProperties, edges,
  workspaceDataFilterColumns/References, userGroups) during YAML
  serialization. Unlike blanket sorting, this preserves server order
  for arrays that tests assert on.

- Fix XML group sorting in bytes branch: _sort_xml_groups() was only
  called for str bodies, skipping bytes bodies from compressed
  responses. Now applied in both branches of custom_before_request()
  and custom_before_response().

- Add _stabilize_hash_placeholders() to rename session-global numeric
  hash indices (EXECUTION_NORMALIZED_3) to format-based names
  (EXECUTION_NORMALIZED_CSV, EXPORT_NORMALIZED_XLSX) for export
  cassettes, with per-cassette re-indexing for non-export hashes.

- Add firstname/lastname/email to demo_user.json fixture so staging
  user profile matches local Docker bootstrap.

risk: low
Re-recorded all VCR cassettes against local Docker to pick up the
new normalization: targeted JSON array sorting for referenceProperties,
edges, workspaceDataFilterColumns/References, and userGroups.

risk: low
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.18%. Comparing base (19e1cc1) to head (fcd47aa).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1521   +/-   ##
=======================================
  Coverage   78.18%   78.18%           
=======================================
  Files         228      228           
  Lines       14930    14930           
=======================================
  Hits        11673    11673           
  Misses       3257     3257           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jaceksan jaceksan merged commit 5ca5548 into master Apr 9, 2026
13 checks passed
@jaceksan jaceksan deleted the fix/cassette-normalization-and-sorting branch April 9, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants