Skip to content

feat(tests): overhaul staging test infrastructure and update fixtures#1392

Merged
jaceksan merged 4 commits intomasterfrom
tests_rewrite_staging
Mar 17, 2026
Merged

feat(tests): overhaul staging test infrastructure and update fixtures#1392
jaceksan merged 4 commits intomasterfrom
tests_rewrite_staging

Conversation

@tychtjan
Copy link
Copy Markdown
Contributor

@tychtjan tychtjan commented Mar 5, 2026

Summary

Added staging test support (PYTHON_DEX), which required rewrites in the test
infrastructure (dynamic PostgreSQL URL replacement instead of duplicating fixtures)
and a new CI workflow for pre-merge staging tests.

Changes by commit

1. chore: regenerate API client from updated OpenAPI spec

  • Updated OpenAPI-generated API client with new models and endpoints
  • Reviewer: auto-generated, safe to skip

2. feat(ci): add staging test infrastructure and harden test setup

  • Add staging-tests.yaml workflow (triggered by label, /test-staging comment, or manual dispatch)
  • Overhaul conftest.py for dynamic data source URL replacement (staging vs local)
  • Add clean_staging.py and upload_demo_layout.py helper scripts
  • Unify token passing: all staging make targets use TOKEN= argument
  • Fix LLM provider patch to use InAttributes after spec removed PatchAttributes
  • Add alternativeDataSourceId to data source fixture
  • Document staging test workflow in CONTRIBUTING.md

3. chore: regenerate test cassettes

  • Regenerated all cassettes against staging environment
  • Reviewer: regenerated fixtures, safe to skim

Test plan

  • Passes local tests
  • Passes staging environment tests

JIRA: DX-431
risk: low

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 5, 2026

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 77.24%. Comparing base (e650ab3) to head (5bc7b49).

Files with missing lines Patch % Lines
.../catalog/organization/entity_model/llm_provider.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1392      +/-   ##
==========================================
- Coverage   77.36%   77.24%   -0.12%     
==========================================
  Files         228      227       -1     
  Lines       14775    14672     -103     
==========================================
- Hits        11430    11334      -96     
+ Misses       3345     3338       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tychtjan tychtjan force-pushed the tests_rewrite_staging branch from 44844f3 to 54e15b1 Compare March 12, 2026 07:59
@tychtjan
Copy link
Copy Markdown
Contributor Author

tychtjan commented Mar 12, 2026

I analyzed why there were some of the bigger deletes in the fixtures

 # OLD: cleanup + verification assertion
  finally:
      sdk.catalog_organization.delete_jwk("demoJwk")
      assert len(sdk.catalog_organization.list_jwks()) == 0  # <-- this GET is now gone

The new pattern uses safe_delete() which only calls the delete, no verification list:

  # NEW: cleanup only, no verification
  finally:
      safe_delete(sdk.catalog_organization.delete_jwk, "demoJwk")

This removes the final GET ...?page=0&size=500 call that verified the entity was gone.
That's why every cassette has a trailing interaction block removed — it was the VCR recording of
that verification GET. This is entirely expected and correct. The verification was fragile on staging (where other entities may exist, making len(...) == 0 fail), and the safe_delete + snapshot/restore approach is more robust.

@tychtjan tychtjan force-pushed the tests_rewrite_staging branch 2 times, most recently from b8b6a13 to d2e0811 Compare March 12, 2026 09:19
@tychtjan tychtjan force-pushed the tests_rewrite_staging branch 4 times, most recently from c88213d to add5390 Compare March 16, 2026 08:44
@tychtjan tychtjan force-pushed the tests_rewrite_staging branch 2 times, most recently from cf35e1a to be7b556 Compare March 17, 2026 07:37
Update all auto-generated API client code (docs, api, models) and
schema JSON files to match the latest OpenAPI specification.
- Add staging test workflow and env-based config overrides (GD_TEST_ENV)
- Remove deprecated LLM endpoint entity model, service methods, and tests
- Tighten test assertions to use exact counts and equality checks
- Replace fragile fixture-file patching with in-memory inject_ds_connection()
- Add clean_staging.py and org-id layout symlinks
- Add GRPC_FORGE_HOST/PORT env vars to docker-compose api-gw service
Re-record all VCR cassettes against local environment to reflect
changes from the latest API client regeneration.
@tychtjan tychtjan force-pushed the tests_rewrite_staging branch from 6621b72 to 7b95d35 Compare March 17, 2026 08:31
Move BigQuery/Databricks dummy credentials to shared config section so
they are defined once and reused by all environments. The staging env
previously used structurally invalid placeholders (e.g. "dummy-bigquery-token")
that failed backend schema validation, forcing a pytest.skip. Now both
local and staging use the same Base64-encoded service account JSON that
passes BigQueryDataSourceValidator.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jaceksan jaceksan enabled auto-merge March 17, 2026 10:24
@jaceksan jaceksan merged commit 4dc3f87 into master Mar 17, 2026
10 of 12 checks passed
@jaceksan jaceksan deleted the tests_rewrite_staging branch March 17, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants