Skip to content

Reduce CI flakiness in three iceberg test modules#396

Merged
sfc-gh-mslot merged 1 commit into
mainfrom
marcoslot/fix-flaky-tests
Jul 3, 2026
Merged

Reduce CI flakiness in three iceberg test modules#396
sfc-gh-mslot merged 1 commit into
mainfrom
marcoslot/fix-flaky-tests

Conversation

@sfc-gh-mslot

@sfc-gh-mslot sfc-gh-mslot commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Problem

CI on `main` has been flapping over the past few weeks. Three failure
modes account for most of the noise:

  1. `test_data_file_pruning.py::test_simple_data_pruning_for_data_types` —
    fails with `AssertionError: failed to refresh object catalog table`
    when the object-store catalog refresh hasn't observed the pushed
    metadata yet. The structurally identical test in
    `test_partition_pruning.py` already has `@pytest.mark.flaky(reruns=2)`
    (added in bbd1176, May 8); this sibling was missed.
  2. `test_polaris_catalog_writable.py::test_sequences_serial_and_generated_columns[rest]` —
    fails with `AssertionError: Data file column stats mismatch` when
    the pg_catalog data-file stats and the REST metadata diverge by a
    row, which I've seen on multiple unrelated PRs and on `main`
    (26891987114, 26848484450, 26773102088, 26753357058, 27272760340).
  3. `test_object_store_catalog.py::{test_schema_mismatch, test_partitioned_read_only, test_unsupported_modifications_for_read_only, test_multiple_readers_writers}` —
    fails with `psycopg2.errors.DuplicateSchema: schema "object_store_sc1" already exists`.
    This isn't really a flaky test — it's a cascade. When an earlier
    test in the module fails midway through (most recently due to a
    missing `pg_lake_ducklake` extension), it skips its
    `DROP SCHEMA ... CASCADE` and the next several tests fail on the
    leftover schema instead of running.

Solution

Three small, narrow changes:

  • `test_simple_data_pruning_for_data_types`: add
    `@pytest.mark.flaky(reruns=2)` to match its sibling in
    `test_partition_pruning.py`.
  • `test_sequences_serial_and_generated_columns`: add
    `@pytest.mark.flaky(reruns=2)`.
  • `test_object_store_catalog.py`: add a module-scoped autouse fixture
    that runs `DROP SCHEMA IF EXISTS object_store_sc1, object_store_sc2 CASCADE`
    before each test. A single failure in this module no longer poisons
    the rest of it.

The first two are defensive masking and don't fix the underlying
race / consistency issue, but they match the prevailing convention in
the repo (`pytest-rerunfailures` was added in bbd1176 specifically for
this kind of test-harness-side flapping). The third is a real fix for
the cascade.

Test plan

  • CI passes
  • None of the three target tests appear in subsequent failure logs

@sfc-gh-mslot sfc-gh-mslot force-pushed the marcoslot/fix-flaky-tests branch from f4ecd62 to fc40407 Compare June 12, 2026 09:48

@sfc-gh-okalaci sfc-gh-okalaci left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_data_file_pruning.py::test_simple_data_pruning_for_data_types
fails with AssertionError: failed to refresh object catalog table
when the object-store catalog refresh hasn't observed the pushed
metadata yet. The structurally identical test in
test_partition_pruning.py already has @pytest.mark.flaky(reruns=2)
(added in bbd1176, May 8); this sibling was missed.

This was not missed, I think I objected to that, because I couldn't realize that it was due to object_store catalog.

It felt like pruning should be very deterministic, but then I think I'm fine with this explanation now.

- test_data_file_pruning.py::test_simple_data_pruning_for_data_types now
  has @pytest.mark.flaky(reruns=2). The same test logic in
  test_partition_pruning.py was already marked flaky; this version was
  missed and has been failing on main with "failed to refresh object
  catalog table".
- test_polaris_catalog_writable.py::test_sequences_serial_and_generated_columns
  now has @pytest.mark.flaky(reruns=2). It has been intermittently
  failing the data-file-stats cross-check between pg_catalog and the
  REST metadata across multiple unrelated PRs.
- test_object_store_catalog.py drops object_store_sc1 / object_store_sc2
  before each test via an autouse fixture. When one test fails midway,
  the next test was failing immediately with "schema already exists",
  cascading the noise. Defensive cleanup keeps a single failure from
  poisoning the rest of the module.

Signed-off-by: Marco Slot <marco.slot@snowflake.com>
@sfc-gh-mslot sfc-gh-mslot force-pushed the marcoslot/fix-flaky-tests branch from fc40407 to 666305d Compare June 23, 2026 13:03
@sfc-gh-mslot sfc-gh-mslot merged commit 01c529d into main Jul 3, 2026
64 checks passed
@sfc-gh-mslot sfc-gh-mslot deleted the marcoslot/fix-flaky-tests branch July 3, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants