Skip to content

test(bigframes): Disable gcsfs cache for tests#16995

Merged
TrevorBergeron merged 4 commits into
mainfrom
tbergeron_gcsfs_fix
May 7, 2026
Merged

test(bigframes): Disable gcsfs cache for tests#16995
TrevorBergeron merged 4 commits into
mainfrom
tbergeron_gcsfs_fix

Conversation

@TrevorBergeron
Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a session-scoped pytest fixture to disable the gcsfs listings cache, preventing failures caused by stale file metadata during system tests. The review feedback suggests that simply instantiating GCSFileSystem is insufficient for global effect; instead, the configuration should be set via fsspec.config.conf and the instance cache should be cleared, requiring an additional import of fsspec.

# gcsfs by default uses a cache that can be stale, causing file loads to
# fail if the file was uploaded indirectly (eg via bq export job) during the
# course of the tests. disable the cache to avoid this.
gcsfs.GCSFileSystem(use_listings_cache=False)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Instantiating gcsfs.GCSFileSystem(use_listings_cache=False) only creates a single instance with the cache disabled and does not affect other instances created elsewhere in the codebase (e.g., by bigframes internals or pandas). To globally disable the listings cache for all gcsfs instances during the test session, you should set the configuration in fsspec.config.conf and clear the instance cache to ensure any existing instances are recreated with the new settings.

Suggested change
gcsfs.GCSFileSystem(use_listings_cache=False)
fsspec.config.conf["gcs"] = {"use_listings_cache": False}
gcsfs.GCSFileSystem.clear_instance_cache()

from datetime import datetime
from typing import Dict, Generator, Optional

import gcsfs
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add fsspec to the imports. This is required to globally configure gcsfs settings (such as disabling the listings cache) for the entire test session via the fsspec configuration system.

Suggested change
import gcsfs
import fsspec
import gcsfs

@TrevorBergeron TrevorBergeron marked this pull request as ready for review May 7, 2026 20:48
@TrevorBergeron TrevorBergeron requested review from a team as code owners May 7, 2026 20:48
@TrevorBergeron TrevorBergeron requested review from shuoweil and removed request for a team May 7, 2026 20:48
@TrevorBergeron TrevorBergeron merged commit 40f459e into main May 7, 2026
31 checks passed
@TrevorBergeron TrevorBergeron deleted the tbergeron_gcsfs_fix branch May 7, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants