docs: remove import bigframes.pandas as bpd boilerplate from many samples #2147

tswast · 2025-10-07T21:56:38Z

Also, fixes several constructors that didn't take a session for compatibility with multi-session applications.

🦕

…ples Also, fixes several constructors that didn't take a session for compatibility with multi-session applications.

bigframes/bigquery/_operations/ai.py

bigframes/operations/ai.py

bigframes/operations/semantics.py

TrevorBergeron · 2025-10-07T23:54:34Z

bigframes/conftest.py

+    doctest_namespace["np"] = np
+    doctest_namespace["pd"] = pd
+    doctest_namespace["pa"] = pa
+    doctest_namespace["bpd"] = polars_session


I wonder if instead, we should just inject the polars session as global session? Not sure all the methods are the same, but I guess it works so far?

Unfortunately, there's quite a bit that isn't supported yet on the Polars session. Doing it this way means that we can override bpd to be the BQ version in the samples itself with a simple import.

TrevorBergeron · 2025-10-08T00:00:48Z

bigframes/session/__init__.py

+    # These are included so that Session and bigframes.pandas can be used
+    # interchangeably.


If this is purely for doctests, or can we just inject session for doctests? or are we trying to enable some other stuff

IMO, it's important for consistency. I actually uncovered a few cases where the session should have been supplied to things like to_datetime() with local data but wasn't.

TrevorBergeron · 2025-10-08T00:02:11Z

bigframes/pandas/__init__.py

+) -> Union[pandas.Timestamp, datetime.datetime, bigframes.series.Series]:
+    return global_session.with_default_session(
+        bigframes.session.Session.to_datetime,


Not sure I understand this change/

to_datetime() has code paths that take local data. It was using the global session implicitly when it constructed the Series objects. Now it can take a session explicitly.

TrevorBergeron · 2025-10-08T00:02:56Z

third_party/bigframes_vendored/pandas/core/generic.py

        **Examples:**

            >>> import bigframes.pandas as bpd
-            >>> bpd.options.display.progress_bar = None


why do we still need some of the bpd imports?

This is how I made some samples use the BQ session instead of the Polars session. In this case, hash() is unimplemented:

third_party/bigframes_vendored/pandas/core/generic.py ..........F. [100%] ================================================================ FAILURES ================================================================ ______________________________ [doctest] third_party.bigframes_vendored.pandas.core.generic.NDFrame.sample _______________________________ 558 dog 4 0 2 559 spider 8 0 1 560 fish 0 0 8 561 <BLANKLINE> 562 [4 rows x 3 columns] 563 564 Fetch one random row from the DataFrame (Note that we use `random_state` 565 to ensure reproducibility of the examples): 566 567 >>> df.sample(random_state=1) UNEXPECTED EXCEPTION: NotImplementedError("Polars compiler hasn't implemented hash()") Traceback (most recent call last): File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/doctest.py", line 1350, in __run exec(compile(example.source, filename, "single", File "<doctest third_party.bigframes_vendored.pandas.core.generic.NDFrame.sample[2]>", line 1, in <module> File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/log_adapter.py", line 197, in wrapper raise e File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/log_adapter.py", line 182, in wrapper return method(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/dataframe.py", line 794, in __repr__ pandas_df, row_count, query_job = self._block.retrieve_repr_request_results( File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/blocks.py", line 1658, in retrieve_repr_request_results head_result = self.session._executor.execute( File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/testing/polars_session.py", line 48, in execute lazy_frame: polars.LazyFrame = self.compiler.compile(array_value.node) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 577, in compile return self.compile_node(node) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 601, in compile_filter return self.compile_node(node.child).filter( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 649, in compile_offsets return self.compile_node(node.child).with_columns( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 601, in compile_filter return self.compile_node(node.child).filter( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 649, in compile_offsets return self.compile_node(node.child).with_columns( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 607, in compile_orderby frame = self.compile_node(node.child) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 639, in compile_projection new_col = self.expr_compiler.compile_expression(bound_expr).alias(name.sql) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 180, in _ return self.compile_op(op, *args) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 184, in compile_op raise NotImplementedError(f"Polars compiler hasn't implemented {op}") NotImplementedError: Polars compiler hasn't implemented hash() /usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/third_party/bigframes_vendored/pandas/core/generic.py:567: UnexpectedException

Aside: As much as possible I'd like to encourage us BigFrames devs to implement our ops in the Polars session as well as BQ, so defaulting to Polars is a subtle nudge in that direction.

tswast · 2025-10-08T14:28:26Z

bigframes/pandas/__init__.py

+) -> Union[pandas.Timestamp, datetime.datetime, bigframes.series.Series]:
+    return global_session.with_default_session(
+        bigframes.session.Session.to_datetime,


to_datetime() has code paths that take local data. It was using the global session implicitly when it constructed the Series objects. Now it can take a session explicitly.

tswast · 2025-10-08T14:29:31Z

bigframes/session/__init__.py

+    MultiIndex.from_tuples = bigframes.core.indexes.MultiIndex.from_tuples  # type: ignore
+    MultiIndex.from_frame = bigframes.core.indexes.MultiIndex.from_frame  # type: ignore
+    MultiIndex.from_arrays = bigframes.core.indexes.MultiIndex.from_arrays  # type: ignore


TODO: these should probably take a Session argument, too.

tswast · 2025-10-08T14:30:47Z

bigframes/conftest.py

+    doctest_namespace["np"] = np
+    doctest_namespace["pd"] = pd
+    doctest_namespace["pa"] = pa
+    doctest_namespace["bpd"] = polars_session


Unfortunately, there's quite a bit that isn't supported yet on the Polars session. Doing it this way means that we can override bpd to be the BQ version in the samples itself with a simple import.

tswast · 2025-10-08T14:43:28Z

third_party/bigframes_vendored/pandas/core/generic.py

        **Examples:**

            >>> import bigframes.pandas as bpd
-            >>> bpd.options.display.progress_bar = None


This is how I made some samples use the BQ session instead of the Polars session. In this case, hash() is unimplemented:

third_party/bigframes_vendored/pandas/core/generic.py ..........F. [100%] ================================================================ FAILURES ================================================================ ______________________________ [doctest] third_party.bigframes_vendored.pandas.core.generic.NDFrame.sample _______________________________ 558 dog 4 0 2 559 spider 8 0 1 560 fish 0 0 8 561 <BLANKLINE> 562 [4 rows x 3 columns] 563 564 Fetch one random row from the DataFrame (Note that we use `random_state` 565 to ensure reproducibility of the examples): 566 567 >>> df.sample(random_state=1) UNEXPECTED EXCEPTION: NotImplementedError("Polars compiler hasn't implemented hash()") Traceback (most recent call last): File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/doctest.py", line 1350, in __run exec(compile(example.source, filename, "single", File "<doctest third_party.bigframes_vendored.pandas.core.generic.NDFrame.sample[2]>", line 1, in <module> File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/log_adapter.py", line 197, in wrapper raise e File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/log_adapter.py", line 182, in wrapper return method(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/dataframe.py", line 794, in __repr__ pandas_df, row_count, query_job = self._block.retrieve_repr_request_results( File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/blocks.py", line 1658, in retrieve_repr_request_results head_result = self.session._executor.execute( File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/testing/polars_session.py", line 48, in execute lazy_frame: polars.LazyFrame = self.compiler.compile(array_value.node) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 577, in compile return self.compile_node(node) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 601, in compile_filter return self.compile_node(node.child).filter( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 649, in compile_offsets return self.compile_node(node.child).with_columns( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 601, in compile_filter return self.compile_node(node.child).filter( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 649, in compile_offsets return self.compile_node(node.child).with_columns( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 607, in compile_orderby frame = self.compile_node(node.child) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 630, in compile_selection return self.compile_node(node.child).select( File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 639, in compile_projection new_col = self.expr_compiler.compile_expression(bound_expr).alias(name.sql) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 180, in _ return self.compile_op(op, *args) File "/usr/local/google/home/swast/.pyenv/versions/3.10.16/lib/python3.10/functools.py", line 926, in _method return method.__get__(obj, cls)(*args, **kwargs) File "/usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/bigframes/core/compile/polars/compiler.py", line 184, in compile_op raise NotImplementedError(f"Polars compiler hasn't implemented {op}") NotImplementedError: Polars compiler hasn't implemented hash() /usr/local/google/home/swast/src/github.com/googleapis/python-bigquery-dataframes/third_party/bigframes_vendored/pandas/core/generic.py:567: UnexpectedException

Aside: As much as possible I'd like to encourage us BigFrames devs to implement our ops in the Polars session as well as BQ, so defaulting to Polars is a subtle nudge in that direction.

…plate

… tswast-doctest-boilerplate

…plate

tswast · 2025-10-08T20:30:47Z

dummy.pkl

Remove this

This reverts commit 765b678.

…-names

…plate

docs: remove import bigframes.pandas as bpd boilerplate from many sam…

8cfaca6

…ples Also, fixes several constructors that didn't take a session for compatibility with multi-session applications.

tswast requested review from a team as code owners October 7, 2025 21:56

tswast requested a review from TrevorBergeron October 7, 2025 21:56

product-auto-label bot added the size: l Pull request size is large. label Oct 7, 2025

blunderbuss-gcf bot assigned sycai Oct 7, 2025

product-auto-label bot added api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. samples Issues that are directly related to samples. labels Oct 7, 2025

fix docs

c85d47f

tswast commented Oct 7, 2025

View reviewed changes

tswast added 4 commits October 7, 2025 23:45

fix unit tests

78bfccf

skip sklearn test

210dc9a

fix snapshot

bed4069

Merge branch 'main' into tswast-doctest-boilerplate

aa23fca

TrevorBergeron reviewed Oct 8, 2025

View reviewed changes

tswast commented Oct 8, 2025

View reviewed changes

tswast added 6 commits October 8, 2025 15:00

plumb through session for from_tuples and from_arrays

20cae2d

Merge remote-tracking branch 'origin/main' into tswast-doctest-boiler…

f57a8c3

…plate

Merge remote-tracking branch 'origin/tswast-doctest-boilerplate' into…

d8ab16b

… tswast-doctest-boilerplate

add from_frame

1dc648b

make sure polars session isnt skipped on Kokoro

9de6f9f

fix apply doctest

5d23dee

tswast mentioned this pull request Oct 8, 2025

test: install polars in doctest session #1775

Closed

4 tasks

tswast added 7 commits October 8, 2025 16:18

make doctest conftest available everywhere

20d7c27

Merge remote-tracking branch 'origin/main' into tswast-doctest-boiler…

534471e

…plate

add python version flexibility for to_dict

fbe606e

disambiguate explicit names

171f3ec

disambiguate explicit name none versus no name

ded5c1e

fix for column name comparison in pandas bin op

841bc64

avoid setting column labels in special case of Series(block)

81f49a6

tswast commented Oct 8, 2025

View reviewed changes

dummy.pkl Outdated

Copy link

Collaborator Author

tswast Oct 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this

tswast added 14 commits October 8, 2025 20:31

revert doctest changes

a97cc93

revert df docstrings

5aaacfe

add polars series unit tests

922bbf4

restore a test

765b678

Revert "restore a test"

4aa47a8

This reverts commit 765b678.

skip null

f75f5bf

skip unsupported tests

a7058ac

revert more docs changes

62d5911

revert more docs

70021f3

revert more docs

9350209

fix unit tests python 3.13

23346b0

add test to reproduce name error

03822d7

Merge remote-tracking branch 'origin/main' into tswast-polars-session…

dc02baf

…-names

add tests for session scoped methods

5df828b

tswast mentioned this pull request Oct 9, 2025

feat: create session-scoped cut, DataFrame, MultiIndex, Index, Series, to_datetime, and to_timedelta methods #2157

Merged

tswast added 5 commits October 9, 2025 15:52

revert new session methods

ddbb32d

fix TestSession read_pandas for Series

d80bfcb

revert more unnecessary changes

0a5a935

even more

1126244

Merge branch 'tswast-session-methods' into tswast-doctest-boilerplate

8b069e8

tswast force-pushed the tswast-doctest-boilerplate branch from c3a23ea to 8b069e8 Compare October 9, 2025 17:18

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Oct 9, 2025

tswast added 4 commits October 9, 2025 18:23

add unit_noextras to improve code coverage

d63a95f

Merge remote-tracking branch 'origin/main' into tswast-polars-session…

6a8f694

…-names

run system tests on latest fully supported

6aadbaf

Merge branch 'tswast-polars-session-names' into tswast-doctest-boiler…

beb3a7d

…plate

product-auto-label bot added size: s Pull request size is small. and removed size: m Pull request size is medium. labels Oct 9, 2025

		# These are included so that Session and bigframes.pandas can be used
		# interchangeably.

docs: remove import bigframes.pandas as bpd boilerplate from many samples #2147

Are you sure you want to change the base?

docs: remove import bigframes.pandas as bpd boilerplate from many samples #2147

Conversation

tswast commented Oct 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants