Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Feature/list tables page size #174

Merged

Conversation

OmriBromberg
Copy link
Contributor

@OmriBromberg OmriBromberg commented May 12, 2021

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • [V] Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • [V] Ensure the tests and linter pass
  • [V] Code coverage does not decrease (if any source code was changed)
  • [V] Appropriate docs were updated (if necessary)

Fixes #173 馃

Hey guys, I explained in the issue a bit but I wanted to implement it so users would be able to change max_results @ dataset.list_tables like the arraysize parameter so I copied how it is treated in the parse_url area and updated the tests and docs for it, the new parameter for it is called list_tables_page_size

also I saw that on line 606
there is this expression self.arraysize = self.arraysize or arraysize which always takes the default arraysize instead of the parsed querystring arraysize because self.arraysize is always initialized to a default value, so I also reversed it to be as so self.arraysize = arraysize or self.arraysize, now the querystring arraysize will take precedence over the default value

@OmriBromberg OmriBromberg requested a review from a team as a code owner May 12, 2021 21:12
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label May 12, 2021
@google-cla
Copy link

google-cla bot commented May 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

馃摑 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

鈩癸笍 Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no This human has *not* signed the Contributor License Agreement. label May 12, 2021
@OmriBromberg
Copy link
Contributor Author

@googlebot I signed it!

@google-cla google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels May 12, 2021
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 5a97fa7 to 085df0f Compare May 12, 2021 21:28
鈥eration as list_tables_page_size

fix: arraysize default always takes priority over querystring
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 085df0f to 5331bcd Compare May 12, 2021 21:31
@jimfulton jimfulton changed the title Feature/list tables page size feat: Feature/list tables page size May 14, 2021
@jimfulton jimfulton added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels May 16, 2021
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels May 16, 2021
@jimfulton jimfulton self-requested a review May 16, 2021 20:32
Copy link
Contributor

@jimfulton jimfulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start. In addition to adding a feature, you found two bugs!

This needs test of the logic that fetches lists of tables. A unit test would be fine and should be straightforward to write.

pybigquery/sqlalchemy_bigquery.py Show resolved Hide resolved
pybigquery/sqlalchemy_bigquery.py Show resolved Hide resolved
@OmriBromberg
Copy link
Contributor Author

OmriBromberg commented May 19, 2021

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

@jimfulton
Copy link
Contributor

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

That's a good question.

If you look at: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/testing/suite/test_reflection.py#L537-L579

SQLAlchemy expects the results to be unqualified, which we satisfy if we have a schema.

Looking at other dialects:

MS SQL requires a schema: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/mssql/base.py#L2877

Oracle assumes there's a default: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/oracle/base.py#L1759-L1760

as does Postgres: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/postgresql/base.py#L3488-L3490

As does sqlite, sort of: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/sqlite/base.py#L1977-L1981

and so on.

So BQ is a little different because there might not be a default.

Amongst the options:

  1. Error if no schema is provided and there's no default.
  2. Qualify the results if we don't have a schema. I prefer using just the data set id in this case. full_table_id adds clutter IMO.

Playing with this some more, you can list schemas in other projects by specifying project_id.schema_id (e.g. "bigquery-public-data.census_bureau_acs"). That makes the use of a colon in full_table_id a little weird.

@jimfulton jimfulton marked this pull request as draft May 24, 2021 20:21
@jimfulton
Copy link
Contributor

The bigquery client now accepts a page_size parameter, so this PR wants to use that.

When your;re ready for me to review, please merge master to get rid of many spurious changes. :)

@OmriBromberg OmriBromberg marked this pull request as ready for review July 4, 2021 06:29
@jimfulton jimfulton added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Jul 6, 2021
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 6, 2021
@jimfulton
Copy link
Contributor

Thanks!

@jimfulton jimfulton merged commit e0f1496 into googleapis:master Jul 6, 2021
jimfulton pushed a commit to jimfulton/python-bigquery-sqlalchemy that referenced this pull request Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dataset.list_tables max_results parameter
3 participants