Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Feature/list tables page size #174

Merged

Conversation

@OmriBromberg
Copy link
Contributor

@OmriBromberg OmriBromberg commented May 12, 2021

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • [V] Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • [V] Ensure the tests and linter pass
  • [V] Code coverage does not decrease (if any source code was changed)
  • [V] Appropriate docs were updated (if necessary)

Fixes #173 馃

Hey guys, I explained in the issue a bit but I wanted to implement it so users would be able to change max_results @ dataset.list_tables like the arraysize parameter so I copied how it is treated in the parse_url area and updated the tests and docs for it, the new parameter for it is called list_tables_page_size

also I saw that on line 606
there is this expression self.arraysize = self.arraysize or arraysize which always takes the default arraysize instead of the parsed querystring arraysize because self.arraysize is always initialized to a default value, so I also reversed it to be as so self.arraysize = arraysize or self.arraysize, now the querystring arraysize will take precedence over the default value

@google-cla
Copy link

@google-cla google-cla bot commented May 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

馃摑 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

鈩癸笍 Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no label May 12, 2021
@OmriBromberg
Copy link
Contributor Author

@OmriBromberg OmriBromberg commented May 12, 2021

@googlebot I signed it!

@google-cla google-cla bot added cla: yes and removed cla: no labels May 12, 2021
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 5a97fa7 to 085df0f May 12, 2021
鈥eration as list_tables_page_size

fix: arraysize default always takes priority over querystring
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 085df0f to 5331bcd May 12, 2021
@jimfulton jimfulton changed the title Feature/list tables page size feat: Feature/list tables page size May 14, 2021
@jimfulton jimfulton self-requested a review May 16, 2021
Copy link
Contributor

@jimfulton jimfulton left a comment

This is a great start. In addition to adding a feature, you found two bugs!

This needs test of the logic that fetches lists of tables. A unit test would be fine and should be straightforward to write.

pybigquery/sqlalchemy_bigquery.py Show resolved Hide resolved
pybigquery/sqlalchemy_bigquery.py Show resolved Hide resolved
@OmriBromberg
Copy link
Contributor Author

@OmriBromberg OmriBromberg commented May 19, 2021

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

@jimfulton
Copy link
Contributor

@jimfulton jimfulton commented May 19, 2021

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

That's a good question.

If you look at: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/testing/suite/test_reflection.py#L537-L579

SQLAlchemy expects the results to be unqualified, which we satisfy if we have a schema.

Looking at other dialects:

MS SQL requires a schema: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/mssql/base.py#L2877

Oracle assumes there's a default: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/oracle/base.py#L1759-L1760

as does Postgres: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/postgresql/base.py#L3488-L3490

As does sqlite, sort of: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/sqlite/base.py#L1977-L1981

and so on.

So BQ is a little different because there might not be a default.

Amongst the options:

  1. Error if no schema is provided and there's no default.
  2. Qualify the results if we don't have a schema. I prefer using just the data set id in this case. full_table_id adds clutter IMO.

Playing with this some more, you can list schemas in other projects by specifying project_id.schema_id (e.g. "bigquery-public-data.census_bureau_acs"). That makes the use of a colon in full_table_id a little weird.

@jimfulton jimfulton marked this pull request as draft May 24, 2021
@jimfulton
Copy link
Contributor

@jimfulton jimfulton commented Jun 8, 2021

The bigquery client now accepts a page_size parameter, so this PR wants to use that.

When your;re ready for me to review, please merge master to get rid of many spurious changes. :)

@OmriBromberg OmriBromberg marked this pull request as ready for review Jul 4, 2021
@jimfulton
Copy link
Contributor

@jimfulton jimfulton commented Jul 6, 2021

Thanks!

@jimfulton jimfulton merged commit e0f1496 into googleapis:master Jul 6, 2021
5 checks passed
jimfulton added a commit to jimfulton/python-bigquery-sqlalchemy that referenced this issue Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants