Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use dataset_id & table_id as the key of set 'grouped_tables' in bigquery_metadata_extractor #421

Merged
merged 1 commit into from Dec 7, 2020

Conversation

xuan616
Copy link
Member

@xuan616 xuan616 commented Dec 7, 2020

Fix the issue that some table keys of sharded tables are missing from BQ metadata extractor. Currently, there is a set 'grouped_tables' storing table_prefix of sharded tables to avoid duplicate metadata records, but it is possible the table prefixes would be same in different datasets.

Summary of Changes

Related issue: 833

Sharded tables will use the concatenation of dataset_id & table_id as the key of set 'grouped_tables' in bigquery_metadata_extractor

Tests

No more tests were added and current test can still work

Documentation

More details are in issue 833

CheckList

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
  • PR passes make test

@feng-tao
Copy link
Member

feng-tao commented Dec 7, 2020

@crazy-2020 could you update the DCO?

@feng-tao feng-tao changed the title Use dataset_id & table_id as the key of set 'grouped_tables' in bigquery_metadata_extractor fix: Use dataset_id & table_id as the key of set 'grouped_tables' in bigquery_metadata_extractor Dec 7, 2020
Signed-off-by: xuans <xuan_shen@outlook.com>
@xuan616
Copy link
Member Author

xuan616 commented Dec 7, 2020

@crazy-2020 could you update the DCO?

Sure, updated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants