Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] docs generate does not find all schemas when there are more than 10K schemas #1218

Open
mikealfare opened this issue May 1, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@mikealfare
Copy link
Contributor

mikealfare commented May 1, 2024

Current Behavior

When there are more than 10K schemas, docs generate does not find all of the schemas, likely because of not paginating the results. This causes objects in those schemas to be missing from the catalog. This affects all docs generate runs for 1.6 and prior, and only docs generate --no-compile for 1.7 and forward. See #1216 for more information.

Expected Behavior

All objects should be included in the catalog.

Steps To Reproduce

  1. Create more than 10K schemas
  2. Create another schema (named test_schema for the sake of reference here)
  3. Run docs generate --no-compile
  4. test_schema will be empty in the catalog

Environment

- OS: all
- Python: all
- dbt-core: all
- dbt-bigquery: all (but differently for 1.7 and forward)

Additional Context

We actually found this during testing because our test schemas were building up for a test class, though this would affect any user who also has 10K schemas.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants