Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using saved questions as the source of models we should remove the row limit #24793

Closed
luizarakaki opened this issue Aug 15, 2022 · 3 comments · Fixed by #24970
Closed

Comments

@luizarakaki
Copy link
Contributor

luizarakaki commented Aug 15, 2022

Is your feature request related to a problem? Please describe.
The query processor has a global row limit of 1048575.

If we use a saved question as the source of the model, when the saved question query runs, we add the global limit.

There are many use cases when users want to cache models with more than 1M rows.

Describe the solution you'd like
Two options;

  1. Remove the limit for model cache
  2. Change it to 1B (minimum), ideally 10B.

How important is this feature to you?
This is important for model cache, as the feature was designed for large models.

@luizarakaki luizarakaki closed this as not planned Won't fix, can't repro, duplicate, stale Aug 15, 2022
@luizarakaki luizarakaki reopened this Aug 15, 2022
@luizarakaki luizarakaki changed the title Remove row limit or define a reasonable number for model cache When use saved questions as the source of models we should remove the row limit Aug 15, 2022
@dpsutton
Copy link
Contributor

select count(*) from dbt_models.page_view -> 7,868,389
select count(*) from metabase_cache_134ba_48.model_3367_page_views -> 1,048,575

@luizarakaki luizarakaki changed the title When use saved questions as the source of models we should remove the row limit When using saved questions as the source of models we should remove the row limit Aug 16, 2022
@luizarakaki
Copy link
Contributor Author

Extra context
Slack thread 1
Slack thread 2

@calherries calherries self-assigned this Aug 23, 2022
@calherries
Copy link
Contributor

calherries commented Aug 23, 2022

Probably related bug: the row limit applies when a saved question is used as a source in native SQL.

Reproduced in stats: select count(*) from {{#3367}} -> 1,048,575

But the bug only happens with native SQL. Using the query builder to create a count on the Page View model yields 7,868,389.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants