Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove "limit 10" from the default SQL queries #151

Closed
xandrew-lynx opened this issue Mar 5, 2021 · 4 comments
Closed

Remove "limit 10" from the default SQL queries #151

xandrew-lynx opened this issue Mar 5, 2021 · 4 comments

Comments

@xandrew-lynx
Copy link
Contributor

The result is limited on display anyways. I delete it every single time and it's annoying. Any drawback of removing these?

@xandrew-lynx
Copy link
Contributor Author

(Except when I forget to remove it and scratch my head on why something is weird downstream... :/ )

@darabos
Copy link
Contributor

darabos commented Mar 5, 2021

Ah, yes please! The counter argument was (as I recall) that "persist: yes" is also a default. So even though in getTableOutput we only fetch the first 10 lines, the backend may still end up computing and storing a lot of data unnecessarily.

How about we set "persist: no" as the default? I think the counter argument for that was that Spark at the time didn't handle long chains well. But I'm just upgrading to Spark 3. Hopefully it's a solved problem by now. A longer chain even allows for better optimization. (E.g. maybe you don't need to compute some intermediate data if you don't need it in the end.) And also we are not building long SQL chains all the time. You can turn on "persist" when needed.

@xandrew-lynx
Copy link
Contributor Author

Sounds good to me. Especially as persisting also messes up ordering when the user adds "order by". :/

@darabos darabos added this to the LynxKite 4.3.0 milestone Mar 24, 2021
@darabos
Copy link
Contributor

darabos commented Oct 11, 2022

Done in #220. Forgot to close it.

@darabos darabos closed this as completed Oct 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants