You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ah, yes please! The counter argument was (as I recall) that "persist: yes" is also a default. So even though in getTableOutput we only fetch the first 10 lines, the backend may still end up computing and storing a lot of data unnecessarily.
How about we set "persist: no" as the default? I think the counter argument for that was that Spark at the time didn't handle long chains well. But I'm just upgrading to Spark 3. Hopefully it's a solved problem by now. A longer chain even allows for better optimization. (E.g. maybe you don't need to compute some intermediate data if you don't need it in the end.) And also we are not building long SQL chains all the time. You can turn on "persist" when needed.
The result is limited on display anyways. I delete it every single time and it's annoying. Any drawback of removing these?
The text was updated successfully, but these errors were encountered: