Allow Schema Caching in Superset SQL Lab Sidebar

[abhi-surapaneni]

Motivation

We are a large scale Superset user leveraging SQL Lab to query Trino cluster/AWS Athena over 15,000 times a day. The biggest pain-point is loading of the schema in the sidebar. It takes up to 300 seconds for just the schema to populate especially when the cluster is under load.

Proposed Change

The proposal is to cache the schema using cache-manager to improve user inter-activeness.

We have implemented this change on our instance of Superset and it has reduced the load time from ~300 seconds to ~150 milliseconds after the request is cached. We would like to contribute this back to the community.

Similar issues

The search for similar issues did not result in anything like this. Our issue is most likely due to Trino and Athena query engines, where fetching schema is not a trivial operation.