Workflow to Support Lazy Loading of Dataset?

[gursharan-info]

I'm considering using the @kanaries/graphic-walker library for generic data visualisation, and I'm finding it extremely useful. However, I have a requirement where I need to handle large datasets efficiently. Currently, the library fetches the entire dataset upfront, which can be problematic when dealing with large amounts of data. I'm assuming that based on demos; please let me know if I'm wrong. We have a requirement to display spatial datasets which can go up to 50GB in size.

I wanted to inquire if it's possible to modify the workflow of the library to support the lazy loading of the dataset as a React Component. Here's the proposed workflow:

• Initially, only fetch the measures and dimensions from the data source. JSON API is preferable, but direct SQL connection is also doable.
• Allow the user to make chart selections such as the x-axis, y-axis, size, and color.
• A button (e.g., "Generate Visualization") should be used to trigger the lazy loading of the dataset. It can either filter via the JSON API,or fetch the actual group/subset via SQL query, based on the selected measures and dimensions.
• Once the dataset is fetched, generate the visualization using the selected chart options.

Please let me know if this is something that can be done using this library as an import without needing to customize the internals.

[ObservedObserver]

We have a branch working on a related feature: Add a new option that the dataset will not be loaded into front-end. In this case, the query task will be sent to a customized server to handle the computation and only return the view data (which is usually much smaller than the raw data).

It can allow graphic-walker to handle much larger dataset than before. The feature is already in testing and will be published this week.

[gursharan-info]

Thanks for the reply. I will be waiting for the feature to try.

[maxott]

Any update on that?