Rerunning query results in infinite load

[pooriaPoorsarvi]

Is this a new bug in Dekart?

• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When you rerun dekart with a new query on the same query tab, it can result in infinite load time for data. On reload of the page, new data is shown, however.

Expected Behavior

The new data should be shown after a finite period of time on the screen (Downloading Map Data step should be finished on the front end. This is likely because the front end is not getting the status update from the backend in a correct manner).

Steps To Reproduce

Go to dekart instance (i.e. playground)

run :

SELECT
    primary_type,
    district,
    latitude,
    longitude,
    date
from `bigquery-public-data.chicago_crime.crime`
LIMIT 1500

Then run :

SELECT
    primary_type,
    district,
    latitude,
    longitude,
    date
from `bigquery-public-data.chicago_crime.crime`
LIMIT 1550

On the same query tab. It should get stuck on Downloading Map Data step.

Even on the playground, this issue can be reproduced.

However, if you run a query that has been saved before, this does not seem to happen.

Relevant log output

This is the error from the playground

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading '1')
    at e.value (row-data-container.js:71:23)
    at t.maybeToDate (data-utils.js:137:13)
    at i (kepler-table.js:321:37)
    at Array.map (<anonymous>)
    at t.getQuantileDomain (data-scale-utils.js:29:61)
    at e.value (kepler-table.js:332:16)
    at n.value (base-layer.js:1037:20)
    at base-layer.js:928:36
    at Array.forEach (<anonymous>)
    at n.value (base-layer.js:921:40)


A better output can be found by running it locally:

Error: Failed to add data to map: Cannot read properties of undefined (reading '1')
    at dataset.js:88:9
I @ message.js:26
(anonymous) @ dataset.js:87
await in (anonymous) (async)
(anonymous) @ index.js:16
(anonymous) @ redux.js:51
dispatch @ redux.js:691
(anonymous) @ report.js:125
(anonymous) @ report.js:119
(anonymous) @ index.js:16
(anonymous) @ redux.js:51
dispatch @ redux.js:691
(anonymous) @ report.js:34
onMessage @ grpc.js:64
(anonymous) @ grpc-web-client.umd.js:1
e.rawOnMessage @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1
e.onTransportChunk @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1
Promise.then (async)
e.pump @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1
Promise.then (async)
e.send @ grpc-web-client.umd.js:1
e.sendMessage @ grpc-web-client.umd.js:1
e.send @ grpc-web-client.umd.js:1
t.invoke @ grpc-web-client.umd.js:1
F @ grpc.js:56
a @ grpc.js:72
(anonymous) @ grpc-web-client.umd.js:1
e.rawOnEnd @ grpc-web-client.umd.js:1
e.onTransportEnd @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1
Promise.then (async)
e.pump @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1
Promise.then (async)
e.pump @ grpc-web-client.umd.js:1
(anonymous) @ grpc-web-client.umd.js:1

Environment

- Docker
- Playground

Which database adapter are you using with dekart?

BigQuery

Additional Context

No response

[delfrrr]

Checking 👀

[delfrrr]

Hey @pooriaPoorsarvi

It appears that addDataToMap in Kepler library sometimes fails with such error

Issue is partially addressed by v0.12.1-rc.0 (v0.12.1 will come soon): now correct error message will be displayed with reload button (which commonly helps). Next step will investigate and fix it in kepler fork.

[delfrrr]

v0.12.1-rc.0 is already deployed to playground.

[pooriaPoorsarvi]

@delfrrr Thanks! is the dependency update also deployed on the master branch (or for the Kepler.gl fork made by the dekart team)?

[delfrrr]

In v0.12.1 there is only error handling, not fix for kepler.gl

dekart uses already fork of Kepler with few bugfixes https://github.com/dekart-xyz/kepler.gl/pkgs/npm/kepler.gl
when fix ready it will be integrated to dekart main branch.

[pooriaPoorsarvi]

@delfrrr I see; thanks for the update! We do have a possible workaround that we are using in our fork of dekart.

From what I observed, this error only happens on updating the data (so using addDataToMap for an existing dataset id), so the workaround, is to remove the dataset before adding in the updated version.

This could cause a bug which is, the visualization being reverted back to default on rerunning a query, but a simple fix for that is to save the visState of Kepler.gl before running the query, and then retrieving the layer, and the filters related to that dataset id, after we have removed it and then adding it back before adding back the new data.

[delfrrr]

@pooriaPoorsarvi can you please share code of your workaround?

[pooriaPoorsarvi]

@delfrrr Sure, it's a work in progress that I'm gonna change soon (for example instead of storing things in local storage you could use redux, and there are some other information that needs to be updated) but here is the gist of it:

before the query starts running we save the current state of kepler.gl : https://github.com/pooriaPoorsarvi/dekart/blob/c95c9a7539acf228f13ca02708f65cd531d7d761/src/client/Query.js#L206

We can't rely on the existing system because that is made for when the user saves the report, and that saves it to the backend. But we need to have a consistent view even if the user reruns a query after a minor change (without saving it).

Then once a query has ran, if a dataset layer already exists for it within Kepler, we remove it, and readd the new data, but we use the information we saved from the previous step to make it look like it has the same visual state: https://github.com/pooriaPoorsarvi/dekart/blob/custom-authorization/src/client/actions/dataset.js#L93:~:text=id)%3B-,dispatch(keplerRemove(dataset.id)),%7D,-await%20dispatch(

[delfrrr]

@pooriaPoorsarvi so I tried to fix kepler.gl, but I after fix in one place I receive error in the other place; It does not seam update flow works well in kepler.gl 😐

I use my variation of your fix logic: save config, remove dataset and add it again
https://github.com/dekart-xyz/dekart/pull/98/files

[delfrrr]

@pooriaPoorsarvi fix is available in 0.12.2-rc.0

[pooriaPoorsarvi]

@delfrrr Thank you! will merge it soon with our own code!