Strategy to performant display more than 100K items.

[euzu]

I am using deck.gl 8.7.11, react-map-gl 6.1.19, react 18.2.0, maplibre-gl 2.1.9.

I would like to display more than 100K items in one layer. The layer updates itself several times a second. Therefore I have a requestAnimationFrame loop, in which I create and pass the data multiple times. I use data in binary format {src: binary_data, length: binaray_data,length}. The binary data is updated by a worker thread in the background.

The number of items (and therefore the length) can change several times within one second. Several attributes of the data can change. For this reason the layer must be redrawn every render cycle.

The problem is that the WebGL context is always rebuilt with this solution.

To prevent this I thought of the following.

• I always leave the length at maximum, even if there are less items.
• I always draw the items from 0 to currentLength (which is <= Length).

The problem:
Even with dataComparator = () => false, _dataDiff = () => [{startRow: 0, endRow: currentLength}],
or updateTriggers with random values and discard in shaders the layer will not be drawn correctly.

I don't know how to set single positions to invisible (also in shader with discard led to no result).

What is the strategy to display this size of data without creating new data object every render cycle to avoid recreation of WebGL context ?

[Pessimistress]

creating new data object every render cycle to avoid recreation of WebGL context

Creating new data objects does not "recreate WebGL context". WebGL context is only created once when you first mount the DeckGL component.

I think you are using advanced props without understanding what they mean. In the order of cheap to expensive operations, you have

• Redraw (zoom, pan, auto highlight etc.) - very cheap
• Attribute update (single updateTrigger change) - cheap if using constant, expensive if using accessor function
• Data update (data object shallowly changed, dataComparator returns false) - expensive, all attributes must be recalculated

By setting dataComparator = () => false you are forcing all updates to use the most expensive route.

[Pessimistress]

You should check out using binary attributes. On every update, you create a new data object with the length and attributes that match your next frame. dataComparator and _dataDiff will have no effect if all your attributes are prepacked.

[euzu]

that is exactly whart i am doing. Ok i will remove the dataComparator.
Thank you.

[Pessimistress]

Well you said that you were passing a {src, length}, not {attributes, length}. I can only speculate what you are doing wrong unless you share some of your code.

[euzu]

Sorry, I misunderstood you.

I pass to the layer each time a new {src : customAccessor, length: data-length} object.
the accessor does not change but the data-length can change a lot.

in the layer i define the following:

getPosition: (e: Entity, access: BinaryDataAccess) => access.data.src.getPosition(access.index);
getSize: (e: Entity, access: BinaryDataAccess) => access.data.src.getSize(access.index)
...

I am using react. The class which instantiates deck-gl has a function which updates the react state data every animation frame.

 const accessor = useMemo(() => new MapDataAccessor(), []);
 const [mapData, setMapData] = useState<any>({src: accessor, length:0})

  useEffect(() => {
        let handle = 0;
        const loop = () => {
            setMapData({src: accessor, length: accessor.length});
            handle = requestAnimationFrame(loop);
        }
        handle = requestAnimationFrame(loop)
        return () => cancelAnimationFrame(handle);
    }, [mapData]);


<DeckGL ref={deckGlRef}
                    id={'deckgl-overlay-' + mapId}
                    effects={effects}
                    ContextProvider={MapContext.Provider}
                    useDevicePixels={false}
                    getCursor={getCursor}
                    layers={layers}

That is my current solution.

i am not using updateTriggers, because i had a lot strange render problems.

[Pessimistress]

To get better performance, you would want to remove the accessor functions (getPosition, getSize etc.) and move those to the worker. Please read the example that I linked above.