Poor Map Performance/Usability for 100k+ data points

[neibla]

Summary

With v0.73.1 of Streamlit the pydeck integration has significant performance issues for large datasets compared to jupyter + pydeck integration.

Steps to reproduce

Compare the scroll performance/usability of the map component with jupyter
https://github.com/neibla/streamlit-map-performance-comparison/blob/master/comparison.ipynb
vs Streamlit
https://github.com/neibla/streamlit-map-performance-comparison/blob/master/app.py

Expected behavior:

I'd expect the performance to be similar for at least 100k points

Actual behavior:

Map component unusable with Streamlit for 100k points

Is this a regression?

That is, did this use to work the way you expected in the past?
No, encountered since at least v0.5

Debug info

• Streamlit version: 0.73
• Python version: 3.8
• OS version: MacOS 11.1
• Browser version: Chrome 87.0.4280.88

Additional information

Created a repo comparing the base cases.
https://github.com/neibla/streamlit-map-performance-comparison

[nthmost]

Hi @neibla -- Thanks for providing the side-by-side comparison! We are (unfortunately) aware of the performance issue around large datasets and that's why we're forging ahead with a conversion of our bespoke dataframe handling into the use of Arrow.

Look for that upgrade to hit Streamlit sometime in the next ~2 months.

I'm adding your user story into the product database for charts and performance -- thanks again for the code, it's really helpful!

[jroes]

@kantuni Is it true that Arrow will improve performance for maps as well?

[kantuni]

The Arrow serialization is used for most of the charts, but not for st.pydeck_chart. That being said, I see a substantial increase in performance compared to 0.73.1.

[neibla]

@kantuni thanks for that, I just tried out 1.0 and it looks pretty smooth with that demo project now!

[hayk-skydio]

@kantuni @nthmost - chiming in here to say that for me, using a pydeck chart is still significantly slower in the frontend with st.pydeck_chart than raw HTML. I'm testing the code below and I noticed for 41k points Jupyter would have smooth 60hz pan/zoom, whereas in streamlit it's 2-5hz on my macbook. I discovered that rendering with components.html in streamlit is equivalently fast.

This is slow (but preserves view state when re-rendering):
st.pydeck_chart(deck)

This is fast:
components.html(deck.to_html(as_string=True), height=600)

Code:

    layer = pdk.Layer(
        "ScatterplotLayer",
        df,
        stroked=True,
        filled=True,
        radius_scale=1,
        radius_min_pixels=3,
        radius_max_pixels=10,
        line_width_scale=0.1,
        line_width_min_pixels=1,
        get_position=["longitude", "latitude"],
        get_fill_color=[255, 140, 0],
        get_line_color=[0, 0, 0],
    )

    deck = pdk.Deck(
        layers=[layer],
        # initial_view_state=view_state,
        api_keys={"mapbox": MAPBOX_API_KEY},
        map_provider="mapbox",
        map_style="light",
        tooltip={"text": "{flight}"},
    )

[KeeonTabrizi]

@kantuni @nthmost - chiming in here to say that for me, using a pydeck chart is still significantly slower in the frontend with st.pydeck_chart than raw HTML. I'm testing the code below and I noticed for 41k points Jupyter would have smooth 60hz pan/zoom, whereas in streamlit it's 2-5hz on my macbook. I discovered that rendering with components.html in streamlit is equivalently fast.

This is slow (but preserves view state when re-rendering): st.pydeck_chart(deck)

This is fast: components.html(deck.to_html(as_string=True), height=600)

Code:

    layer = pdk.Layer(
        "ScatterplotLayer",
        df,
        stroked=True,
        filled=True,
        radius_scale=1,
        radius_min_pixels=3,
        radius_max_pixels=10,
        line_width_scale=0.1,
        line_width_min_pixels=1,
        get_position=["longitude", "latitude"],
        get_fill_color=[255, 140, 0],
        get_line_color=[0, 0, 0],
    )

    deck = pdk.Deck(
        layers=[layer],
        # initial_view_state=view_state,
        api_keys={"mapbox": MAPBOX_API_KEY},
        map_provider="mapbox",
        map_style="light",
        tooltip={"text": "{flight}"},
    )

@hayk-skydio I found your solution does generally work - but found that when using a GeoJsonLayer - that there is an issue when extruded=False and pickable=True in (pdk.Layer). This combo cause all elements to disappear on mouse-over. There does not appear to be an issue if either pickable=False or extruded=True.

[KeeonTabrizi]

@neibla @hayk-skydio @kantuni @nthmost @jroes

I believe my issue I just filed may be related to this legacy issue which was confirmed/replicated.

Please see here: #5532

Hoping to get some traction on this one :) as I think a fix will dramatically improve / impact anyone using pydeck in streamlit.