How to efficiently display a map with CircleMarker() beyond 1000 rows

[GitAnalyst]

import pandas as pd
import folium
from folium.plugins import FastMarkerCluster

file_url = 'http://www2.census.gov/geo/docs/maps-data/data/gazetteer/2016_Gazetteer/2016_Gaz_zcta_national.zip'
#Pandas usually infers zips are numerics, but we lose our leading zeroes so let's go with the object dtype
df = pd.read_csv(file_url, sep='\t', dtype={'GEOID' : object}) 
df.columns = df.columns.str.strip() #some column names have some padding

df = df.sample(1000)

folium_map = folium.Map(location=[38, -97],
                        zoom_start=4.4,
                        tiles='CartoDB dark_matter')

# These two lines should create FastMarkerClusters
FastMarkerCluster(data=list(zip(df['INTPTLAT'].values, df['INTPTLONG'].values))).add_to(folium_map)
folium.LayerControl().add_to(folium_map)

for index, row in df.iterrows():

    # generate the popup message that is shown on click.
    popup_text = "{}<br> ALAND: {:,}<br> AWATER: {:,}"
    popup_text = popup_text.format(
                      index,
                      row["ALAND_SQMI"],
                      row["AWATER_SQMI"]
                      )

    folium.CircleMarker(location=(row["INTPTLAT"],
                                  row["INTPTLONG"]),
                        radius= row['ALAND_SQMI']/100,
                        color="#007849",
                        popup=popup_text,
                        fill=False).add_to(folium_map)
folium_map

Problem description

I use CircleMarker() to describe my data by varying radius and color variables (I presented here a more primitive example where only radius varies). The problem is that the map with CircleMarker() doesn't show up (in Jupyter Notebook) when data has more than 1000 lines. I can save maps with more data points to an HTML file, but it is SLOW to navigate.

Expected Output

So to solve this I wanted to use marker clusters. I tried both MarkerCluster() and FastMarkerCluster(). My understanding is that MarkerCluster() provides with enough flexibility, but for me the plot doesn't show up when there are more rows in data than 1000. In the example here I tried to use FastMarkerCluster(). But how to use it so that my data points would retain all the features (radius, color, popup) and would be hidden until you zoom in? And I assume this should solve the issue of not displaying the map with larger data sets.

[ArthurSilverberg]

Haven't encountered this exact problem before but the docs suggest changing prefer_canvas to True when declaring the map.

https://python-visualization.github.io/folium/docs-v0.5.0/modules.html

Have you tried that?

[mpickering]

I had an issue liked this and setting prefer_canvas to True made the performance a lot better.

[GitAnalyst]

Thanks for the suggestion! I tried to add this parameter to folium.Map() but unfortunately I can't say that there is a noticeable difference. I found that there was a similar issue of issues with displaying more than 1000 data points (#812).

[mpickering]

Here's the map that I generated - http://mpickering.github.io/maps.html

There are approximately 3000 circles. I am using Circle and not CircleMarker though.

Here's the file which generates the map - https://github.com/mpickering/rg-map/blob/master/scripts/create-leaflet.py

[Conengmo]

Are you displaying the maps directly in Jupyter? If so it can help to save the map as an html file instead and open that.

Another thing is that we've had problems with Jupyter on Chrome, so if you are using Chrome try maybe Firefox instead.

If nothing new comes up I'll close this issue since it seems a duplicate of #812 (and other related issues that also have been closed).

[GitAnalyst]

Thanks @mpickering for suggesting to use Circle. Still doesn't display in Jupyter >1000 rows, but a nice alternative to CircleMarker.
@Conengmo I have trouble with displaying > 1000 rows in Jupyter. In my case, Opera doesn't display > 1000, Edge didn't display at all and Firefox could display more, but with some distortions. I don't use Chrome for Jupyter, but it should be the same as Opera. When I export to HTML I can view even 10 000 rows (although with some lag).

Would be great if CircleMarker would get more integration with marker clusters for performance & visualization reasons and documented in folium examples notebook (no such examples yet).

For now I'll stick with HTML, thanks guys, closing issue!

[jtbaker]

Would be great if CircleMarker would get more integration with marker clusters for performance & visualization reasons and documented in folium examples notebook (no such examples yet).

@GitAnalyst what do you mean by this? There's an example right here; just replace folium.Marker with a valid folium.CircleMarker and should be good to integrate CircleMarkers into the MarkerCluster.

Just create the MarkerCluster as a variable, loop through your data and add any kind of marker to the cluster with .add_to, then add your cluster variable to the map.

[jtbaker]

@GitAnalyst I reworked your code a little bit and found a few possible issues:

1. The LayerControl was added before the markers. This was causing a JS console error in Chrome, I guess due to the insertion order in the template.
2. fill is set to False on your markers, making them hard to see.

with a few modifications, I got these working well using MarkerCluster in the notebook:

import pandas as pd
import folium
from folium.plugins import FastMarkerCluster, MarkerCluster

file_url = 'http://www2.census.gov/geo/docs/maps-data/data/gazetteer/2016_Gazetteer/2016_Gaz_zcta_national.zip'
#Pandas usually infers zips are numerics, but we lose our leading zeroes so let's go with the object dtype
df = pd.read_csv(file_url, sep='\t', dtype={'GEOID' : object}) 
df.columns = df.columns.str.strip() #some column names have some padding

df = df.sample(1000)

folium_map = folium.Map(location=[38, -97],
                        zoom_start=4.4,
                        tiles='CartoDB dark_matter')

mc = MarkerCluster(name="Marker Cluster")

for index, row in df.dropna().iterrows():
    popup_text = "{}<br> ALAND: {:,}<br> AWATER: {:,}".format(
                      index,
                      row["ALAND_SQMI"],
                      row["AWATER_SQMI"]
                      )
    folium.CircleMarker(location=[row["INTPTLAT"],row["INTPTLONG"]],
                        radius= 10,
                        color="red",
                        popup=popup_text,
                        fill=True).add_to(mc)

mc.add_to(folium_map)

folium.LayerControl().add_to(folium_map)
folium_map

[GitAnalyst]

Very well done, @jtbaker , thank you! Unfortunately, even with marker clusters Jupyter doesn't show more than 1000 data points. I was naively expecting that since marker clusters hide data points, then the problem of visualizing only 1000 data points would be solved. But thanks, this is a great additional functionality!