PyDeck data update speed question

[cornhundred]

Hi, I'm using the data update method described in the PyDeck Conway's life example with binary transfer to update data in a Scatterplot layer

# using something like this to update the data
layer.data = records
r.update()

I have ~100,000 data points and I am able to update the colors of all my points (by interacting with a different Jupyter widget and selecting a different piece of metadata to color by) but it takes ~4 seconds for the PyDeck visual to update. Is that wait time expected and if not is there a way to optimize the speed?

[cornhundred]

Just to provide a little more background, I am calculating the scatterplot point colors based on metadata for >100,000 points. I want to be able to switch the coloring of my scatterplot between categorical colors and coloring based on some metadata value.

Using deck.gl (see https://observablehq.com/@vizgen/deck-gl-jupyter-spatial-and-umap-v0-2-0) I can pass a function to getFillColor (see below)

  const scatterLayer = new deck.ScatterplotLayer({
      id: 'scatterLayer',
      data: data,
      getPosition: apply_map_morph,
      getFillColor: d => {
        var inst_color
        
        if (['undefined'].includes(select_meta_type)){
          // show dim scatterplot
          inst_color = [50, 50, 50, 100]
        } else {

          // value or category based metadata
          if (select_meta_type === 'number'){
            
            // color by value
            ////////////////////////////////////////            
            if (d[select_meta] > 0){
              inst_color = [255, 0, 0, opacity_scale(d[select_meta])] 
            } else {
              inst_color = [50, 50, 50, 15]
            }
            
          } else {
            
            // color by category
            ////////////////////////////////////////
            var rgb = d3.color(cat_colors[d[select_meta]])
            inst_color = [rgb.r, rgb.g, rgb.b, 255 * cat_opacity]
            
          }            
        }
        
        return inst_color
      },    
      ..

Since PyDeck does not allow passing a function to get_fill_color, I have to pre-calculate the colors and update the data attribute. I'm guessing that having to update the layer's data (including positions and colors) might be limiting the update speed (as opposed to passing less data to deck.gl and having the function calculate the colors).

[ibgreen]

You may be hitting an issue where data is being serialized and deserialized as it is sent to the notebook. I believe there is a binary transport option, though @ajduberstein would need to confirm the details.

[ajduberstein]

When you are providing the data update, you're still converting the pandas DataFrame to a binary format in the Jupyter kernel and then transferring it to the frontend. The bottleneck may be that conversion. How long does the initial render take?

[cornhundred]

Hi @ajduberstein I don't see the transport message in the console on mybinder (I'm using Chrome so I can try another browser). The load time and update times are similar. However, I was passing in three columns: position, color, and cluster (used for the tooltip). Based on my reading of the docs I thought I might need to drop the cluster column to get binary transfer to work and things sped up after removal about 2x (but I'm unsure if binary transfer is happening). Do I need to remove columns that are not used for the visual? If so, can I access the index of the data to perform a lookup for the tooltip data I'm not passing directly in the dataframe?

I'll make an example that can run on mybinder later.

[cornhundred]

@ibgreen it turns out I was not using binary_transport (typo'd as binary_transfer - sorry for the confusion) and @ajduberstein I am able to see the Delivering transport message indicating the type as json-with-binary when I run the notebook on Firefox (not Chrome). However, I am running into an error when I try to update the data (this error does not happen when I have binary_transport set to False and this error happens in the MyBinder for PyDeck).

import pydeck as pdk
import numpy as np
import pandas as pd

# Generate Data
###########################
num_rows = 100000
num_cols = 2
np.random.seed(seed=100)
mat = np.random.rand(num_rows, num_cols) * 1000

rows = range(num_rows)
cols = range(num_cols)
rows = [str(i) for i in rows]
cols = [str(i) for i in cols]

df = pd.DataFrame(data=mat, columns=cols, index=rows)
df['r'] = pd.Series(np.float32(0), index=df.index.tolist())
df['g'] = pd.Series(np.float32(0), index=df.index.tolist())
df['b'] = pd.Series(np.float32(255), index=df.index.tolist())

# blue binary transfer dataframe
bt_1 = pd.DataFrame()
bt_1['position'] = df[['0', '1']].values.tolist()
bt_1['color']  = df[['r', 'g', 'b']].values.tolist()

# red binary transfer dataframe
bt_2 = pd.DataFrame()
bt_2['position'] = df[['0', '1']].values.tolist()
bt_2['color']  = df[['b', 'g', 'r']].values.tolist()

# Deck.gl
##################
cell_layer = pdk.Layer(
    id='cell_something',
    type="ScatterplotLayer",
    data=bt_1,
    pickable=True,
    opacity=0.25,
    filled=True,
    radius_scale=3,
    radius_min_pixels=1.25,
    get_fill_color='color',
    get_position='position',
    use_binary_transport=True,    
)

view = pdk.View(
    type="OrthographicView",
    controller=True
)

view_state = pdk.ViewState(
    target=[500, 500, 0],
    zoom=0,
    min_zoom = -4,
    max_zoom = 7
)

# Render
r = pdk.Deck(
    layers=[cell_layer],
    views=[view],    
    initial_view_state=view_state,
    height=800,
)

r.show()

Trying to update the data

# update data and plot 
cell_layer.data = bt_2

results in a key error

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-60f8789883d6> in <module>
----> 1 cell_layer.data = bt_2
      2 r.update()

/usr/local/lib/python3.7/site-packages/pydeck/bindings/layer.py in data(self, data_set)
    125         """
    126         if self.use_binary_transport:
--> 127             self._binary_data = self._prepare_binary_data(data_set)
    128         elif is_pandas_df(data_set):
    129             self._data = data_set.to_dict(orient="records")

/usr/local/lib/python3.7/site-packages/pydeck/bindings/layer.py in _prepare_binary_data(self, data_set)
    153             np_data = np.stack(data_set[column].to_numpy())
    154             # Get rid of the accssor so it doesn't appear in the JSON output
--> 155             del self.__dict__[inverted_accessor_map[column]]
    156             binary_transmission.append(
    157                 {

KeyError: 'get_position'

which happens before I can run r.update(). I can raise this as an issue if necessary.

I also see the same error if I try to udpate the data in the 07 - Binary Transport notebook

nodes_layer.data = nodes.iloc[:1000]

[ajduberstein]

Yep, can confirm this isn't the expected behavior. Thanks for filing #5966. Let me see what I can do to fix it.

[cornhundred]

Thanks @ajduberstein!!