[BUG] Hover tool takes long time to render

[sistemicorp]

Possible regression from bokeh ver 2.3.0 to 2.4.0.
Python 3.8. Chrome Version 93.0.4577.82 (Official Build) (64-bit). Ubuntu 20.

The hover tool is lightening fast to render in v2.3.0, but is very slow in 2.4.0 for the same minimal example.

from bokeh.layouts import layout
from bokeh.io import show
from bokeh.plotting import figure
from bokeh.models import ColumnDataSource, Range1d, LinearScale, LinearAxis, HoverTool

NUM_DATA_POINTS = 48000

source = ColumnDataSource(data={
    "t": [i for i in range(1, NUM_DATA_POINTS)],
    "i": [i for i in range(1, NUM_DATA_POINTS)],
    "y": [i * 2 for i in range(1, NUM_DATA_POINTS)],
})

plot = figure(toolbar_location="above",
              y_range=(0.1, NUM_DATA_POINTS*4),
              x_range=(0, NUM_DATA_POINTS),
              y_axis_type="linear")

line = plot.line(x="t", y="i", line_width=2, source=source, color="green", legend_label="i")
plot.line(x="t", y="y", line_width=4, alpha=0.4, source=source, color="red", legend_label="y")

ht = HoverTool(
    tooltips=[("i", "@i{0.00}"), ("y", "@y{0}"), ("t", "@t{0.000} ms")],
    # >>> Uncomment to make hover tool really slow
    #mode='vline',
    #show_arrow=True,
    #renderers=[line],
)
plot.tools = [ht]
doc_layout = layout()
doc_layout.children.append(plot)
show(doc_layout)

Chrome Inspect console:

bokeh-2.4.0.min.js:520 [Violation] 'requestAnimationFrame' handler took 6005ms
bokeh-2.4.0.min.js:520 [Violation] 'requestAnimationFrame' handler took 5998ms
bokeh-2.4.0.min.js:520 [Violation] 'requestAnimationFrame' handler took 6091ms

[Material-Scientist]

Unfortunately, the issue still persists:

Bokeh 2.3.3 (left) vs Bokeh 2.4.1 (right):

2021-10-18-17-29-20.mp4

Minimal reproducible example:

import xarray as xr
import numpy as np
import pandas as pd
import hvplot.xarray, hvplot.pandas
import holoviews

ds = xr.Dataset(
    data_vars=dict(
        frate=(["time","symbol"], np.random.rand(2307,138))
    ),
    coords=dict(
        symbol=(["symbol"], 
               np.array(['1000SHIBUSDT', '1000XECUSDT', '1INCHUSDT', 'AAVEUSDT', 'ADABUSD',
       'ADAUSDT', 'AKROUSDT', 'ALGOUSDT', 'ALICEUSDT', 'ALPHAUSDT',
       'ANKRUSDT', 'ARUSDT', 'ATAUSDT', 'ATOMUSDT', 'AUDIOUSDT',
       'AVAXUSDT', 'AXSUSDT', 'BAKEUSDT', 'BALUSDT', 'BANDUSDT',
       'BATUSDT', 'BCHUSDT', 'BELUSDT', 'BLZUSDT', 'BNBBUSD', 'BNBUSDT',
       'BTCBUSD', 'BTCDOMUSDT', 'BTCSTUSDT', 'BTCUSDT', 'BTSUSDT',
       'BTTUSDT', 'BZRXUSDT', 'C98USDT', 'CELOUSDT', 'CELRUSDT',
       'CHRUSDT', 'CHZUSDT', 'COMPUSDT', 'COTIUSDT', 'CRVUSDT', 'CTKUSDT',
       'CVCUSDT', 'DASHUSDT', 'DEFIUSDT', 'DENTUSDT', 'DGBUSDT',
       'DODOUSDT', 'DOGEBUSD', 'DOGEUSDT', 'DOTUSDT', 'DYDXUSDT',
       'EGLDUSDT', 'ENJUSDT', 'EOSUSDT', 'ETCUSDT', 'ETHBUSD', 'ETHUSDT',
       'FILUSDT', 'FLMUSDT', 'FTMUSDT', 'FTTBUSD', 'GALAUSDT', 'GRTUSDT',
       'GTCUSDT', 'HBARUSDT', 'HNTUSDT', 'HOTUSDT', 'ICPUSDT', 'ICXUSDT',
       'IOSTUSDT', 'IOTAUSDT', 'IOTXUSDT', 'KAVAUSDT', 'KEEPUSDT',
       'KLAYUSDT', 'KNCUSDT', 'KSMUSDT', 'LINAUSDT', 'LINKUSDT',
       'LITUSDT', 'LRCUSDT', 'LTCUSDT', 'LUNAUSDT', 'MANAUSDT',
       'MASKUSDT', 'MATICUSDT', 'MKRUSDT', 'MTLUSDT', 'NEARUSDT',
       'NEOUSDT', 'NKNUSDT', 'OCEANUSDT', 'OGNUSDT', 'OMGUSDT', 'ONEUSDT',
       'ONTUSDT', 'QTUMUSDT', 'RAYUSDT', 'REEFUSDT', 'RENUSDT', 'RLCUSDT',
       'RSRUSDT', 'RUNEUSDT', 'RVNUSDT', 'SANDUSDT', 'SCUSDT', 'SFPUSDT',
       'SKLUSDT', 'SNXUSDT', 'SOLBUSD', 'SOLUSDT', 'SRMUSDT', 'STMXUSDT',
       'STORJUSDT', 'SUSHIUSDT', 'SXPUSDT', 'THETAUSDT', 'TLMUSDT',
       'TOMOUSDT', 'TRBUSDT', 'TRXUSDT', 'UNFIUSDT', 'UNIUSDT', 'VETUSDT',
       'WAVESUSDT', 'XEMUSDT', 'XLMUSDT', 'XMRUSDT', 'XRPBUSD', 'XRPUSDT',
       'XTZUSDT', 'YFIIUSDT', 'YFIUSDT', 'ZECUSDT', 'ZENUSDT', 'ZILUSDT',
       'ZRXUSDT'])),
        time=(["time"],pd.date_range(start='2019-09-10 08:00:00', end='2021-10-18 00:00:00', periods=2307)),)
)

plot = ds.hvplot.heatmap(
    'time','symbol','frate',cmap='coolwarm',
)
plot

2021-10-18-18-01-53.mp4

[sistemicorp]

Version 2.4.1 "fixed" the slow hover for my scenario (line graph) in general.

However, when I plot a large amount of data, the hover tool rendering slows down the overall UI; I think as the user slides the mouse across the plot, the hover is trying to render and then when the mouse continues off the plot, hover is still computing, the user presses a button (off plot) and the UI is slow to respond. I found that if hover tool is disabled, the UI (button) is responsive.

I tried to review the fix to see if I could suggest an improvement, but I failed.
It looked to me that the hover tool was doing a linear search to find the plot data - could that search be binary?
Could the hover lookup take place if the user held the mouse in position for Xms, thus don't try and track a moving mouse?

[antoniomolram]

Hi,

I still can see the performance issue. I was using 2.4.0 and now I just updated to 2.4.2. Still the performance is still poor. Is working but with a big amount of data the delay is really insane. @sistemicorp describe it quite well. I would like to let it open this issue until this performance issue is clarify

[bryevdv]

@antoniomolram I can't reproduce any issue with the original 50k point line-based MRE in the OP, nor can I reproduce any issue with the huge 300k point heatmap MRE added later, which is why this issue was closed. If you are seeing an issue with some other usage pattern, please open a new issue referring to this one. But note it is absolutely necessary that you provide a complete Minimal Reproducible Example. We cannot investigate what we cannot reproduce.

It looked to me that the hover tool was doing a linear search to find the plot data - could that search be binary?

@sistemicorp Most glyphs use a fast RTree-based spatial index that is much faster even than binary search. If there are any issues, it is not with indexing. Same advice to you you: We cannot investigate what we cannot reproduce. A complete minimal reproducer is always necessary for this kind of issue.

[bryevdv]

For reference here is Bokeh 2.4.2 and the 300k point heatmap (OSX/Safari)

What does "big amount of data" mean? 30k points? 300k? Or do you mean something like 3 million points? If so that is out of scope for Bokeh and you should look at tools like DataShader. But I have no idea which case you might have. Concrete details are always important and necessary up front in order to prevent a lot of needless and time-consuming back and forth and speculation.

[antoniomolram]

hi, yes, is more an 4M points. But for me the interesting part is if a disable the hover is fine, I can zoom the data and is working fine. What maybe can be useful is an option to click the data and see the hover information instead of showing the data just moving the mouse around.

[antoniomolram]

and about the example if I find time and will publish it with random data. I need to generate the random data first.

[bryevdv]

What maybe can be useful is an option to click the data and see the hover information instead of showing the data just moving the mouse around.

You could implement a CustomJS selection callback to update a div or text/labels on the plot. I will state categorically that 4M points is out of scope for normal Bokeh usage and reiterate that Holoviews+Datashader is really probably a much better avenue at that scale.

[stas-sl]

Screen.Recording.2022-01-29.at.16.07.37.mp4

I believe this is a HoloViews issue, that's why I created it there (holoviz/holoviews#5198), though it might be related somehow

Update:
Actually, as discussed there it seems, to be more on bokeh's side rather than holoviews'. It is reproduced if you set hover_color, even if it is just a constant:

import bokeh
import numpy as np
import pandas as pd

from bokeh.plotting import figure, show
from bokeh.io import output_notebook

output_notebook()

n = 30000
ds = pd.DataFrame({'x': np.random.rand(n), 'y': np.random.rand(n), 'z': np.random.rand(n)})

p = figure(tooltips=[('x', '@x'), ('y', '@y'), ('z', '@z')])
p.scatter('x', 'y', source=ds, hover_color='red')
show(p)