isin is slow

[maartenbreddels]

import numpy as np
import vaex
N = 1_000_000
M = 10
s = np.random.randint(ord('a'), ord('z'), (N, M), np.uint32).view('U10').reshape(N)
s_selection = s[:1000]
df = vaex.from_arrays(s=s)
%%timeit -n3 -r3
df.s.isin(s_selection).count()

Takes 6 seconds! That is way too slow, we should use a hashmap/set here.

[maartenbreddels]

Originally reported by @stephanpcpeters (private via slack)

[joshreback]

Hi, I think this may be the cause of my StackOverflow query here: https://stackoverflow.com/questions/62292479/performance-tips-for-using-vaex. Are there any workarounds for this?

[maartenbreddels]

No workaround, work for me :)
Hope to release the performance improvement in #822 very soon.

[maartenbreddels]

Let us know if this solve it
pip install "vaex-core>=2.0.3"

[maartenbreddels]

Note that for optimal performance, one should not pass a numpy arrays with strings, if read from hdf5 or arrow, it's fine, otherwise do sth like:

import pyarrow as pa
df = vaex.from_arrays(s=pa.array(s))