df.loc much slower compared to pandas

[ayushdas]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux 7 (Core)
• Modin version (modin.__version__): 0.8.3
• Python version: 3.6.8
• Code we can use to reproduce:
  %%timeit -r 4
  import pandas as pd
  d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [5,6]}
  df = pd.DataFrame(d)
  df = df.set_index(['col1', 'col2'])
  df.loc[1]

%%timeit -r 4
import modin.pandas as pd
d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [5,6]}
df = pd.DataFrame(d)
df = df.set_index(['col1', 'col2'])
df.loc[1]

Describe the problem

df.loc is taking much longer to run in Modin pandas as opposed to the vanilla pandas. The timing has been recorded by averaging over 4 runs by running the code in Jupiter notebook using timeit. (refer code snippet for reproducing the code).

Source code / logs

3.41 ms ± 268 µs per loop using vanilla pandas df.loc
29.4 ms ± 514 µs per loop using Modin pandas df.loc

[devin-petersohn]

Hi @ayushdas. Modin is best benchmarked on larger datasets that come from a file. It looks like you are using a 3x2 dictionary to build a dataframe to benchmark.

I will also add that we have spent significant time on optimizing things that are generally slow in pandas and made sure that things that are fast in pandas are reasonably fast. I do not think that saving 26 milliseconds is worth spending the amount of engineering effort to bring Modin's time below that of pandas in this case. In fact a human cannot even independently perceive the difference between 3ms and 29ms, and in the context of an entire script it is just noise.

We have spent our engineering effort on those operations that take many seconds/minutes.

[ayushdas]

Hi @devin-petersohn, the above code is just an example snippet. I actually have a large DataFrame of ~500k rows. Where the indexes are unique identifiers. Accessing the identifier using .loc in pandas seems to be faster than modin. While the diff may just be a few ms, I am running a simulation over 700 days where I access the data using .loc repeatedly. It was expected that the parallelisation should have made the .locates faster in modin. Is it right to expect this? (Also I have tried join and merge to access the data but they aren't faster too).

[devin-petersohn]

Pandas uses an indexing structure to make it fast, close to constant time depending on the passed value(s). We have a similar indexing structure, but indexing into distributed memory is more costly than contiguous memory. There are further optimizations possible, but parallelism will not help with a constant-time algorithm.

Join and merge are not completely optimized yet, we have some parallel implementation but there is more to be done. These explanations of what is supported can be found in the documentation: https://modin.readthedocs.io/

Are you using loc in a loop?

[ayushdas]

Yes, but I think that may not be a good idea in this case. Will think of alternatives.

[devin-petersohn]

apply can be used as an alternative to looping.

We are working on a query optimizer that can detect python loops and compose a single call, but it is not finished yet. It is somewhat complicated to perfectly detect because of early loop exits and skipping.

[ayushdas]

Sounds cool! Thanks.