[BUG-REPORT] df.shift(period) with negative period returns incorrect result when df has more than 2**20 rows

[mateusglucas]

Description
When using df.shift(period) with negative period in a dataframe with more than 2**20=1048576 rows, an incorrect result is returned.

Software information

• Vaex version: 4.5.0
• Vaex was installed via: pip3
• OS: Ubuntu 20.04.3 LTS

Example

import pandas as pd
import numpy as np
import vaex as vx

N=2**20+1 # 1048577
a = pd.DataFrame()
a['A']=np.linspace(1,N,N)

b = vx.from_pandas(a)

print(b.shift(-1))

#          A
0          1048577.0
1          2.0
2          3.0
3          4.0
4          5.0
...        ...
1,048,572  1048573.0
1,048,573  1048574.0
1,048,574  1048575.0
1,048,575  1048576.0
1,048,576  --

[mateusglucas]

I've found something at line 261 of shift.py:

chunk_size = chunk_size or 1024**2 # TODO: should we have a constant somewhere

Increasing the 1024**2 to a value greater or equal to the row number produces the correct result.

[mateusglucas]

I think this issue is caused by the continue in line 162 in shift.py. When a dataframe has more than 1024**2 rows, this line prevents to yield the first chunk.

[mateusglucas]

Closed via #1608