PERF-#7150: Reduce peak memory consumption

[anmyachev]

What do these changes do?

Keeping large objects (in our case, dataframes) in local variables when they are no longer needed is extremely memory-intensive. I propose to review all such places and try to free up memory as early as possible.

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
• passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
• signed commit with git commit -s
• Resolves Reduce peak memory consumption #7150
• tests added and passing
• module layout described at docs/development/architecture.rst is up-to-date

[anmyachev]

@YarShev @dchigarev ready for review

[YarShev]

Since we manually delete objects now, how does this affect performance? Did you try running any benchmark?

[anmyachev]

Since we manually delete objects now, how does this affect performance? Did you try running any benchmark?

del statement simply removes a reference to the object, allows the garbage collector to free memory earlier, but does not explicitly call it, so there is no direct impact on performance.

[YarShev]

How much does this reduce peak memory consumption?

[anmyachev]

It depends on the size of the objects in partitions. Here we don't keep one extra copy of the entire dataframe until the end of the function.

[YarShev]

Do you have any results on a benchmark we run regularly? It would be great to look at.

[anmyachev]

No, only for synthetic bench (roughly repeats what happens when performing an operation on axial partitions):

import numpy as np
import pandas as pd
import time

df1 = pd.DataFrame(np.random.rand(10**6, 2 * 10**2))  # ~ 1GB
df2 = pd.DataFrame(np.random.rand(10**6, 2 * 10**2))  # ~ 1GB
partitions = (df1, df2)

dataframe = pd.concat(list(partitions), axis=0, copy=False)
del partitions, df1, df2

result = dataframe.abs()

del dataframe

time.sleep(5)

# case with del save around ~4GB of peak memory consumption

I guess we can just look at the results of the built-in (in CI) memory consumption check?

[arunjose696]

In timedf the memory consumption is calculated by computing peak memory consumption by checking the stats similar to htop by computing the memory in child thread for every 0.001 seconds.

The peak memory calculated is printed as max_system_memory in the CI at the end of benchmark run in timedf if we need to compare memory consumption on any of our regular benchmarks.