Possible memory leak when calling predict() many times

[hsm207]

Describe the bug

I have a huge dataframe that I would like to iterate over and make prediction using contextual bandits. The dataframe fits in my machine (32 GB RAM).

Python quickly crashes because the RAM used rapidly increases every time a call to predict or learn is made.

To Reproduce

Run this code:

import pandas as pd
import numpy as np
from vowpalwabbit import pyvw

vw = pyvw.vw("--cb_explore 2 --cover 5 --quiet")

n = 8000000
data = np.random.normal(size=[n, 8])
data = pd.DataFrame(data)
data["label"] = np.random.binomial(1, 0.5, size=n)

for f0, f1, f2, f3, f4, f5, f6, f7, label in data.itertuples(index=False):
    features = f"f0:{f0} f1:{f1} f2:{f2} f3:{f3} f4:{f4} f5:{f5} f6:{f6} f7:{f7}"
    vw.predict(features)

Expected behavior

Code to execute the dataframe row iteration should run to completion without blowing up RAM usage.

Observed Behavior

RAM rapidly increase to max available RAM when the for loop is executing and then machine hangs.

Environment

What version of VW did you use? 8.7.0.post1
What OS or language did you use? Ubuntu 18.04.3 LTS with Python bindings

Additional context

Are there any alternative to use vw to train a contextual bandit on very long log records?

[hsm207]

@jackgerrits Is building from source the only way to use this fix now?

[jackgerrits]

Yes building from source is the only way to get this until we release 8.8.0.

Please see here for instruction about how to install the python bindings from source: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Python

[hsm207]

Is the link updated?

The Linux part says:

It looks to me like its installing from PyPI

[jackgerrits]

Ah yes sorry about that. Follow the instructions here to get dependencies installed: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Dependencies#ubuntu

Then in the root of the repo run:

python setup.py install

[hsm207]

Thanks. I can confirm that this issue is solved.