GLM keeps crashing when running a loop. GPU memory gets full

[khevn]

Kernel died after few hundred iterations with wide data

def lasso_search(m, n, s, verbose = True):
    # generating the data
    X = np.random.uniform(-100, 100, size = (m, n))
    coefs = np.random.randn(n)
    const_coef = np.random.randn(1)

    # index of sparse coefficients
    zero_coef_loc = random.sample(range(n), s)
    coefs[zero_coef_loc] = 0

    y = np.dot(X, coefs) + const_coef

    start = time.time()
    lasso = h2o4gpu.Lasso()
    lasso_model = lasso.fit(X, y)
    print('time to train:', time.time() - start)
    

    zero_coef_index = np.where(lasso_model.X[0] == 0)
    
    if verbose:
        print('original coeffs', coefs)
        print('Lasso coefficients:', lasso_model.X[0])
        print(const_coef)
        print(zero_coef_loc)
        print(zero_coef_index[0])
    
    if np.array_equiv(zero_coef_index[0], np.sort(zero_coef_loc)):
        sr = 1
    else:
        print(zero_coef_index[0], s)
        sr = 1.0 * zero_coef_index[0].shape[0] / s
    
    return sr


m_list = []
s_list = []
sr_list = []

n= 1000
m_ratio = 2
s_ratios = [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]

m = int(m_ratio * n)
for s_ratio in s_ratios:
    s = int(s_ratio * n)
    s_list.append(s)
    for i in range(100):
        print(i, m, n, s)
        sr = lasso_search(m, n, s, verbose = False)
        sr_list.append(sr)

[khevn]

The Kernel crash is due to GPU memory being full. After the model is trained, it's not released from GPU memory. Therefore, the memory gets full very quickly when training multiple models in a loop.

Also, trying del model in python doesn't work

[mdymczyk]

Tried this yesterday and noticed the same problem (though with notebooks, so memory was increasing only by ~2M with each fit). The finish methods (which are supposed to free up all the pointers) also didn't seem to work.

[rockNroll87q]

Any solution to this?
Still present in 0.3.0.9999, with Ridge Regression. Python3.6 crashes with the error:
Cuda failure /root/repo/src/gpu/matrix/matrix_dense.cu:1954 'out of memory'

[pseudotensor]

#204

It's a long-term bug. I recommend using xgboost's glm.

[socathie]

This bug is extremely inconvenient. My jupyter notebook crashes every few iterations and was unable to finish the job..

My current solution is to do the following:
from numba import cuda
and after each iteration, do:
cuda.close()

[sh1ng]

@socathie Could you try Bleeding edge version.