one_hot_encoding uses too much gpu memory

[sklam]

A user reported that one_hot_encoding with float64 uses too much memory. It was observed that float32 uses the correct amount of memory.

[sklam]

Upon initial investigation using nvidia-smi, I don't find any evidence that is one_hot_encoding overallocating.

[sklam]

This may be a issue with auto printing in ipython notebook (i.e. last line non-assignment line in cell). If the dataframe df appears on the last line of a cell:

In[1]: df

It will be printed and also stored in the Out dictionary. So, a later del df will not fully remove the dataframe from memory because Out has a reference to it. A del Out[1] will work here but it is tedious to deal with that in every cell. Another workaround is to explicitly print(df) or str(df) to avoid Out acquiring a reference on any large object.

[sklam]

The issue is most likely due to how gpu dataframes are used in the notebook. Closing.