model.save_model pickle error, _pickle.PicklingError: Can't pickle <function <lambda> at 0x000002716F0FF620>: it's not found as pytorch_tabular.models.node.utils.<lambda>

[N950]

I know that lambda functions are not pickle-able, and I haven't defined any lambda functions in my file neither in my model params

model.model.save_model(path + "/model/torch_checkpoint")

File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\pytorch_tabular\tabular_model.py", line 643, in save_model
joblib.dump(self.callbacks, os.path.join(dir, "callbacks.sav"))
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 480, in dump
NumpyPickler(f, protocol=protocol).dump(value)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 409, in dump
self.save(obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 781, in save_list
self._batch_appends(obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 805, in _batch_appends
save(x)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 634, in save_reduce
save(state)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 610, in save_reduce
save(args)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 736, in save_tuple
save(element)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 634, in save_reduce
save(state)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 634, in save_reduce
save(state)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 631, in save_reduce
self._batch_setitems(dictitems)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 634, in save_reduce
save(state)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 631, in save_reduce
self._batch_setitems(dictitems)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 852, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 634, in save_reduce
save(state)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 847, in _batch_setitems
save(v)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\joblib\numpy_pickle.py", line 282, in save
return Pickler.save(self, obj)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 922, in save_global
(obj, module_name, name))
_pickle.PicklingError: Can't pickle <function at 0x000002716F0FF620>: it's not found as pytorch_tabular.models.node.utils.

[manujosephv]

yeah.. This was a bug in 0.3. Can you test 0.4.0? I just pushed it to PyPi...

This was because there was a lambda function in the mode definition for NODE.

[N950]

I just tried the 0.4.0, still having the same error from the lambdas in models/node/utils.py, I changed those to functions now I get another error

_pickle.PicklingError: Can't pickle <function NODEModel._build_network.<locals>.subset at 0x00000199E03038C8>: it's not found as pytorch_tabular.models.node.node_model.NODEModel._build_network.<locals>.subset

[manujosephv]

Strange... because it is saving for me. I've even added the saving and loading to pytest..

But looking at the error message, I may know the fix. I have a nested function called subset in that class definition. moving that out may resolve the problem.. Let me make that change and push to develop..

[N950]

Ok I'll test the develop branch once you've pushed

[manujosephv]

have pushed the update to develop .. Can you check now?

[N950]

Still not working, I got the lambda error from node/utils.py

Saving model..
Directory is not empty. Overwriting the contents.
Traceback (most recent call last):
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 269, in _getattribute
obj = getattr(obj, subpath)
AttributeError: module 'pytorch_tabular.models.node.utils' has no attribute ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 918, in save_global
obj2, parent = _getattribute(module, name)
File "C:\Users\asus\anaconda3\envs\dsa2\lib\pickle.py", line 272, in _getattribute
.format(name, obj))
AttributeError: Can't get attribute '' on <module 'pytorch_tabular.models.node.utils' from 'C:\Users\asus\anaconda3\envs\dsa2\lib\site-packages\pytorch_tabular\models\node\utils.py'>

During handling of the above exception, another exception occurred:
....
....
_pickle.PicklingError: Can't pickle <function at 0x0000024C19CBC048>: it's not found as pytorch_tabular.models.node.utils.

[manujosephv]

found a couple more lambda functions in the code. Have replaced them as well and pushed a new commit. Can you check now? I should work.

[N950]

Ok I'll check
Thx for your effort, I really appreciate it

[N950]

It worked !
Thank you very much

[manujosephv]

Glad it worked. Closing the issue. I'll port the fix to main branch soon..