Bad values in fds_to_keep

[pcpiela]

I have written a microservice using Klein that wraps backend code for running tensorflow deep vision models. When I try to access the main service endpoint I get the following error:

exception bad value(s) in fds_to_keep

I found another problem description very similar to mine (RasaHQ/rasa#1432) that suggests rolling back the version of the scikit-learn package to 0.19.2 as a workaround. I have verified the workaround, but would like to better understand the root cause of the problem, and figure out the best path forward.

I am wondering whether anyone else has run into this problem.

Thanks. Peter

[glyph]

This is almost certainly a bug in Twisted, not in Klein. What version of Twisted are you using? I’d be interested to know what changed in scikit-learn that interfered with file descriptor inheritance...

[pcpiela]

Sorry for the delay in responding. I am using Twisted 19.2.1. I am looking at the scikit document to get to get a sense for the change that is causing this error.

[pcpiela]

This change could be responsible. Feature The version of joblib bundled with Scikit-learn is now 0.12. This uses a new default multiprocessing implementation, named loky. While this may incur some memory and communication overhead, it should provide greater cross-platform stability than relying on Python standard library multiprocessing. #11741 by the Joblib developers, especially Thomas Moreau and Olivier Grisel.

[glyph]

@pcpiela it would be wonderful if you could coordinate with the scikit-learn developers to file an issue on Twisted so we can address this and make process management more reliable for everybody!