Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Disable IPython History in executing preprocessor #1017
IPython has a feature where command history is written to a SQLite database. When running multiple kernels in parallel, this feature runs into problems, as the SQLite database will be open from several processes, and sometimes hits a locking error.
To fix this, we have disabled SQLite history in IPython Kernels launched from the preprocessing executor, as recommended by ipython/ipython#11460.
This issue was originally brought up in the context of multiprocessing (see nteract/papermill#239) and this PR partially solves that problem, and adds a test to prove that two notebook kernels, started from the same process can be run simultaneously.
I am working on a separate PR to test and maintain multiprocessing support, as there is at least one more issue to be solved.
Thanks @MSeal for all of the help and guidance during PyCon sprints to figure this out.
IPython saves history in an SQLite DB. This can break in unfortunate ways when running multiple IPython kernels simultaneously. NBConvert will now default to disabling history saving in IPython kernels.
I think the following are the outstanding questions from the review of this PR:
We've started the discussion for each of these questions above – I would love more feedback about all of them so that we can merge this PR (rather than having it get stale).
Sorry we've been swamped after PyCon catching up. I agree let's not let this go stale and get some merges. On that note I'll follow-up on getting a jupyter_client release for the other fix you made.
I checked the first box. I think that's fine and I've got some more permissions on that project now to help maintain it.
The other two I find maintainable enough as is. I'm going to merge now and if anyone feels strongly enough they can open an issue or PR to address it differently.
Thanks for the improvement!
This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there: