Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable IPython History in executing preprocessor #1017

Merged
merged 7 commits into from May 23, 2019
Next

Disable IPython history when preprocessing

IPython saves history in an SQLite DB. This can break in unfortunate ways when
running multiple IPython kernels simultaneously.

NBConvert will now default to disabling history saving in IPython kernels.
  • Loading branch information
alexrudy committed May 6, 2019
commit cb2ef94840731f34a748860d984d4dca342d88be
@@ -220,6 +220,20 @@ class ExecutePreprocessor(Preprocessor):
)
).tag(config=True)

ipython_hist_file = Unicode(
default_value=':memory:',

This comment has been minimized.

Copy link
@MSeal

MSeal May 8, 2019

Collaborator

👍 We should have probably had this default a long time ago regardless of parallelism concerns.

help="""Path to file to use for SQLite history database for an IPython kernel.
The specific value `:memory:` (including the colon
at both end but not the back ticks), avoids creating a history file. Otherwise, IPython
will create a history file for each kernel.
When running kernels simultaneously (e.g. via multiprocessing) saving history a single
SQLite file can result in database errors, so using `:memory:` is recommended in non-interactive
contexts.
""").tag(config=True)

kernel_manager_class = Type(
config=True,
help='The kernel manager class to use.'
@@ -268,6 +282,8 @@ def start_new_kernel(self, **kwargs):
'kernelspec', {}).get('name', 'python')
km = self.kernel_manager_class(kernel_name=self.kernel_name,
config=self.config)
if km.ipykernel and self.ipython_hist_file:

This comment has been minimized.

Copy link
@mpacer

mpacer May 13, 2019

Member

Is ipykernel an attribute of a KernelManager? It doesn't seem to be https://jupyter-client.readthedocs.io/en/stable/api/manager.html#jupyter_client.KernelManager

If not, I'd prefer to know where this is being assigned.

This comment has been minimized.

Copy link
@alexrudy

alexrudy May 13, 2019

Author Contributor

It is a property of the Kernel Manager but it isn't documented. I used it to encapsulate the logic about using an ipython specific argument, rather than doing my own check against the kernel name before passing this argument in.

self.extra_arguments += ['--HistoryManager.hist_file={}'.format(self.ipython_hist_file)]
km.start_kernel(extra_arguments=self.extra_arguments, **kwargs)

kc = km.client()
@@ -0,0 +1,46 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from IPython import get_ipython"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"ip = get_ipython()\n",
"assert ip.history_manager.hist_file == ':memory:'"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
@@ -238,6 +238,7 @@ def assert_notebooks_equal(expected, actual):
("Unicode.ipynb", dict(kernel_name="python")),
("UnicodePy3.ipynb", dict(kernel_name="python")),
("update-display-id.ipynb", dict(kernel_name="python")),
("Check History in Memory.ipynb", dict(kernel_name="python")),
]
)
def test_run_all_notebooks(input_name, opts):
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.