papermill --prepare-only fails if kernel is missing

[pburka]

When running with --prepare-only, I don't expect papermill to evaluate my notebook. However, if I attempt to process a notebook whose kernel is not installed in the current environment, I get a "No such kernel" error. Is this expected?

$ papermill --prepare-only --parameters_yaml "{BEGIN: 20010101, END: 20071231}" -- ./.ipython/notebooks/papermill-test.ipynb  foo.ipynb
Input Notebook:  ./.ipython/notebooks/papermill-test.ipynb
Output Notebook: foo.ipynb
Traceback (most recent call last):
  File "/nas/dft/ire/pburka/.conda/envs/papermill/bin/papermill", line 11, in <module>
    load_entry_point('papermill===0.16.1', 'console_scripts', 'papermill')()
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/papermill/cli.py", line 124, in papermill
    report_mode=report_mode,
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/papermill/execute.py", line 52, in execute_notebook
    nb = parameterize_notebook(nb, kernel_name, parameters)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/papermill/execute.py", line 110, in parameterize_notebook
    param_content = translate_parameters(kernel_name, parameters)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/papermill/translators.py", line 212, in translate_parameters
    return papermill_translators.kernel_translator(kernel_name).codify(parameters)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/papermill/translators.py", line 21, in kernel_translator
    kernelspec = get_kernel_spec(kernel_name)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/jupyter_client/kernelspec.py", line 367, in get_kernel_spec
    return KernelSpecManager().get_kernel_spec(kernel_name)
  File "/nas/dft/ire/pburka/.conda/envs/papermill/lib/python3.6/site-packages/jupyter_client/kernelspec.py", line 236, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3

If I install the missing kernel, papermill completes as expected.

[VincentBLortie]

Currently, when there is no translator for the notebook's kernel, the installed kernel's kernelspec is used to determine the notebook language (and find an appropriate translator). Perhaps we could use the language specified in the notebook's metadata instead (right next to the kernel name), so that it can be parameterized despite the missing kernel.

There could be an issue if a user has multiple kernels with the same name but different languages on different systems, but that sounds a bit far-fetched...

There's also the case where the --kernel option is used. In that situation I think it'd be best not to rely on anything in the notebook metadata, as it has been overridden. I would continue looking for an installed kernel in that case.

[MSeal]

I think making the logic try language, then kernel name, then lookup kernel for language match would be ideal here. That way a user could register a new translator to the specific kernel/language definition in their ipynb if they really needed to execute without the kernel installed, but still respect custom kernels that define their language.

[VincentBLortie]

So, right now the order of precedence is:

1. kernel name
2. language (from kernelspec)

See translators.py:

def kernel_translator(self, kernel_name):
    kernelspec = get_kernel_spec(kernel_name)
    if kernel_name in self._translators:
        return self._translators[kernel_name]
    elif kernelspec.language in self._translators:
        return self._translators[kernelspec.language]
    raise PapermillException(
        "No parameter translator functions specified for kernel '{}' or language '{}'".format(
            kernel_name, kernelspec.language
        )
    )

Are you saying that you would change this to:

1. language (from ipynb)
2. kernel name
3. language (from kernelspec)

I assumed that kernel name came before language because translators for kernels are, in a way, more specific than translators for a language, no?

[VincentBLortie]

Perhaps:

If --kernel is specified:

1. kernel name (from --kernel)
2. language (from kernelspec)

Else:

1. kernel name (from ipynb)
2. language (from kernelspec)
3. language (from ipynb)

[MSeal]

Actually I think you're right about matching 1) kernel name then 2) language.

Would swapping # 2 and # 3 from the else case make more sense? So we check for static values before requiring a kernelspec to be loaded?

If --kernel is specified:

1. kernel name (from --kernel)
2. language (from kernelspec)

Else:

1. kernel name (from ipynb)
2. language (from ipynb)
3. language (from kernelspec)

[VincentBLortie]

So I've given it a bit more thought and I'm starting to think this is more complicated than it needs to be, or should be.

As far as I know, in normal use, the appropriate translator will be the same whether you look at the ipynb or the kernelspec.

How about just using something like this:

1. kernel name from ipynb
2. language from ipynb

I have trouble coming up with compelling user stories where it would make a difference to base the translator choice on the --kernel option or on the kernelspec. There would need to be a scenario where:

• The translator to be used for the kernel name and/or language saved in the .ipynb is different from the translator to be used for the specified --kernel (saving Python code in an R notebook?) or...
• The .ipynb is saved with kernel name K, which is there indicated to be for language X, but the kernel installed on my system under that same name (K) is actually for a different language, Y.

Basing the translator choice exclusively on the ipynb means that it will yield the same result on different machines. Also, I'm not sure we should change the scope of the --kernel option from "Name of kernel to run" to "Name of kernel to run OR to consider when preparing the notebook"

[MSeal]

I think the comment on "this is more complicated than it needs to be" is right. If a user really wants special translators they can deregister/register their own in place of the built-ins. Let's go with this latest pattern you outlined.

[MSeal]

PR merged, working on a final issue or two before cutting a release to add the changed behavior.