Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: provide temporary IPYTHONDIR for notebook execution in order to avoid race conditions in https://github.com/ipython/ipython/blob/master/IPython/paths.py#L20 upon execution of multiple notebooks at the same time. #1280

Merged
merged 2 commits into from Nov 26, 2021

Conversation

johanneskoester
Copy link
Contributor

@johanneskoester johanneskoester commented Nov 25, 2021

Description

This should fix this error in uncovar: https://github.com/IKIM-Essen/uncovar/runs/4291269611?check_suite_focus=true#step:7:760

Traceback (most recent call last):
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 361, in start
    self.convert_notebooks()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 533, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 498, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 427, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 186, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 204, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 323, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 80, in preprocess
    with self.setup_kernel():
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/client.py", line 460, in setup_kernel
    self.start_new_kernel(**kwargs)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/client.py", line 416, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 96, in ensure_async
    result = await obj
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 657, in start_kernel
    kernel_cmd, kw = self.pre_start_kernel(**kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 292, in pre_start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 210, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 114, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 228, in get_kernel_spec
    resource_dir = self._find_spec_directory(kernel_name.lower())
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 204, in _find_spec_directory
    for kernel_dir in [kd for kd in self.kernel_dirs if os.path.isdir(kd)]:
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 577, in __get__
    return self.get(obj, cls)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 540, in get
    default = obj.trait_defaults(self.name)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 158, in _kernel_dirs_default
    dirs.append(os.path.join(get_ipython_dir(), 'kernels'))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/IPython/paths.py", line 70, in get_ipython_dir
    os.makedirs(ipdir)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/github/home/.ipython'

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

@sonarcloud
Copy link

sonarcloud bot commented Nov 26, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@johanneskoester johanneskoester merged commit 4d70da1 into main Nov 26, 2021
5 of 6 checks passed
@johanneskoester johanneskoester deleted the notebook-ipythondir branch Nov 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant