nbconvert failed: [Errno 13] Permission denied

[eburgueno]

I've been trying to figure out the root cause of this for a while, but I can't see what's wrong with our config. When trying to export a notebook as PDF from JupyterLab I am getting a permission denied error:

[userA@host1 ~]$ jupyter nbconvert --to pdf Untitled.ipynb
(...)
PermissionError: [Errno 13] Permission denied: '/software/jupyterhub/share/jupyter/nbconvert/templates/latex/article.tplx'

/software/jupyterhub is where we deployed JupyterLab and share/jupyter is in the data path, but I don't understand why ~/.local/share/jupyter isn't tried first:

[userA@host1 ~]$ jupyter --path
config:
    /home/userA/.jupyter
    /software/jupyterhub/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/userA/.local/share/jupyter
    /software/jupyterhub/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/userA/.local/share/jupyter/runtime

[userA@host1 software]$ jupyter --data-dir
/home/userA/.local/share/jupyter

If /software/jupyterhub/share/jupyter exists and is writeable to userA, the directories nbconvert/templates/latex get created and the conversion works (but only for userA):

[userA@host1 ~]$ ls -l /software/jupyterhub/share/jupyter/
total 152
drwxr-sr-x. 13 swuser  common 301 Jan 13 12:48 kernels
drwxr-sr-x.  8 swuser  common 181 Jan 10 14:25 lab
drwxr-sr-x.  4 swuser  common  62 Jan 13 09:47 nbextensions

[userA@host1 ~]$ touch /software/jupyterhub/share/jupyter/test_rw

[userA@host1 ~]$ jupyter nbconvert --to pdf Untitled.ipynb
[NbConvertApp] Converting notebook Untitled.ipynb to pdf
(...)
[NbConvertApp] PDF successfully created
[NbConvertApp] Writing 28002 bytes to Untitled.pdf

[userA@host1 ~]$ ls -l /software/jupyterhub/share/jupyter/
total 152
drwxr-sr-x. 13 swuser  common 301 Jan 13 12:48 kernels
drwxr-sr-x.  8 swuser  common 181 Jan 10 14:25 lab
drwxrwsr-x.  3 userA   common  27 Jan 14 15:17 nbconvert
drwxr-sr-x.  4 swuser  common  62 Jan 13 09:47 nbextensions
-rw-rw-r--.  1 userA   common   0 Jan 14 15:18 test_rw

[userA@host1 ~]$ ls -lR /software/jupyterhub/share/jupyter/nbconvert/
/software/jupyterhub/share/jupyter/nbconvert/:
total 32
drwxrwsr-x. 3 userA  common 23 Jan 14 15:17 templates

/software/jupyterhub/share/jupyter/nbconvert/templates:
total 32
drwx--S---. 2 userA  common 0 Jan 14 15:17 latex

/software/jupyterhub/share/jupyter/nbconvert/templates/latex:
total 0

If /software/jupyterhub/share/jupyter is NOT writeable to userA, nbconvert works for all users as expected.

Any ideas?

[eburgueno]

I should include what versions we're using, right?

[userA@host1 ~]$ jupyter --version
jupyter core     : 4.6.1
jupyter-notebook : 6.0.1
qtconsole        : 4.6.0
ipython          : 7.11.1
ipykernel        : 5.1.3
jupyter client   : 5.3.4
jupyter lab      : 1.2.4
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.3
traitlets        : 4.3.3

[MSeal]

Thanks for providing all those details -- the versions and what was tried eliminates a lot of things to investigate as you're on the latest version.

So in doing in a little refresher on how data paths work in nbconvert, the logic today is supposed to try each path 1-by-1 until it finds one it can use for the local cache. However when the data path is missing completely it tries to make the missing directories before using them. I can reproduce the issue you have by making the parent directory (/software/jupyterhub/share in this case) owned by root and removing the jupyter dir (/software/jupyterhub/share/jupyter). If you instead have the jupyter dir (/software/jupyterhub/share/jupyter) present but owned by root it correctly tries the next directory in the list. The mkdir code path isn't failing and causing the library to try the next directory as it should be, so I'd categorize this strictly as a bug.

To fix your particular setup I believe making sure that userA has write permissions to /software/jupyterhub/share (and read permissions the /software and /software/jupyterhub should enable it to create the directory as expected in your setup. Alternatively, if you pre-generate /software/jupyterhub/share/jupyter with write permissions for userA and ensure all parent directories have read permissions for userA it should also work.

The logic here predates my involvement in the project but we've been making some larger fixes / rewrites of these things for the 6.0 release. I'll tag this as a bug for the 6.0 milestone.

[eburgueno]

@MSeal Thank you for looking into it!

the logic today is supposed to try each path 1-by-1 until it finds one it can use for the local cache.

That was my assumption, however I don't understand why is not ~/.local/share/jupyter tried in the first instance, considering that should mostly always be read-write. It is also the first one on the list, at least if I'm to believe jupyter_core.paths

>>> from jupyter_core.paths import jupyter_path
>>> print(jupyter_path('nbconvert'))
['/home/userA/.local/share/jupyter/nbconvert', '/software/jupyterhub/share/jupyter/nbconvert', '/usr/local/share/jupyter/nbconvert', '/usr/share/jupyter/nbconvert']

In my testing it didn't matter if /home/userA/.local/share/jupyter/nbconvert existed or not: if /software/jupyterhub/share/jupyter/ was writeable it got used first.

Alternatively, if you pre-generate /software/jupyterhub/share/jupyter with write permissions for userA and ensure all parent directories have read permissions for userA it should also work.

We have a multi-user environment and Jupyter has been deployed as our system user swuser. My workaround has been to set permissions on /software/jupyterhub/share/jupyter to 755, which then works for userA and everyone else by -correctly- writing into ~/.local/share/jupyter (another possibility might be to set permissions on /software/jupyterhub/share/jupyter to 777, but the sysadmin in me strongly dislikes that idea so I didn't even test it).

[MSeal]

In my testing it didn't matter if /home/userA/.local/share/jupyter/nbconvert existed or not: if /software/jupyterhub/share/jupyter/ was writeable it got used first.

Hmm, I'm not sure there. In my local tests I ran tonight it behaved as you just explained / expected it to in terms of directory it tries first. Can you check the permissions and owners on each of the directories for /home/userA/.local/share/jupyter/nbconvert in your setup?

The code involved is rather more complicated than it needs to be. Can you also check TemplateExporter().extra_loaders? Those get added before the path loader for template files. And double check TemplateExporter().get_template_paths() and TemplateExporter().get_prefix_root_dirs() as those have some override settings that might be messing with the paths being searched. This part of the code I dislike a lot but worry about the number of https://xkcd.com/1172/ issues if I cleaned it too dramatically.

[MSeal]

I'm going to close this issue as there's been no follow up in several months. Please post if you want to reopen with additional detials.

[stela2502]

Hi! I am currently using singularity to set up a jupyter notebook in a server HPC environment as this would be usable by us not root users.
Now I have the same problem as described earlier and I wonder - why is this still such a problem here.

try:
  OK=1
  print(x)
except:
  OK =0
  print("did not work")
  
print (OK)

If you could encapsulate the problem like that it would simply be fixed. Please do that!

[stela2502]

So now I have hand fixed this shit!
The function _get_conf in nbconvert/exporters/templateexporter.py is the problem and a simple try: except: block does fix the whole problem.

    def _get_conf(self):
        conf = {}  # the configuration once all conf files are merged
        for path in map(Path, self.template_paths):
            try:
                conf_path = path / 'conf.json'
                if conf_path.exists():
                    with conf_path.open() as f:
                        conf = recursive_update(conf, json.load(f))
            except:
                ## not important
                OK=1

        return conf

What did keep you from applying such a simple fix?

[MSeal]

That is not a fix the problem and has side-effects of silently ignoring config that is often important to proper execution of template conversions.

A better thing to investigate is determining what template paths you don't have permissions to and why your user doesn't have read permissions to those paths. If there's a good reason for this we could discuss what could change.

Since the issue was closed the template and config loading code was substantially overhauled, though read requirements to template paths remains present.

[stela2502]

I define a fix as to fix one problem. And sure it might build up new problems.
The fast is that you already know that there will be a path that is writable and you seam to 'just' want to find the most high level path to read from.
So this function will hopefully do this. The fact that this function just breaks if the path is not creatable is such a serious BUG that the whole package becomes UNUSABLE!
So even a not perfect fix will fix this problem. So please include this fix and fix your other issues in a more sophisticated way?

[stela2502]

I add a warning to the fix, but a warning is all you should actually throw. Not an error.

[stela2502]

The warning would be like this:
_get_config failed on path '/usr/local/share/jupyter/nbconvert/templates/html' - please make sure you have write access if you encounter unexpected behavior
I hope this code is now up to your expectations and you can also follow my train of thought from problem to fix.