Unicode error on invalid filenames

[vext01]

Hey,

I've just installed Ipython and Jupyter into a Python-2.7 virtualenv on my OpenBSD machine.

When invoking ipython notebook the following exception is thrown:

...
Created new window in existing browser session.
[E 11:01:40.917 NotebookApp] Unhandled error in API request
    Traceback (most recent call last):
      File "/tmp/c/lib/python2.7/site-packages/notebook/base/handlers.py", line 436, in wrapper
        result = yield gen.maybe_future(method(self, *args, **kwargs))
      File "/tmp/c/lib/python2.7/site-packages/tornado/gen.py", line 870, in run
        value = future.result()
      File "/tmp/c/lib/python2.7/site-packages/tornado/concurrent.py", line 215, in result
        raise_exc_info(self._exc_info)
      File "/tmp/c/lib/python2.7/site-packages/tornado/gen.py", line 230, in wrapper
        yielded = next(result)
      File "/tmp/c/lib/python2.7/site-packages/notebook/services/contents/handlers.py", line 129, in get
        path=path, type=type, format=format, content=content,
      File "/tmp/c/lib/python2.7/site-packages/notebook/services/contents/filemanager.py", line 346, in get
        model = self._dir_model(path, content=content)
      File "/tmp/c/lib/python2.7/site-packages/notebook/services/contents/filemanager.py", line 249, in _dir_model
        os_path = os.path.join(os_dir, name)
      File "/tmp/c/lib/python2.7/posixpath.py", line 73, in join
        path += '/' + b
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xb0 in position 26: ordinal not in range(128)
[E 11:01:40.919 NotebookApp] {
      "Accept-Language": "en-US,en;q=0.8", 
      "Accept-Encoding": "gzip, deflate, sdch", 
      "Connection": "keep-alive", 
      "Accept": "application/json, text/javascript, */*; q=0.01", 
      "User-Agent": "Mozilla/5.0 (X11; OpenBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36", 
      "Host": "localhost:8888", 
      "Referer": "http://localhost:8888/tree", 
      "Cookie": "__S=Q6uwuZs6NjSkTT3WOAMSt2", 
      "X-Requested-With": "XMLHttpRequest"
    }
[E 11:01:40.919 NotebookApp] 500 GET /api/contents?type=directory&_=1439546499953 (127.0.0.1) 22.39ms referer=http://localhost:8888/tree

The browser also displays the traceback in the space reserved for the list of notebooks.

I am however, able to create a notebook and use the notebook. Just the index page is broken.

Thanks

[minrk]

What do you get from pip list, and python -m platform?

[minrk]

And what is the exact path that's failing?

[vext01]

Hey,

Thanks for getting back to me. I've just tried again using a fresh python-2.7 virtualenv. Sure enough, we still have the issue.

The failing path is: http://localhost:8888/tree

pip list:

$ /tmp/ve/bin/pip list
You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
backports.ssl-match-hostname (3.4.0.2)
certifi (2015.9.6.2)
decorator (4.0.2)
functools32 (3.2.3.post2)
ipykernel (4.0.3)
ipython (4.0.0)
ipython-genutils (0.1.0)
Jinja2 (2.8)
jsonschema (2.5.1)
jupyter-client (4.0.0)
jupyter-core (4.0.4)
MarkupSafe (0.23)
mistune (0.7.1)
nbconvert (4.0.0)
nbformat (4.0.0)
notebook (4.0.4)
path.py (8.1.1)
pexpect (3.3)
pickleshare (0.5)
pip (7.0.3)
ptyprocess (0.5)
Pygments (2.0.2)
pyzmq (14.7.0)
setuptools (17.0)
simplegeneric (0.8.1)
terminado (0.5)
tornado (4.2.1)
traitlets (4.0.0)
wheel (0.24.0)

platform:

OpenBSD-5.8-amd64-64bit

[vext01]

I've just tried using a python3.4 virtualenv. The traceback does not occur under this setup.

[minrk]

@vext01 sorry, I meant the filesystem path that's failing. Can you send the path on the filesystem, and a listing of the directory that it's trying to list?

[vext01]

Hey.

I'm not at my computer, but it was just my home directory. So you suspect a
filename is causing the crash? I can try later with another path, and look
for troublesome characters in filenames in my home directory.

Cheers
On 10 Sep 2015 21:54, "Min RK" notifications@github.com wrote:

@vext01 https://github.com/vext01 sorry, I meant the filesystem path
that's failing. Can you send the path on the filesystem, and a listing of
the directory that it's trying to list?

—
Reply to this email directly or view it on GitHub
#287 (comment).

[minrk]

There is a non-ascii path somewhere, causing the failure. It is presumably in one of two places:

1. the directory being listed, or
2. one of the filenames in the listing

[vext01]

Ah, well here is the problem file:

-rw-------    1 edd   edd      1024 May 25  2014 .?    set document.colors.dirs = "black"??    ## document.colors.increase_contrast [

I would have been confused, but I know what this is all about. There was a bug in the elinks web browser (which IIRC is now fixed), which caused elinks to write a file whose name was (some of) the contents of the elinks config file. This is one such file.

Obviously filename is crazy, but we could try to make this a little more robust. Something like this:

--- filemanager.py.orig Fri Sep 11 16:29:30 2015
+++ filemanager.py  Fri Sep 11 16:32:56 2015
@@ -246,7 +246,12 @@
             model['content'] = contents = []
             os_dir = self._get_os_path(path)
             for name in os.listdir(os_dir):
-                os_path = os.path.join(os_dir, name)
+                try:
+                    os_path = os.path.join(os_dir, name)
+                except UnicodeDecodeError as e:
+                    self.log.warn(
+                        "failed to decode filename '%s': %s", name, e)
+                    continue
                 # skip over broken symlinks in listing
                 if not os.path.exists(os_path):
                     self.log.warn("%s doesn't exist", os_path)

Cheers

[Carreau]

do you want to try sending a PR ?