File in sub-dir of staticdir in long-path notation would not load #1638

Safihre · 2017-10-10T20:51:27Z

If a staticdir is supplied in Windows long-path notation, serving a file from a subdirectory would fail because os.stat does not like the mixing of forward and backwards slashes.
Tested on both Python 2.7.14 and 3.6.2.

In the included test the old behavior would be to do:

os.stat('\\?\C:\cherrypy\static/index.html')

Which fails due to the mixed slashes, resulting in a 404.

codecov · 2017-10-10T21:05:11Z

Codecov Report

Merging #1638 into master will decrease coverage by 0.1%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1638      +/-   ##
==========================================
- Coverage   77.06%   76.96%   -0.11%     
==========================================
  Files         106      106              
  Lines       14299    14306       +7     
==========================================
- Hits        11020    11010      -10     
- Misses       3279     3296      +17

webknjaz

Could you please deal with reduced coverage?

webknjaz · 2017-10-11T13:06:55Z

cherrypy/lib/static.py

@@ -314,6 +314,10 @@ def staticdir(section, dir, root='', match='', content_types=None, index='',
    branch = request.path_info[len(section) + 1:]
    branch = urllib.parse.unquote(branch.lstrip(r'\/'))

+    # On Windows, correct the path for long-path support
+    if os.name == 'nt':
+        branch = branch.replace('/', '\\')


I think it's better to start relying on pathlib

Pathlib is not available in Python 2.7 or anything below 3.4 so I don't think it applies right now.

https://pypi.python.org/pypi/pathlib/

You want to introduce a whole new dependency just for a 2 line change?

Well, this just looks redundant to me, that's why I'm looking into using this package, which handles path processing way better than such simplistic things.

And I don't truly understand what is this \\? notation and why one would like to combine different delimiter styles. I just want this change, which I cannot check, to not reduce reliability.

Agreed, Windows is weird. Only using this notation we can reliably get all paths (including those with unicode) to work on Windows. The mixing of delimiter styles is inherent to Cherrypy running on Windows, in that sense it's actually surprising that it worked so far. Windows users will supply paths in Windows style.
Unfortunately I have no experience with pathlib and combined with my lack of understanding of cherrypy internals, me implementing it would probably not work out well.
Simple isn't necessarily bad ;)

Okay, let's postpone pathlib integration question, but I'll ask you to document the problem this change solves in code via comment.

webknjaz · 2017-10-11T13:07:11Z

cherrypy/test/test_static.py

@@ -132,6 +132,14 @@ def dynamic(self):
                'error_page.404': error_page_404,
            }
        }
+
+        # Windows specific long-path support
+        if(platform.system() == 'Windows'):


Don't use parens.

Plus this doesn't need to be conditional.

webknjaz · 2017-10-11T13:09:07Z

cherrypy/test/test_static.py

+        if(platform.system() == 'Windows'):
+            rootconf['/static-long'] = {
+                'tools.staticdir.on': True,
+                'tools.staticdir.dir': '\\\\?\\%s' % curdir,


Why don't you use r'string' notation here?

Safihre · 2017-10-11T13:39:00Z

What can I do about the reduced coverage? Have you looked at the report?
It says reduced coverage because it's platform specific and probably codecov is not testing the Windows specific stuff.

webknjaz · 2017-10-11T13:49:30Z

@Safihre It looks like codecov's CI detection script has broke:
https://ci.appveyor.com/project/jaraco/cherrypy/build/1.0.330/job/wljtenhosjpuqhjd#L884

Safihre · 2017-10-11T13:53:29Z

That explains.
Probably because it's building a PR and thus doesn't include secret-vars and thus the key is missing.

webknjaz · 2017-10-11T13:58:37Z

@Safihre it's not related to PR. codecov works w/o tokens for public builds in public CIs

webknjaz · 2017-10-11T14:12:29Z

Refs:

[BUG] Broken AppVeyor autodetection codecov/codecov-python#114 (PR: Fix AppVeyor env var check codecov/codecov-python#115)
CI env vars change broke codecov's CI detection appveyor/ci#1853

webknjaz · 2017-10-12T19:04:48Z

cherrypy/lib/static.py

+    # Python normally converts this but not when the staticdir is
+    # supplied in long-path notation, eg: \\?\C:\static\js/script.js
+    if os.name == 'nt':
+        branch = branch.replace('/', '\\')


Please use r'\'

That's not possible:

E branch = branch.replace('/', r'\') E ^ E SyntaxError: EOL while scanning string literal

EDIT: not possible in general with only 1 char, not just as function parameter.

webknjaz · 2017-10-13T13:22:52Z

@Safihre plz also rebase your branch in order for it to pick up my workaround for codecov's bug

webknjaz · 2017-10-15T09:18:12Z

cherrypy/test/test_static.py

@@ -185,6 +189,14 @@ def test_static(self):
        #   we just check the content
        self.assertMatchesBody('^Dummy stylesheet')

+    @pytest.mark.skipif(platform.system() != 'Windows', reason='Windows only')
+    def test_static_longpath(self):
+        """Test serving of a file in subdir of a Windows long-path staticdir"""


Please add a period at the end of docstring. This is required by PEP 257.

(this is the last nitpick and I'll merge PR after fixing it)

webknjaz · 2017-10-15T09:48:32Z

@Safihre thank you!

webknjaz requested changes Oct 11, 2017

View reviewed changes

cherrypy deleted a comment Oct 11, 2017

cherrypy deleted a comment Oct 12, 2017

webknjaz reviewed Oct 12, 2017

View reviewed changes

cherrypy deleted a comment Oct 12, 2017

cherrypy deleted a comment Oct 13, 2017

webknjaz reviewed Oct 15, 2017

View reviewed changes

File in sub-dir of staticdir in long-path notation would not load

83fad0c

cherrypy deleted a comment Oct 15, 2017

webknjaz approved these changes Oct 15, 2017

View reviewed changes

webknjaz merged commit b2ceaae into cherrypy:master Oct 15, 2017

File in sub-dir of staticdir in long-path notation would not load #1638

File in sub-dir of staticdir in long-path notation would not load #1638

Safihre commented Oct 10, 2017

codecov bot commented Oct 10, 2017 •

edited

webknjaz left a comment

webknjaz Oct 11, 2017

Safihre Oct 11, 2017

webknjaz Oct 11, 2017

Safihre Oct 11, 2017

webknjaz Oct 11, 2017

Safihre Oct 11, 2017

webknjaz Oct 11, 2017

webknjaz Oct 11, 2017

webknjaz Oct 11, 2017

webknjaz Oct 11, 2017

Safihre commented Oct 11, 2017

webknjaz commented Oct 11, 2017

Safihre commented Oct 11, 2017

webknjaz commented Oct 11, 2017

webknjaz commented Oct 11, 2017 •

edited

webknjaz Oct 12, 2017

Safihre Oct 12, 2017 •

edited

webknjaz Oct 15, 2017

webknjaz commented Oct 13, 2017

webknjaz Oct 15, 2017

webknjaz Oct 15, 2017

webknjaz commented Oct 15, 2017

File in sub-dir of staticdir in long-path notation would not load #1638

File in sub-dir of staticdir in long-path notation would not load #1638

Conversation

Safihre commented Oct 10, 2017

codecov bot commented Oct 10, 2017 • edited

Codecov Report

webknjaz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Safihre commented Oct 11, 2017

webknjaz commented Oct 11, 2017

Safihre commented Oct 11, 2017

webknjaz commented Oct 11, 2017

webknjaz commented Oct 11, 2017 • edited

Choose a reason for hiding this comment

Safihre Oct 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Oct 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Oct 15, 2017

codecov bot commented Oct 10, 2017 •

edited

webknjaz commented Oct 11, 2017 •

edited

Safihre Oct 12, 2017 •

edited