Fix _cpreqbody.Entity to handle Content-Disposition filename* with encoding by pawciobiel · Pull Request #1695 · cherrypy/cherrypy

pawciobiel · 2018-03-01T16:14:39Z

Fix _cpreqbody.Entity to handle Content-Disposition filename* with encoding
Change _cpcompat.unquote_qs to work with unicode on py2
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

fix for #1694

What is the new behavior (if this is a feature change)?

File uploads with
Content-Disposition: form-data; name="myFile"; filename*=utf-8''upload_test_file_%C5%82%C3%B3%C4%85%C3%A4.txt
should work now.

codecov · 2018-03-06T14:32:25Z

Codecov Report

Merging #1695 into master will decrease coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1695      +/-   ##
==========================================
- Coverage   80.73%   80.65%   -0.09%     
==========================================
  Files         105      105              
  Lines       13570    13591      +21     
==========================================
+ Hits        10956    10962       +6     
- Misses       2614     2629      +15

webknjaz · 2018-03-12T22:38:22Z

        fnames = ['boop.csv', 'foo, bar.csv', 'bar, xxxx.csv', 'file"name.csv',
-                  'file;name.csv', 'file; name.csv']
+                  'file;name.csv', 'file; name.csv',
+                  ('test_łóąä.txt', 'utf-8')


m/b u'test_łóąä.txt'?

So I wasn't sure but I think it may be better to pass str not unicode object here
and this is why I put "# coding: utf-8" at the top of the file.

webknjaz

Linters must not fail https://travis-ci.org/cherrypy/cherrypy/jobs/349840614#L482

…encoding * Change _cpcompat.unquote_qs to work with unicode on py2 - fix for #1694

pawciobiel · 2018-04-06T11:25:19Z

@webknjaz I've fixed pylint stuff. Could you give it another chance please?

webknjaz · 2018-06-18T11:22:26Z

                    self.filename = self.filename[1:-1]
+            elif 'filename*' in disp.params:
+                # @see https://tools.ietf.org/html/rfc5987
+                fn_encoding, _, filename = disp.params['filename*'].split("'")


Could you please keep a language part?

webknjaz · 2018-06-18T11:23:13Z

+            elif 'filename*' in disp.params:
+                # @see https://tools.ietf.org/html/rfc5987
+                fn_encoding, _, filename = disp.params['filename*'].split("'")
+                self.filename = unquote_qs(filename, fn_encoding)


I'm not sure whether we'd want to have only one filename field. I think it's appropriate to save all of them.

We could have some @property returning just one thing though.

Hmm. That seems out of scope for this change. This change only seeks to add support for non-ascii filenames. Supporting multiple filenames can be done independently.

Not if it breaks access to the old field

webknjaz · 2018-06-18T11:25:13Z

+        return urllib.parse.unquote(atom_spc, encoding=encoding, errors=errors)
+    else:
+        if isinstance(atom_spc, six.text_type):
+            atom_spc = str(atom_spc)


Is this covered by tests?

webknjaz · 2018-06-18T11:26:46Z

+
+        filename_encoding = None
+        if isinstance(filename, (list, tuple)):
+            filename, filename_encoding = filename


There's too much logic in here. Tests should be simple and keep testing old cases.

webknjaz · 2018-06-18T11:27:45Z

-                  'file;name.csv', 'file; name.csv']
+        fnames = [
+            'boop.csv', 'foo, bar.csv', 'bar, xxxx.csv', 'file"name.csv',
+            'file;name.csv', 'file; name.csv', ('test_łóąä.txt', 'utf-8')


this difference in supplied types is redundant.

I don't understand the comment. The only addition is the one new filename with non-ascii characters. I don't see how that's redundant to all the other names with ascii characters.

I don't remember why I wrote it back then 🤷‍♂️

webknjaz · 2018-06-18T11:28:46Z

@jaraco could you take a look at this, please?

jaraco · 2018-08-07T20:04:48Z

I apologize I haven't had time to review this yet. I still plan to do so.

jaraco · 2018-08-13T17:34:36Z

                    self.filename.endswith('"')
                ):
                    self.filename = self.filename[1:-1]
+            elif 'filename*' in disp.params:


RFC says filename* should be preferred to filename if both are specified, so this should be if and not elif.

jaraco · 2018-08-13T19:39:37Z

+            elif 'filename*' in disp.params:
+                # @see https://tools.ietf.org/html/rfc5987
+                fn_encoding, _, filename = disp.params['filename*'].split("'")
+                self.filename = unquote_qs(filename, fn_encoding)


Hmm. That seems out of scope for this change. This change only seeks to add support for non-ascii filenames. Supporting multiple filenames can be done independently.

jaraco · 2018-08-13T19:46:02Z

+            elif 'filename*' in disp.params:
+                # @see https://tools.ietf.org/html/rfc5987
+                fn_encoding, _, filename = disp.params['filename*'].split("'")
+                self.filename = unquote_qs(filename, fn_encoding)


Is unquote_qs the right operation here? I'm skeptical because it translates + to , which I don't see in the spec. That's also a bug with the existing _cp_compat; it's doing too much already. Let's fix that first.

So I can't remember exactly but I think the way I tested it on some older version was I POSTed a file with spaces and some UTF8 chars in it's name using requests and saw that requests is using quote.

requests is often not a good example of following RFCs, unfortunately.
As for + vs %20 for , I guess it depends on how client decides to encode that. I think I saw something regarding this in query params spec in some rfc.

jaraco · 2018-08-13T19:48:40Z

+        if isinstance(filename, (list, tuple)):
+            filename, filename_encoding = filename
+        if filename_encoding:
+            if six.PY3:


If at all possible, tests shouldn't switch Python versions. It should be possible to write universal code. Switching on Python versions adds complexity and potential pitfalls.

jaraco · 2018-08-13T19:50:20Z

           https://github.com/cherrypy/cherrypy/issues/1146/
-           https://github.com/cherrypy/cherrypy/issues/1397'''
+           https://github.com/cherrypy/cherrypy/issues/1397
+           https://github.com/cherrypy/cherrypy/issues/1694'''


It would have been nice if the preceding commit had formatted without the trailing end symbol, but they didn't so now we get a messy diff. Let's not repeat that mistake and instead terminate the multiline string on another line.

jaraco · 2018-08-13T19:52:03Z

-                  'file;name.csv', 'file; name.csv']
+        fnames = [
+            'boop.csv', 'foo, bar.csv', 'bar, xxxx.csv', 'file"name.csv',
+            'file;name.csv', 'file; name.csv', ('test_łóąä.txt', 'utf-8')


I don't understand the comment. The only addition is the one new filename with non-ascii characters. I don't see how that's redundant to all the other names with ascii characters.

jaraco · 2018-08-13T19:54:27Z

This is a great PR. I have just a few nitpicks... and one thing I wish to correct before this PR is accepted (in unquote_qs). I'll continue to work it from the PR branch.

jaraco · 2018-08-13T20:08:29Z

+    if six.PY3:
+        return urllib.parse.unquote(atom_spc, encoding=encoding, errors=errors)
+    else:
+        if isinstance(atom_spc, six.text_type):


I'm hoping this change is unnecessary, because it complicates the behavior.

jaraco · 2018-08-13T21:28:47Z

I've updated the _cpcompat.py with the changes I wanted to see (conflicting with this PR); tomorrow, I hope to revisit this and implement the required changes herein to address the conflict and other concerns I raised above.

…ames.

Issues were addressed.

webknjaz · 2018-08-16T08:17:00Z

-                 (key, filename))
+
+        fn_key, encoded = encode_filename(filename)
+        tmpl = \


please use braces instead of escaping EOL

cherrypy deleted a comment Mar 7, 2018

webknjaz reviewed Mar 12, 2018

View reviewed changes

webknjaz requested review from a team, jaraco and webknjaz March 12, 2018 22:39

webknjaz previously requested changes Mar 12, 2018

View reviewed changes

* Fix _cpreqbody.Entity to handle Content-Disposition filename* with …

af2ffdf

…encoding * Change _cpcompat.unquote_qs to work with unicode on py2 - fix for #1694

cherrypy deleted a comment Mar 13, 2018

jaraco force-pushed the master branch from b7344b6 to 4ce8547 Compare May 28, 2018 18:15

webknjaz reviewed Jun 18, 2018

View reviewed changes

webknjaz requested review from a team, jaraco and webknjaz and removed request for jaraco June 18, 2018 11:28

jaraco self-assigned this Aug 13, 2018

jaraco requested changes Aug 13, 2018

View reviewed changes

jaraco reviewed Aug 13, 2018

View reviewed changes

Allow 'filename*' to override 'filename' per RFC.

b8181e4

jaraco added 6 commits August 13, 2018 19:30

Assign a variable for the language, even if unused.

f44710a

Restore dequote_qs and pass native string

61534f8

Merge branch 'master' into pawciobiel/1694-fix-for-encoded-filenames

fd60384

Extract function for encoding filenames. Assume UTF-8 for non-ascii n…

046d731

…ames.

Normalize links to bugs.

4790a23

Remove fname resolver, no longer needed.

ffe4ecf

jaraco approved these changes Aug 14, 2018

View reviewed changes

jaraco added 2 commits August 14, 2018 07:43

Update changelog.

f034458

Add docstrings.

05cfecd

cherrypy deleted a comment Aug 14, 2018

cherrypy deleted a comment from jaraco Aug 14, 2018

jaraco merged commit f82ca27 into cherrypy:master Aug 14, 2018

webknjaz reviewed Aug 16, 2018

View reviewed changes

webknjaz mentioned this pull request Sep 9, 2018

handling defined filename encoding in Content-Disposition #1694

Closed

1 task

webknjaz mentioned this pull request May 26, 2019

Fix serving files with diacritics in name under their original name properly #1777

Closed

15 tasks

Uh oh!

Conversation

pawciobiel commented Mar 1, 2018

Uh oh!

codecov Bot commented Mar 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

pawciobiel commented Apr 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz commented Jun 18, 2018

Uh oh!

jaraco commented Aug 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawciobiel Aug 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaraco commented Aug 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaraco commented Aug 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Mar 6, 2018 •

edited

Loading

pawciobiel Aug 16, 2018 •

edited

Loading