bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

shihai1991 · 2020-09-12T18:04:33Z

encodings.normalize_encoding() now ignores non-ASCII characters.

https://bugs.python.org/issue39337

update test

Lib/test/test_source_encoding.py

This reverts commit 0fcafb8.

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst

Lib/test/test_source_encoding.py

* origin/master: (147 commits) Fix the attribute names in the docstring of GenericAlias (pythonGH-22594) bpo-39337: Add a test case for normalizing of codec names (pythonGH-19069) bpo-41557: Update Windows installer to use SQLite 3.33.0 (pythonGH-21960) bpo-41976: Fix the fallback to gcc of ctypes.util.find_library when using gcc>9 (pythonGH-22598) bpo-41306: Allow scale value to not be rounded (pythonGH-21715) bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (pythonGH-22595) bpo-41376: Fix the documentation of `site.getusersitepackages()` (pythonGH-21602) Revert "bpo-26680: Incorporate is_integer in all built-in and standard library numeric types (pythonGH-6121)" (pythonGH-22584) bpo-41923: PEP 613: Add TypeAlias to typing module (python#22532) Fix comment about PyObject_IsTrue. (pythonGH-22343) bpo-38605: Make 'from __future__ import annotations' the default (pythonGH-20434) bpo-41905: Add abc.update_abstractmethods() (pythonGH-22485) bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests (pythonGH-22575) bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (pythonGH-22566) Post 3.10.0a1 Python 3.10.0a1 bpo-41584: clarify when the reflected method of a binary arithemtic operator is called (python#22505) bpo-41939: Fix test_site.test_license_exists_at_url() (python#22559) bpo-41774: Tweak new programming FAQ entry (pythonGH-22562) bpo-41936. Remove macros Py_ALLOW_RECURSION/Py_END_ALLOW_RECURSION (pythonGH-22552) ...

Lib/test/test_codecs.py

vstinner · 2020-10-11T12:09:23Z

Lib/test/test_codecs.py

+        self.assertEqual(out, 'utf_8')
+        out = encodings.normalize_encoding('utf   8')
+        self.assertEqual(out, 'utf_8')
+        out = encodings.normalize_encoding('UTF 8')


Would you mind to add a comment to explain that the function does not convert upper case letters to lower case letters, just to make the purpose of this test even more explicit?

To be honest, I don't know how to exact explain it~
I found a case in https://github.com/python/cpython/blob/master/Lib/locale.py#L358.
Looks like It's fine to update encodings.normalize_encoding() to conver to lower-case.

just describe the fact, Lol~

Maybe we can enhance this encodings.normalize_encoding()? I am not sure~

Lib/test/test_codecs.py

vstinner · 2020-10-11T12:12:53Z

Lib/encodings/__init__.py

@@ -61,7 +61,8 @@ def normalize_encoding(encoding):
        if c.isalnum() or c == '.':
            if punct and chars:
                chars.append('_')
-            chars.append(c)
+            if c.isascii():
+                chars.append(c)


I wanted to ask you to add a ".. versionchanged:: 3.10" entry in the documentation, but then I noticed that the encodings module was never documented! Oh!

If end user will use this function or module, I can try to create the doc, but I need some time to do it :)

It can and must be addressed in a separated PR anymore. The lack of documentation should not hold this change.

ok, copy that.

Lib/test/test_codecs.py

vstinner

LGTM.

@taleinat @serhiy-storchaka or @malemburg: Do you want to double check this PR?

Lib/test/test_codecs.py

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst

Doc/whatsnew/3.10.rst

vstinner · 2020-10-14T15:43:37Z

Merged, thanks.

shihai1991 · 2020-10-14T15:47:25Z

thanks victor for your tons of comments ;)

…cters (pythonGH-22219)

encodings.normalize_encoding() should ignore non-ASCII letters

8ae6408

the-knights-who-say-ni added the CLA signed label Sep 12, 2020

bedevere-bot added the awaiting review label Sep 12, 2020

shihai1991 changed the title ~~encodings.normalize_encoding() should ignore non-ASCII letters~~ WIP: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

update test

0fcafb8

update test

shihai1991 commented Sep 12, 2020

View reviewed changes

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

shihai1991 changed the title ~~WIP: encodings.normalize_encoding() should ignore non-ASCII letters~~ [WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

shihai1991 changed the title ~~[WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

Revert "update test"

dea24d8

This reverts commit 0fcafb8.

shihai1991 mentioned this pull request Sep 13, 2020

bpo-37751: Document the change in What's New in Python 3.9 #17997

Closed

shihai1991 mentioned this pull request Sep 29, 2020

bpo-39337: Add a test case for normalizing of codec names #19069

Merged

taleinat requested a review from vstinner September 30, 2020 06:44

vstinner reviewed Oct 1, 2020

View reviewed changes

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst Outdated Show resolved Hide resolved

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

apply victor's comment

dbb0062

shihai1991 changed the title ~~bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters Oct 3, 2020

vstinner reviewed Oct 8, 2020

View reviewed changes

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

shihai1991 added 2 commits October 9, 2020 12:34

apply victor's comment

3fa221f

vstinner reviewed Oct 9, 2020

View reviewed changes

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

apply victor's comments

2e73d13

vstinner reviewed Oct 11, 2020

View reviewed changes

apply victor's comment

95c1d98

vstinner reviewed Oct 12, 2020

View reviewed changes

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

apply victor's comment

03bfd9b

vstinner approved these changes Oct 12, 2020

View reviewed changes

Lib/test/test_codecs.py Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Oct 12, 2020

apply victor's comment

38f28bd

vstinner reviewed Oct 13, 2020

View reviewed changes

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst Outdated Show resolved Hide resolved

apply victor's comment

5982784

vstinner reviewed Oct 13, 2020

View reviewed changes

Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved

apply victor's comment

4ecb8a1

shihai1991 changed the title ~~bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters Oct 13, 2020

vstinner merged commit c5b049b into python:master Oct 14, 2020

bedevere-bot removed the awaiting merge label Oct 14, 2020

xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…

bb71c6d

…cters (pythonGH-22219)

adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…

5959a19

…cters (pythonGH-22219)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

shihai1991 commented Sep 12, 2020 •

edited

vstinner Oct 11, 2020

shihai1991 Oct 12, 2020 •

edited

shihai1991 Oct 12, 2020

shihai1991 Oct 12, 2020

vstinner Oct 11, 2020

shihai1991 Oct 12, 2020

vstinner Oct 12, 2020

shihai1991 Oct 12, 2020

vstinner left a comment

vstinner commented Oct 14, 2020

shihai1991 commented Oct 14, 2020

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

Conversation

shihai1991 commented Sep 12, 2020 • edited

vstinner Oct 11, 2020

Choose a reason for hiding this comment

shihai1991 Oct 12, 2020 • edited

Choose a reason for hiding this comment

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

vstinner Oct 11, 2020

Choose a reason for hiding this comment

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

vstinner Oct 12, 2020

Choose a reason for hiding this comment

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

vstinner left a comment

Choose a reason for hiding this comment

vstinner commented Oct 14, 2020

shihai1991 commented Oct 14, 2020

shihai1991 commented Sep 12, 2020 •

edited

shihai1991 Oct 12, 2020 •

edited