Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove six-related code and __future__ imports #4121

Merged
merged 54 commits into from Dec 5, 2019
Merged

Remove six-related code and __future__ imports #4121

merged 54 commits into from Dec 5, 2019

Conversation

elacuesta
Copy link
Member

@elacuesta elacuesta commented Nov 3, 2019

Based on #4115 and #4114, supersedes #4003

wRAR and others added 30 commits Oct 31, 2019
@wRAR
Remove Python 2-only tests.
b44bd6f
@wRAR
Restore tests/test_proxy_connect.py and update it to modern mitmproxy.
b0d6f49
@wRAR
Mark failing proxy tests.
439e37f
@wRAR
Fix the skip message for test_download_gzip_response.
186f9d8
@wRAR
Remove six.PY2 and six.PY3 conditionals.
bbd9f4b
@wRAR
Remove unneeded and unused code from XmlItemExporter.
de7789e
@wRAR
Deprecate scrapy.utils.gz.read1.
c2898fd
@wRAR
Remove cStringIO imports.
cea2f5e
@wRAR
Some text function messages cleanup, deprecate to_native_str.
5b70b05
@wRAR
Remove an unused six import.
3ac4b43
@wRAR
Simplify some more imports.
75b1d05
@wRAR
Replace to_native_str calls with to_unicode.
397e883
@wRAR
Remove Py2-only code that checks sys.version_info.
7299e91
@wRAR
Fix a duplicate ref name in docs.
8641231
@wRAR
Fix test_proxy_connect.py for py3.5.
a5eb59b
@wRAR
Use an older mitmproxy for py3.5.
5eb01b6
@wRAR
Improve the test_https_tunnel_without_leak_proxy_authorization_header…
e0c5c72 
… change.
@wRAR
Only xfail test_https_connect_tunnel_error on 3.6+.
3c9963a
@wRAR
Use self.proc.communicate() after killing mitmdump.
4b0cdf7
@wRAR
Rename tests/py3-ignores.txt to tests/ignores.txt.
350aa67
@wRAR @Gallaecio
Improve the dbm module ref.
48b8ac6 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
@elacuesta
Remove __future__ imports
415526d
@elacuesta
Remove __future__ imports from tests
c0bfaef
@elacuesta
Remove six.moves occurrences
df00389
@elacuesta
Remove six.moves occurrences from tests
5ab0f18
@elacuesta
Remove six.iter* occurrences
1aba513
@elacuesta
Fix bad import
68bf192
@elacuesta
Remove six type wrappers
ce8e515
@elacuesta
Remove six imports
54a786b
@elacuesta
Remove six.get_method_*
ac62524
@elacuesta elacuesta changed the title Remove six-related code Remove six-related code and __future__ imports Nov 3, 2019
@elacuesta elacuesta added this to the v2.0 milestone Nov 3, 2019
@elacuesta elacuesta mentioned this pull request Nov 3, 2019
Merged
@elacuesta elacuesta marked this pull request as ready for review Nov 3, 2019
Gallaecio
Gallaecio reviewed Nov 5, 2019
View changes
Copy link
Member

@Gallaecio Gallaecio left a comment

Amazing work.

I’ve left a lot of feedback, but mostly aesthetic. Feel free to ignore and mark as resolve those aesthetic changes not affecting lines that you actually changed.

scrapy/exporters.py Show resolved Hide resolved
scrapy/extensions/feedexport.py Show resolved Hide resolved
scrapy/http/headers.py Show resolved Hide resolved
scrapy/linkextractors/sgml.py Show resolved Hide resolved
scrapy/settings/__init__.py Show resolved Hide resolved
tests/test_urlparse_monkeypatches.py Show resolved Hide resolved
tests/test_utils_datatypes.py Outdated Show resolved Hide resolved
tests/test_utils_deprecate.py Show resolved Hide resolved
tests/test_utils_misc/__init__.py Outdated Show resolved Hide resolved
tests/test_utils_python.py Outdated Show resolved Hide resolved
Gallaecio
Gallaecio reviewed Nov 5, 2019
View changes
Copy link
Member

@Gallaecio Gallaecio left a comment •
edited

(accidental duplicate comment)

@elacuesta @Gallaecio
Remove unused import (urllib.parse.unquote)
fe31695 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
@elacuesta
Copy link
Member Author

@elacuesta elacuesta commented Nov 5, 2019

@Gallaecio I agree with the proposed sorting suggestions, but I believe the PR is already large enough as it is (that said, I probably already sorted some of them). If you don't mind, I'd prefer to leave that import cleanup for separate PRs. What do you think?

@elacuesta
Copy link
Member Author

@elacuesta elacuesta commented Nov 22, 2019

Added two pickle-related checks to the bandit skip list (B301, B403). The problem appeared with the change from from six.moves import cPickle as pickle to import pickle. I'm not sure what else to do besides skipping the check, I thought #4135 might make a difference but I'm not sure, since B403 fails early at the import stage.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Nov 22, 2019

Pickle is always a potential danger if it handles uncontrolled input data, I believe that’s why bandit checks it. If we are sure that a given pickle usage is safe, I think we can just mark that specific usage as such.

wRAR
wRAR approved these changes Dec 5, 2019
View changes
@wRAR wRAR merged commit 076f076 into master Dec 5, 2019
1 of 2 checks passed
@wRAR wRAR deleted the remove-six-code branch Dec 5, 2019
@elacuesta elacuesta mentioned this pull request Feb 5, 2020
Merged
@elacuesta elacuesta mentioned this pull request Nov 23, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wRAR wRAR

@Gallaecio Gallaecio

Assignees
No one assigned
Labels
cleanup
Projects
None yet
Milestone
v2.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@elacuesta @Gallaecio @wRAR