-
Notifications
You must be signed in to change notification settings - Fork 10.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle typing issues hidden by follow_imports=skip #5805
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5805 +/- ##
=======================================
Coverage 88.94% 88.95%
=======================================
Files 162 162
Lines 11002 11011 +9
Branches 1798 1796 -2
=======================================
+ Hits 9786 9795 +9
- Misses 937 938 +1
+ Partials 279 278 -1
|
@@ -11,7 +11,7 @@ jobs: | |||
- python-version: "3.11" | |||
env: | |||
TOXENV: pylint | |||
- python-version: 3.7 | |||
- python-version: 3.8 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed because types-lxml
is not available for 3.7.
3.7 EOL is in 4 months so I don't think this is a big problem.
@@ -83,7 +88,7 @@ def _remove_connection(self, errors: List[BaseException], key: Tuple) -> None: | |||
pending_requests = self._pending_requests.pop(key, None) | |||
while pending_requests: | |||
d = pending_requests.popleft() | |||
d.errback(errors) | |||
d.errback(ResponseFailed(errors)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is indeed not covered and I guess this is why it wasn't caught by tests.
warn_on_generator_with_return_value(spider, request.errback) | ||
dfd = defer_fail(result) | ||
dfd.addErrback(request.errback) | ||
if request.errback: | ||
dfd.addErrback(request.errback) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addErrback(None)
is allowed and does nothing, but it's not documented (there is a comment saying "Default value used to be None" so maybe it was documented before but is now kept for backwards compativility) so I changed this.
@@ -50,7 +58,7 @@ def from_response( | |||
response: TextResponse, | |||
formname: Optional[str] = None, | |||
formid: Optional[str] = None, | |||
formnumber: Optional[int] = 0, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Passing None never worked.
This removes
follow_imports = skip
and fixes/ignores the issues that appeared.There are many ignored issues regardingtwisted.internet.reactor
, it should often be possible to cast it into a suitable class/interface instead but I'm not sure how useful is that and also we still ignore interfaces (there is a WIP mypy plugin for them which I decided not to use). I also need to try ignoringtwisted.internet.reactor
viafollow_imports = skip
but Im not sure if it will work.I've found a typing issue in w3lib (scrapy/w3lib#211) which we should fix and remove from here, and a possible error in the H2 code which we may need to check and rewrite.