Hey @kmike! Thanks for the quick response.
First I thought about leaving only from_crawler, but to keep this backwards compatible (and also because of your comment in the original issue: "add from_crawler support to dupefilters in addition to from_settings") I kept both methods. It follows the same chain of precedence that can be found in https://github.com/scrapy/scrapy/blob/1.4/scrapy/middleware.py#L35-L40
@kmike: Related to this: is it possible that the MiddlewareManager is affected by this too? If I'm not mistaken, all of its subclasses (DownloaderMiddlewareManager, SpiderMiddlewareManager, ExtensionManager, ItemPipelineManager) are created using from_crawler, and none of them are supposed to be overridden by the user (except maybe the pipeline manager using the ITEM_PROCESSOR setting, but that it's not even documented). The only occurrence of the manager using from_settings is in the tests/test_middleware.py file (test_enabled_from_settings)
What do you think about some cleanup in https://github.com/scrapy/scrapy/blob/1.4/scrapy/middleware.py#L28-L58 to keep only the from_crawler method? In a different PR, of course.
@elacuesta I think this PR looks good, it needs just a small testing tweak. It'd be also good to add a test for dupefilters without from_crawler/from_settings methods. What's the reason for supporting them, by the way?
Hello Mikhail! I think I addressed your latest comments (setting the string to compare in the class itself, add a test for dupefilters created without from_crawler/from_settings methods). Not sure what would be the case for the directly created dupefilters, maybe someone needs to do some initialization but doesn't need the crawler nor the settings and just a child class with a custom constructor is enough?
Thanks @elacuesta! Test coverage is not complete because tests still don't check that dupefilter without from_crawler / from_settings methods work: dupefilter you're using inherits from a base class which has these methods.
I didn't realize that before, thanks! I changed the test case, the new class doesn't implement any of the dupefilter methods but I think that's not the point of the test, it's just to ensure the right class is used.
However, I'd prefer to avoid copy-paste, and use the same function to create middlewares/extensions and dupefilters. Such function can be found in #1605, which is almost ready to merge as well; I wonder if you're up to finishing it, and using as a base for your PR :)
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.