Having the external dependencies in extras_require would be useful to raise
awareness of our optional features, let projects that depend on those features
declare it in the proper syntax (like scrapy[JMESpath]) and have us thinking
twice before adding more of those.
Add optional external dependencies to extras_require
I'm not sure if we need this kind granularity.
For example, if we will ever want to remove JMESPath for some reason, we'll have to keep 'JMESPath' in extras_require because if users have scrapy[JMESPath] in their requirements.txt installation will start to fail. I think it is better to name extra requirements after Scrapy features which are enabled by them, like you've done with scrapy[s3]. E.g. scrapy[shell] can install IPython, etc.
Another option is to just add scrapy[full] extra requirement which will install all optional dependencies.
@kmike if we no longer need the jmespath package to support jmespath, then we can declare an empty set of requirements. if we drop the jmespath feature at all, then it's up to the package depending on it to update its dependencies.
@barraponto you're right there are ways to deal with it. But what's an advantage of writing pip install scrapy[jmespath] instead of pip install scrapy jmespath?
pip install scrapy[jmespath]
pip install scrapy jmespath
That is just because jmespath is both a feature name and a package name. But should the dependencies for that feature change (adding or removing packages), projects depending on scrapy[jmespath] should still work.
@kmike We have control over the version installed :)