[BUG] Development dependencies are included in install_requires

[snazzyfox]

Describe the bug
Installing this library also installs all libraries used for local development (list below). This makes it impossible to use different versions of these tools in development. It also requires installing all of these tools in production for the bot to function, which unnecessarily bloats the installation size by over 80MB (on windows)

Reproducing steps
Install the library as directed in the documentation, pip install discord-py-interactions

What's normally expected
The library itself and only code necessary for running the bot is downloaded and installed.

What actually happened
Development tools for this project are also installed.

If pip is used, other local tools are overwritten with the specific versions used in this project. If poetry is used, the tools are either downgraded to the version pinned in this library, or an error is generated when there is a version conflict. Either way devs using this library cannot use their own tooling.

Output from pip:

Successfully installed Jinja2-3.0.3 MarkupSafe-2.0.1 Pygments-2.10.0 Sphinx-4.1.2 alabaster-0.7.12 babel-2.9.1 backports.entry-points-selectable-1.1.1 black-21.11b1 certifi-2021.10.8 cfgv-3.3.1 charset-normalizer-2.0.9 click-8.0.3 colorama-0.4.4 discord-py-interactions-4.0.1 distlib-0.3.4 docutils-0.17.1 filelock-3.4.0 identify-2.4.0 imagesize-1.3.0 isort-5.9.3 nodeenv-1.6.0 orjson-3.6.3 packaging-21.3 pathspec-0.9.0 platformdirs-2.4.0 pre-commit-2.16.0 pyparsing-3.0.6 regex-2021.11.10 requests-2.26.0 six-1.16.0 snowballstemmer-2.2.0 sphinx-hoverxref-1.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 tomli-1.2.3 urllib3-1.26.7 virtualenv-20.10.0

pipdeptree shows that only aiohttp, and orjson are necessary for this package to function, the rest of them should be removed and instead placed in an extra dependency (e.g. pip install discord-interactions[dev]):

discord-py-interactions==4.0.1
  - aiohttp [required: ==3.7.4.post0, installed: 3.7.4.post0]
    - async-timeout [required: >=3.0,<4.0, installed: 3.0.1]
    - attrs [required: >=17.3.0, installed: 21.2.0]
    - chardet [required: >=2.0,<5.0, installed: 4.0.0]
    - multidict [required: >=4.5,<7.0, installed: 5.2.0]
    - typing-extensions [required: >=3.6.5, installed: 3.10.0.2]
    - yarl [required: >=1.0,<2.0, installed: 1.6.3]
      - idna [required: >=2.0, installed: 3.2]
      - multidict [required: >=4.0, installed: 5.2.0]
  - black [required: ==21.11b1, installed: 21.11b1]
    - click [required: >=7.1.2, installed: 8.0.3]
      - colorama [required: Any, installed: 0.4.4]
    - mypy-extensions [required: >=0.4.3, installed: 0.4.3]
    - pathspec [required: >=0.9.0,<1, installed: 0.9.0]
    - platformdirs [required: >=2, installed: 2.4.0]
    - regex [required: >=2021.4.4, installed: 2021.11.10]
    - tomli [required: >=0.2.6,<2.0.0, installed: 1.2.3]
    - typing-extensions [required: >=3.10.0.0, installed: 3.10.0.2]
    - typing-extensions [required: !=3.10.0.1, installed: 3.10.0.2]
  - colorama [required: ==0.4.4, installed: 0.4.4]
  - flake8 [required: ==3.9.2, installed: 3.9.2]
    - mccabe [required: >=0.6.0,<0.7.0, installed: 0.6.1]
    - pycodestyle [required: >=2.7.0,<2.8.0, installed: 2.7.0]
    - pyflakes [required: >=2.3.0,<2.4.0, installed: 2.3.1]
  - isort [required: ==5.9.3, installed: 5.9.3]
  - orjson [required: ==3.6.3, installed: 3.6.3]
  - pre-commit [required: ==2.16.0, installed: 2.16.0]
    - cfgv [required: >=2.0.0, installed: 3.3.1]
    - identify [required: >=1.0.0, installed: 2.4.0]
    - nodeenv [required: >=0.11.1, installed: 1.6.0]
    - pyyaml [required: >=5.1, installed: 5.4.1]
    - toml [required: Any, installed: 0.10.2]
    - virtualenv [required: >=20.0.8, installed: 20.10.0]
      - backports.entry-points-selectable [required: >=1.0.4, installed: 1.1.1]
      - distlib [required: >=0.3.1,<1, installed: 0.3.4]
      - filelock [required: >=3.2,<4, installed: 3.4.0]
      - platformdirs [required: >=2,<3, installed: 2.4.0]
      - six [required: >=1.9.0,<2, installed: 1.16.0]
  - Sphinx [required: ==4.1.2, installed: 4.1.2]
    - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
    - babel [required: >=1.3, installed: 2.9.1]
      - pytz [required: >=2015.7, installed: 2021.3]
    - colorama [required: >=0.3.5, installed: 0.4.4]
    - docutils [required: >=0.14,<0.18, installed: 0.17.1]
    - imagesize [required: Any, installed: 1.3.0]
    - Jinja2 [required: >=2.3, installed: 3.0.3]
      - MarkupSafe [required: >=2.0, installed: 2.0.1]
    - packaging [required: Any, installed: 21.3]
      - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.6]
    - Pygments [required: >=2.0, installed: 2.10.0]
    - requests [required: >=2.5.0, installed: 2.26.0]
      - certifi [required: >=2017.4.17, installed: 2021.10.8]
      - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
      - idna [required: >=2.5,<4, installed: 3.2]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
    - setuptools [required: Any, installed: 58.3.0]
    - snowballstemmer [required: >=1.1, installed: 2.2.0]
    - sphinxcontrib-applehelp [required: Any, installed: 1.0.2]
    - sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
    - sphinxcontrib-htmlhelp [required: >=2.0.0, installed: 2.0.0]
    - sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
    - sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
    - sphinxcontrib-serializinghtml [required: >=1.1.5, installed: 1.1.5]
  - sphinx-hoverxref [required: ==1.0.0, installed: 1.0.0]

Versions

• I am using discord.py versions 1.7 and below with my code.
  • I am using 2.0 or higher, or a modified fork.
• I am using dis-snek with my code.
• I am not using any of the listed above and am using the library code alone.

[i0bs]

This isn't really a bug. All of the tools you see in our requirements file are simply ported over to install_requires. These tools are needed to run the most minimal bot possible. These tools are also downloaded in case anyone would wish to help contribute to our architecture without needing to manually install them on their own.

We also have hard locked versions of the tools that we download exactly for that problem: having multiple versions of the same tools for development. If you have your virtual environment configured to have these tools install to that, you should have no issues with multiple versions at all.

[snazzyfox]

These tools are needed to run the most minimal bot possible.

This is absolutely not true. Why would I need to install a dozen sphinx plugins and flake8 for a bot to function? It's not like the bot itself needs to generate documentation or lint code. In fact, the minimal bot example on this page still functions without a hitch after I manually deleted all those library files.

These tools are also downloaded in case anyone would wish to help contribute to our architecture without needing to manually install them on their own.

Developers who wish to contribute to this library are free to install these tools using an extra, which is common practice. Force installing them for everyone who install the library "in case they want to contribute" seems like a bad idea.

We also have hard locked versions of the tools that we download exactly for that problem: having multiple versions of the same tools for development. If you have your virtual environment configured to have these tools install to that, you should have no issues with multiple versions at all.

Are you suggesting that bots should be developed in a separate virtualenv where this library is not installed? How would the bot even function in that case? To make it clear - you are force-installing pinned versions of dev tools into the bot that depends on this library. This means they also must be downloaded not only during testing but also in production as well.

For some perspective, the bot that I have running on a (monkeypatched) v3 of this library totals about 70MB after installation. That's including this library and a dozen others including major ones like database interfaces with binaries. It's appalling that this library force installs more than the size of everything else combined in unnecessary downloads.

[i0bs]

Why would I need to install a dozen sphinx plugins and flake8 for a bot to function?

This really falls under the "helps people easily begin to contribute."

Developers who wish to contribute to this library are free to install these tools using an extra, which is common practice.

We can remove some of the dev tools from requirements.txt, but we already use the extra requirement path for setup and configuration. This is also somewhat of a moot point since a vast majority of contributors will not use extras, which is ironic since it's a "common practice." But, just to give benefit of the doubt, we can also update our CONTRIBUTING file to include a better download example to target specifically the extra requirements.

The fact of the matter is, that's a convenience preference. It's perfectly fine to also have them in the regular requirements.txt file. What hurts is seeing this:

Are you suggesting that bots should be developed in a separate virtualenv where this library is not installed?

Yes, I am suggesting that. If you're so worried about specific versions of our requirements potentially tampering with the versions that you use in your project, you should really reconsider how you're installing it. Virtual environments were created in mind of handling something like this. It's up to you if you want to handle it on a global scale, but bots should be developed based off of their current modules/dependencies in their env/venv.

How would the bot even function in that case?

By specifically importing the modules from that virtualenv. IDEs like PyCharm can easily do this, I'm not sure what might handicap other IDEs, or even a text editor such as VSCode from doing this. For reference, I use my own venv in VSCode to handle this.

This means they also must be downloaded not only during testing but also in production as well.

This is probably the best point that I can see being raised in your concern of "dev-only" dependencies being needed in the main project requirements.

[snazzyfox]

This really falls under the "helps people easily begin to contribute."

This falls more under "help people easily contribute but make it hard to use". An analogy would be PCs/phones that come preloaded with 200 pieces of software that cannot be uninstalled or replaced with something people prefer because "it helps people easily get started".

but bots should be developed based off of their current modules/dependencies in their env/venv.

I don't think you understand the point of what a "library" is then. When developing a bot, this library is included as a dependency in their install_requires. What would you do if, for example, aiohttp one day requires you to download 80MB of extra code and stops functioning unless you downgrade most of your tooling?

Virtual environments were created in mind of handling something like this.

Virtual environments are created so that separate projects can have their own environments managed separately. It does not separate a particular project with code that it depends on. What you are actually suggesting is that I use your library in a bot, but at the same time I should not install this library at all. Are you developing this library in a virtualenv where aiohttp is not installed at all?

IDEs like PyCharm can easily do this

Again, that's not how virtualenvs work. Yes, IDEs help create and manage virtualenvs, But you are suggesting that I place the code I write for my project in a SEPARATE virtualenv from the dependencies that I install (but do not develop for).

[i0bs]

An analogy would be PCs/phones that come preloaded with 200 pieces of software that cannot be uninstalled or replaced with something people prefer because "it helps people easily get started".

This is an incorrect analogy. As you mentioned, you can manually delete these. These aren't "uninstallable" parts of the project setup. We'll move forward with moving them to the extras installation list for setup.

What would you do if, for example, aiohttp one day requires you to download 80MB of extra code and stops functioning unless you downgrade most of your tooling?

We have specific versions being downloaded, so this isn't applicable. I see where you're trying to get at, but this is already covered by hard-locking our installations. aiohttp for all intents and purposes should never have to be constantly upgraded to the very latest, that would be a horrible approach. To put this in perspective, discord.py only allows installation of a specific range of that module as a dependency due to python limitations of how far back the async/await iterators are allowed to go.

It does not separate a particular project with code that it depends on.

Yes they do. Although, you would be using pipenv and not regular pip for this. Read this article for more information on this.

But you are suggesting that I place the code I write for my project in a SEPARATE virtualenv from the dependencies that I install

That's not what I'm suggesting. If I was suggesting anything, it's targeting your dependencies to be installed into* that venv.

Here is the key point I've taken from this Issue so far:

• We'll take all of our development related dependencies and put them into extras for setup so people may run the command line pip install -e .[extra] easily when reading our contribution requirements.

If I am missing any, please feel free to mention and correct me

[snazzyfox]

This is an incorrect analogy. As you mentioned, you can manually delete these.

That's a quick manual test I performed to make sure things still work without them. In practice, there will be less contributors to this project than there are users of it. It's unrealistic to expect all non-contributors to run 50 uninstall commands after every install to remove stuff they do not want.

We have specific versions being downloaded, so this isn't applicable.

That is not the point of the discussion. Pinning is the right thing to do - I'm pinned to v3 of this library for now as well. By this logic, I should never upgrade to v4 because it's forcing me to use its dev toolset with no option to choose my own.

Although, you would be using pipenv and not regular pip for this. Read this article for more information on this.

This does not solve the problem at hand - pipenv, poetry, etc are still working on top of virtualenv. Installing this library, regardless of which tool is used, either overwrites any local tools installed or causes version conflicts. I'm using poetry myself which helpfully errors out and points out the conflict as originating from this library, but other tools may not. It is also naive to expect/require that all consumers of a library to use a particular dependency management system when multiple popular ones exist.

Here is the key point I've taken from this Issue so far:

Thank you - this is all I'm after, since this issue is making it impossible for me to update to v4 in any of the code bases I help maintain without rebuilding their dev tooling.