Proposition: improve dependency management with pipenv

[Seb0]

Coming in with another proposition for your repo (feel free to dismiss if not important to you or not required :) ):
Fix dependency management using pipenv. Why? This article explains it very well and I could only re-articulate the info from it:

https://realpython.com/pipenv-guide/#dependency-management-with-requirementstxt

And here is the pipenv official doc:
https://docs.pipenv.org/en/latest/basics/

I'm using the same mechanism with docker-compose and pipenv and am happy about this configuration. Just my 2 cents for today.

[Timoeller]

Thanks again for another good idea @Seb0
We will talk about this internally. Our framework is targeted towards making NLP as easy as possible to run and deploy on your infastructure. So pipenv could be a good way to overcome the python dependency mess, especially for production environments.
Some questions pop up though:
Are there any downsides to using pipenv? Are there alternatives? What about other big ML frameworks: what are they using or why are/aren't they using pipenv?
Having answers to these questions would greatly speed up our decision process. So if you want to go ahead and give it the answers a try that would be fantastic.

[Seb0]

Potential Downsides: pipenv's philosophy is, that it wants to establish a deterministic reproducible application. So if you wanted python 2 and python 3 environments -- for example to run as 2 separate environments (e.g. for tests) -- you're out of luck with pipenv.
Also, diving deeper into this, I realized that there seems to be a divide between applications and libraries in terms of how to handle dependency issues in python:

Pipenv is designed solely for packaging applications, per the maintainers' admission. It's not suited for libraries and is not designed to be. If you want to replace pip, you should have a look at Poetry.

Another alternative would be the classical way of pinning dependencies and pip freeze, also explained in article from realpython (see above).
The way you're handling it with setup.py is also a way that some people seem to like.

As to why this is not the case for some of the bigger libs? I don't know. But Data Scientists aren't the best developers sometimes :D

Checkout this article (and maybe even the associated reddit discussion that is linked)
https://medium.com/@DJetelina/pipenv-review-after-using-in-production-a05e7176f3f0

[tholor]

Thanks for your evaluation and the articles!

It's not suited for libraries and is not designed to be.

As I understand this, pipenv is a great project but probably not the best choice for FARM. Since we also didn't face any problems yet due to dependency versioning, this doesn't have highest priority in our backlog. We will therefore (for now) keep the standard approach via setup.py and pin versions when issues pop up.