New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support poetry, flit, pipenv, or ...? #3270

Open
stevepiercy opened this Issue Apr 26, 2018 · 20 comments

Comments

Projects
None yet
8 participants
@stevepiercy
Member

stevepiercy commented Apr 26, 2018

This is a placeholder for supporting whatever becomes the successor to pip and managing virtual environments in Pyramid, including poetry, flit, and pipenv. Please add to it.

Questions

  1. How and where to install SUCCESSOR?
  2. Where does installation of SUCCESSOR fit into our current installation documentation?
  3. Are there any features of SUCCESSOR that we would use beside installation of packages and distribution of an app?
  4. What would happen to the environment variable $VENV in most of our commands?
  5. How do we deal with features in setup.py only, specifically entry points of console_scripts and paster.app_factory?
  6. What will we do with the scaffolds and pcreate?
  7. Should we remove instructions for how to make a distributable Pyramid application, and replace it with references to suggested tools?
  8. Assuming a rewrite of commands for SUCCESSOR, do we retain or remove Windows commands for everything except installation and setup of an environment, with a disclaimer that it is up to the user to know how to translate Unix commands to their platform of choice? Refs: #3260 (comment)

Things to update

  1. Documentation wherever pip is used.
  2. Pyramid's packaging related files, including setup.py.
  3. Pyramid cookiecutters.

References

  1. Poetry: https://github.com/sdispater/poetry
  2. Summary article about pipenv: https://realpython.com/pipenv-guide/
@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 26, 2018

The main issue I have seen in my experience using pipenv is that it doesn't mix very cleanly with a setup.py. It's possible but you run into questions about where to define the dependencies. I have one project where the Pipfile is literally just a shim pointing to the setup.py where all the deps were declared. This works pretty well actually but it does feel odd.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6.5"

[packages]
myapp = {path = ".", editable = true}

[dev-packages]
myapp = {path = ".", extras = ["testing"], editable = true}

The workflow here is to basically edit setup.py and then run pipenv lock which updates Pipfile.lock and you never really change the Pipfile.

The main reasons why we use setup.py in our projects is for console_scripts and paster.app_factory entry points. If a) python had a better way to define entry points, or b) if we were less dependent on them, then we could probably embrace pipenv a little bit more cleanly.

@stevepiercy

This comment has been minimized.

Member

stevepiercy commented Apr 26, 2018

What would be a better way to define entry points? Is the PyPA working on this?

This does not bode well:
https://docs.pipenv.org/search/?q=entry+point

@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 26, 2018

Entry points is an issue that pypa is starting to talk about on distutils-sig as the next likely thing to get split out from setuptools... but very slowly. The more practical option I think will be to rip setup.py out of the guide entirely and just show people how to make pyramid apps without needing to make a distributable python package. Lots of projects seem to like that approach - see warehouse for an example.

@stevepiercy

This comment has been minimized.

Member

stevepiercy commented Apr 26, 2018

I've updated and added to the questions above.

Regarding making a distributable Python package, do you mean you want to remove the these two pages?

If so, I'd like to at least mention why someone would want to make a dist and provide references to resources for how to do so, instead of simply deleting those pages from the two wiki tutorials.

Or do you mean something else?

@piotr-dobrogost

This comment has been minimized.

piotr-dobrogost commented Apr 27, 2018

Maybe https://github.com/takluyver/entrypoints could be used instead of setuptools/pkg_resources for entry points?

@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 27, 2018

@piotr-dobrogost Unfortunately the issue here is how to declare entry points, not how to consume them. That project only looks to consume existing *.egg-info/entry_points.txt files in sys.path.

@piotr-dobrogost

This comment has been minimized.

piotr-dobrogost commented Apr 27, 2018

Ok, how about switching from setuptools to flit which does have support for entry points (https://flit.readthedocs.io/en/latest/pyproject_toml.html#entry-points-sections)?

@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 27, 2018

I like flit and would be open to promoting it more. Unfortunately for the purposes of integrating with pipenv it's basically the same as setup.py - it's making your project a redistributable package and thus provides a place to define dependencies which conflicts with the Pipfile. We would need to make a decision where the dependencies are defined. For example I think you'd end up with a Pipfile like I pasted above.

The alternative that integrates with pipenv more cleanly is the flask approach of punting on this and not advocating patterns that use setup.py. You're basically responsible yourself for putting your project on the PYTHONPATH however you want - usually just running it out of the CWD. The real benefits to having an installable package are:

  1. Your project, even in editable mode, can be run from anywhere. You can import it and thanks to pth files it's always on the path.
  2. Your project has a way to describe dependencies.
  3. Your project has a way to provide implementations for entry points.

pipenv's approach (in my experience using it) has been to directly compete with point 2 and doesn't provide a solution for points 1 and 3. It seems mostly useful for pulling together third party projects into a lockfile and less about managing the link between the project-under-development and the virtualenv. Please correct me if anyone disagrees, but this is what I've seen, and yes I have used it for a real project in this way.

@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 30, 2018

https://github.com/sdispater/poetry is by far the best solution I have seen to managing a project yet. I think it blows pipenv's approach out of the water from what I have seen so far and solves all the issues I've mentioned above.

@merwok

This comment has been minimized.

Contributor

merwok commented Apr 30, 2018

Compared to flit for example, it seems like poetry is made by one person who does not follow Python packaging discussions, so I’m not sure about the degree of integration (i.e. implementation of standards) that this tool provides. For example, the author doesn’t see the point of develop installs, and doesn’t accept the uses cases brought up in the ticket.

@bertjwregeer

This comment has been minimized.

Member

bertjwregeer commented Apr 30, 2018

@merwok there are a variety of projects that don't use develop installs at all, for example warehouse doesn't instead it expects warehouse to be in it's PYTHONPATH automatically. Others like Flask and I think Django too, punt on having installable packages, unless you manually do a bunch of work (its not documented, and not the default recommendation) and thus they don't have a need for editable installs.

While we have promoted setup.py and having installable packages for Pyramid applications it is not a requirement. The author also agrees that there are some use cases for editable installs: sdispater/poetry#34 (comment) and seems amenable to adding them for dependencies.

I personally really like the idea of a single file that contains package pins/Information rather than a "Pipfile" vs "setup.py" (or one of it's replacements) where the data is duplicated, or you end up having on rely on the other.

Using flit doesn't do a bunch of the project management things you might have come to expect from tools like npm/yarn.

@mmerickel

This comment has been minimized.

Member

mmerickel commented Apr 30, 2018

Right... poetry solves the editable install issue by basically requiring you to use poetry run or poetry scripts for your commands during development... which, coming from nodejs with yarn run and npm run is actually a really nice thing in my opinion.

Finally, the sdist/wheel it outputs if you do choose to poetry build does contain the appropriate metadata for pip to install it correctly in anyone else's environment. With this in mind I've re-thought the benefits of an editable install entirely. For example, it's a pretty common problem for people to forget to do a pip install -e . after changing entry points. This doesn't suffer from that issue because it makes sure they're correct each time you run your commands in development mode.

On top of that, I guess that as @bertjwregeer said, poetry is adding some editable install support in a future version anyway.

@stevepiercy stevepiercy changed the title from Support pipenv to Support pipenv, flit, poetry, or ...? May 5, 2018

@stevepiercy stevepiercy changed the title from Support pipenv, flit, poetry, or ...? to Support poetry, flit, pipenv, or ...? May 13, 2018

@stevepiercy stevepiercy added the docs label Jul 31, 2018

@josuemontano

This comment has been minimized.

josuemontano commented Aug 25, 2018

After reading this thread I decided to try poetry, this is the project I created: https://github.com/josuemontano/API-platform.

The development configuration works as expected, but in production running poetry build doesn't seem to work properly, gunicorn can't find the package created by poetry -I might be missing something. poetry develop works fine in production.

@bertjwregeer

This comment has been minimized.

Member

bertjwregeer commented Aug 29, 2018

@josuemontano poetry build will create an sdist/wheel from your repository. You'll want to use tooling like pip to install the built packages (see dist/ folder after you run poetry build).

@josuemontano

This comment has been minimized.

josuemontano commented Sep 3, 2018

Got it. Thanks @bertjwregeer! https://github.com/josuemontano/API-platform/blob/master/scripts/predeploy#L3-L4. Hopefully I can make a cookiecutter out of this soon.

@mmerickel

This comment has been minimized.

Member

mmerickel commented Oct 5, 2018

FWIW to get rid of pyramid's requirement for an entry point it's possible to replace use = egg:myapp in your ini file with simply use = call:myapp:main. It's something we could consider just to make the "common case" for people a bit simpler when integrating with pipenv where entry points are not considered something an application would need, only libraries. This would remove the need for a setup.py in most cases and all dependencies would be managed through pipenv directly.

@cjw296

This comment has been minimized.

Member

cjw296 commented Oct 5, 2018

Was any progress ever made of having config optionally come from something other than an ini file?

@tseaver

This comment has been minimized.

Member

tseaver commented Oct 5, 2018

@mmerickel

It's something we could consider just to make the "common case" for people a bit simpler when integrating with pipenv where entry points are not considered something an application would need, only libraries.

I guess the ironies are in the fire, given that entry points were designed for applications, rather than libraries.

@mmerickel

This comment has been minimized.

Member

mmerickel commented Oct 5, 2018

Was any progress ever made of having config optionally come from something other than an ini file?

@cjw296 yes, see pyramid's usage of plaster in newer releases

I guess the ironies are in the fire, given that entry points were designed for applications, rather than libraries.

The irony is not lost on me... but it is lost on the pipenv developers that have closed issues opened about adding entry point support to it with significant hostility. The main thing to point out is that your link references "installed distributions" and "packages" and pipenv explicitly wants nothing to do with these things. It only wants to manage a virtualenv of stuff, and is not interested in helping you develop a package. That's between you and your build system (setuptools/flit/etc). Since entry points are currently defined as package metadata, pipenv has no intention to provide any specific support for them above what I described in #3270 (comment) (installing your package in editable mode with a setup.py or pyproject.toml next to it).

@cjw296

This comment has been minimized.

Member

cjw296 commented Oct 6, 2018

Given the hostile maintainers and lack of functionality, I'd advocate just giving up on pipenv and putting weight behind poetry. I've found it to be a much more pleasant tool to work with, it features a real dependency solver, and the maintainer is polite, pragmatic and practical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment