New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new tutorial on installing packages #369

Merged
merged 7 commits into from Aug 31, 2017

Conversation

Projects
None yet
5 participants
@theacodes
Member

theacodes commented Aug 29, 2017

Towards #301

Note this is written in the new tutorial format - it is intended to be short, concise, minimal, and opinionated. More guides and tutorials will be added to cover the topics presently covered by the existing installing tutorials.

Also note: this is in the new tutorials directory, so it isn't intended to be immediately consumed. Once the supplemental materials (guides, etc.) are available we can replace the current installing tutorial with this one.

@theacodes theacodes requested review from ddbeck and ncoghlan Aug 29, 2017

pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
Installing virtualenv

This comment has been minimized.

@theacodes

theacodes Aug 29, 2017

Member

@ncoghlan @dstufft I'm curious if I even need this section - should we just tell them to use venv since this tutorial is Python 3 focused (we can add a note that for Python 2 they need to install and use virtualenv instead). WDYT?

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

I like the idea of moving Python 2 content to a separate "Setting up Python 2 environments" guide, and then referencing that from relevant locations.

However, for this tutorial, I think this is a place where #118 and #367 apply, so we probably want to start recommending pip install --user pipenv, rather than recommending that folks manage their virtual environments by hand.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

(I initially suggested pipsi install pew, but that introduces a bootstrapping problem, where you first need to do pip install --user pipsi in order to get access to the pipsi command. We may still want to suggest that at some point, but it's orthogonal to the question of recommending an environment manager, and currently still has enough bootstrapping problems that it led to @kennethreitz describing the following optional pipsi-based bootstrapping approach for pipenv: http://docs.pipenv.org/en/latest/advanced.html#fancy-installation-of-pipenv)

This comment has been minimized.

@kennethreitz

kennethreitz Aug 29, 2017

Contributor

Everyone I know uses virtualenv on Python 3, not venv.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

@kennethreitz Same here, but I'm not sure I actually know anyone that didn't either learn Python 2 first, or else works professionally on maintaining Python 2 software :)

Either way, if we recommend pipenv as the default dependency & environment manager, then whether it's using venv or virtualenv underneath becomes a hidden implementation detail.

This comment has been minimized.

@kennethreitz

kennethreitz Aug 29, 2017

Contributor

@ncoghlan yeah, and pipsi requires users to modify their $PATH, which is just asking too much for a general guide, i think.

@ncoghlan

I like the general structure, but I think it would make sense to use these new tutorials to start recommending the more opinionated higher level tools as the default starting point.

.. Note:: If you're on Linux or macOS, you may need to run this with ``sudo``.
In general, you should avoid invoking pip with ``sudo`` as this installs
packages globally. However, virtualenv and pip are core utilities that
make sense to be installed globally.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

Nope, never ever recommend sudo pip, not even for virtualenv and pip. What folks can do is pip install --upgrade --user virtualenv, since their personally upgraded copy will shadow the system one for them, without being at risk of being overwritten by reinstallation of the system level package.

pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
Installing virtualenv

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

I like the idea of moving Python 2 content to a separate "Setting up Python 2 environments" guide, and then referencing that from relevant locations.

However, for this tutorial, I think this is a place where #118 and #367 apply, so we probably want to start recommending pip install --user pipenv, rather than recommending that folks manage their virtual environments by hand.

pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
Installing virtualenv

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

(I initially suggested pipsi install pew, but that introduces a bootstrapping problem, where you first need to do pip install --user pipsi in order to get access to the pipsi command. We may still want to suggest that at some point, but it's orthogonal to the question of recommending an environment manager, and currently still has enough bootstrapping problems that it led to @kennethreitz describing the following optional pipsi-based bootstrapping approach for pipenv: http://docs.pipenv.org/en/latest/advanced.html#fancy-installation-of-pipenv)

This tutorial walks you through installing and using Python packages. It will
show you how to install and use the necessary tools and make strong
recommendations on best practices. Keep in mind that the Python packging

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

Typo: packging

This tutorial walks you through installing and using Python packages. It will
show you how to install and use the necessary tools and make strong
recommendations on best practices. Keep in mind that the Python packging
ecosystem is a varied one so this isn't the only way to install and use

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

Slight wording tweak: ...a broad and varied one...

show you how to install and use the necessary tools and make strong
recommendations on best practices. Keep in mind that the Python packging
ecosystem is a varied one so this isn't the only way to install and use
Python packages, however, it does cover the most common use cases.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

Tweak: most common tools and use cases

you read this tutorial make sure you have all the packaging tools by reading
:doc:`installing-packaging-tools`.
Creating a virtualenv

This comment has been minimized.

@ncoghlan

ncoghlan Aug 29, 2017

Member

As per #367, let's lead with pipenv for the new tutorials. So this section would introduce pipenv --three (initialisation), pipenv shell (activation) and pipenv install (installing components)

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 29, 2017

Thanks for using Requests as the example! 🍰

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 29, 2017

Thanks for reviewing @kennethreitz and @ncoghlan. I agree on pipenv - let me throw together a draft on that (hopefully tomorrow) and we'll go from there. That'll likely reduce this tutorial to just one page. :)

Two questions:

  1. Does --user require additional setup? Doesn't the user need to put site.USER_BASE in their PATH? (According to pypa/pip#3813 it looks like that's true).
  2. Is pipenv well tested on Windows? I didn't see anything in the documentation about Windows (e.g. how does pipenv shell work?).

We can add separate guides for using pip/virtualenv for those who want to know.

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 29, 2017

@jonparrott yep, we have a full Windows test suite (down at the moment, but it's historically been running and everything is kept up to date) and put an exhaustive amount of work to support Windows. Pew provides Windows support for pew shell, thankfully.

Windows isn't mentioned in the documentation, because it shouldn't need to be specially mentioned — it should just work.

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 29, 2017

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 29, 2017

I have a get-pipenv Python script (lifted from get-pip) that will install pip and then pipenv for you.

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 29, 2017

@ncoghlan

This comment has been minimized.

Member

ncoghlan commented Aug 29, 2017

Good point re PATH - I'm used to pip install --user Just Working, since Fedora's defaults are set up that way :)

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 30, 2017

Alright - re-wrote this to use Pipenv. :) Please take another look.

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 30, 2017

\o/

@kennethreitz

🍰

python --version
You should get some output like ``3.6.1``. If you do not have Python, please

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

Why not 3.6.2?

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

It's just example output (my mac has 3.6.1)

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

I'd just document it as 3.6.2, personally, even if I had 3.6.1 installed.

This comment has been minimized.

@theacodes
.. code-block:: python
pip install --user pipenv

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor
.. code-block:: bash

    $ pip install --user pipenv
you'll need to add the `user base`_'s ``bin`` directory to your ``PATH``.
You can find the user base by running ``import site; print(site.USER_BASE)``
in Python. For example, on Linux this will return ``~/.local`` so you'll
need to add ``~/.local/bin`` to your ``PATH``.

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

Maybe add a note that this is typically done in ~/.profile?

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

python -m site is a straightforward command to run that prints the value of USER_BASE (amongst other things)

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

TIL! I switched to using that and added links to documentation on how to update your path.

.. code-block:: bash
cd myproject

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

I really like having $ preceding shell commands

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

But I can understand why you'd want to not have that, if you don't want Windows users to feel alienated. I don't think it harms them, though.

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

Based on user research from cloud.google.com we determined that users indeed do get tripped up on this (as they tend to copy/paste). There's a css trick to add the $ in the margin so that it's not picked up on copy, but that would need to be done separately from this PR. The blocks are all annotated as bash, so if we do decide to do it at a later date it'll be easy.

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

Gotcha!

Creating a Pipfile for this project...
Creating a virtualenv for this project...
Using base prefix '/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6'

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

I def think 3.6.2 should be used for the docs, as best practice :)

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

Again, just example output. It'll be difficult to keep this up-to-date with the latest patch version, but we can likely try to bump for every minor version.

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

I'd just patch it by hand :)

This comment has been minimized.

@theacodes
response = requests.get('https://httpbin.org/ip')
print('Your IP is {}'.format(response.json['origin']))

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

needs to be {0} to work w/ Python 2.6. Not sure if that's a target or not.

This comment has been minimized.

@theacodes
@kennethreitz

Love it!

Next steps
----------
Congratulations, you now know how to install and use Python packages! ✨ 🍰 ✨

This comment has been minimized.

@kennethreitz
@ncoghlan

I really like the direction this is going - just some comments on specifics inline.

`Source <https://github.com/pypa/pipfile>`__
``Pipfile`` and its sister ``Pipfile.lock`` are a replacement for the existing

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

Rather than "... a replacement for the existing standard :ref:pip's ...", I'd word this as "... a higher level application-centric alternative to :ref:pip's lower level ...".

I think that's important, as Pipfile deliberately isn't a complete replacement for requirements.txt, and if you're doing things that don't quite fit the "develop and publish an integrated Python application" usage model, you're quite likely to end up wanting to drop back down to the less opinionated lower level interface.

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

I just lifted this directly from PIpfile's repo, but changed it to match your wording.

recommendations on best practices. Keep in mind that the Python packaging
ecosystem is a broad and varied one so this isn't the only way to install and
use Python packages, however, it does cover the most common tools and use
cases.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

I think we may be able to provide better guidance on applicability here. Perhaps replace the second sentence with something like:

Keep in mind that Python is used for a great many different purposes, and precisely how you want to manage your dependencies may change based on how you decide to publish your software. The guidance presented here is most directly applicable to the development and deployment of network services (including web applications), but is also very well suited to managing development and testing environments for any kind of project.

And then once we have a guide or discussion to link to, we can add a pointer to that with text like "If you find this approach isn't working well for you or your use case, you may want to explore these [other options](<link>)"

This comment has been minimized.

@theacodes
cases.
.. Note:: This guide is written for Python 3, however, these instructions
should work fine on Python 2.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

I'd qualify this as Python 2.7 (so we can ignore 2.6 compatibility)

This comment has been minimized.

@theacodes
python --version
You should get some output like ``3.6.2``. If you do not have Python, please

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

Given your discussion with @kennethreitz , perhaps we could abuse the |release| notation to make it easy to update the Python version that we use for examples in one central place?

Alternatively, we could set rst_prolog to define a custom substitution field: https://stackoverflow.com/questions/36320992/sphinx-documentation-variables/36331678#36331678

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

This is probably a good place to mention the following error with a caveat like "If you're new to programming and the command line in general, you may see an error like this":

>>> python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'python' is not defined

and explain that all of the code blocks in the tutorial are intended to be run at an operating system prompt, rather than at the interactive Python prompt.

https://opentechschool.github.io/python-beginners/en/getting_started.html#what-is-python-exactly was the best tutorial I could find on the exact details of how to open one.

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

I tried the substitution but it doesn't work in code blocks or inline code markup, so it doesn't help here. :(

Great suggestion about the error. Added. :)

Pipenv will install the excellent `Requests`_ library and create a ``Pipfile``
for you in your project's directory. The :ref:`Pipfile` is used to track which
dependencies your project needs in case you need to re-install them, such as
when you share your project with others. You should get output similar to this:

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

I'd add in parentheses (although the exact paths shown will vary based on your operating system and how you installed Python)

This comment has been minimized.

@theacodes
you'll need to add the `user base`_'s ``bin`` directory to your ``PATH``.
You can find the user base by running ``import site; print(site.USER_BASE)``
in Python. For example, on Linux this will return ``~/.local`` so you'll
need to add ``~/.local/bin`` to your ``PATH``.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 30, 2017

Member

python -m site is a straightforward command to run that prints the value of USER_BASE (amongst other things)

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 30, 2017

@theacodes

All comments addressed. @ncoghlan please take another look? :)

`Source <https://github.com/pypa/pipfile>`__
``Pipfile`` and its sister ``Pipfile.lock`` are a replacement for the existing

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

I just lifted this directly from PIpfile's repo, but changed it to match your wording.

recommendations on best practices. Keep in mind that the Python packaging
ecosystem is a broad and varied one so this isn't the only way to install and
use Python packages, however, it does cover the most common tools and use
cases.

This comment has been minimized.

@theacodes
cases.
.. Note:: This guide is written for Python 3, however, these instructions
should work fine on Python 2.

This comment has been minimized.

@theacodes
python --version
You should get some output like ``3.6.2``. If you do not have Python, please

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

I tried the substitution but it doesn't work in code blocks or inline code markup, so it doesn't help here. :(

Great suggestion about the error. Added. :)

you'll need to add the `user base`_'s ``bin`` directory to your ``PATH``.
You can find the user base by running ``import site; print(site.USER_BASE)``
in Python. For example, on Linux this will return ``~/.local`` so you'll
need to add ``~/.local/bin`` to your ``PATH``.

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

TIL! I switched to using that and added links to documentation on how to update your path.

Pipenv will install the excellent `Requests`_ library and create a ``Pipfile``
for you in your project's directory. The :ref:`Pipfile` is used to track which
dependencies your project needs in case you need to re-install them, such as
when you share your project with others. You should get output similar to this:

This comment has been minimized.

@theacodes
Merge branch 'new-tutorial-installing-packages' of github.com:jonparr…
…ott/python-packaging-user-guide into new-tutorial-installing-packages
It's because this command is intended to be run in a *shell* (also called
a *terminal* or *console*). See the Python for Beginners
`getting started tutorial`_ for an introduction to using your operating
system's shell and interacting with Python.

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

love this

site information including the user base. For example, on Linux this will
return ``USER_BASE: '~/.local'`` so you'll need to add ``~/.local/bin`` to
your ``PATH``. On Linux and macOS you can set your ``PATH`` permanently
by `modifying ~/.profile`_. On Windows you can set the user

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

`` around ~/.profile ?

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

rst doesn't like it. :)

This comment has been minimized.

@kennethreitz

kennethreitz Aug 30, 2017

Contributor

Ah, the annoyances or ReST. It's also impossible to have both bold and italics at the same time!

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

It's the worst option except for all of the other options.

network services (including web applications), but is also very well suited to
managing development and testing environments for any kind of project.
.. Note:: This guide is written for Python 3, however, these instructions

This comment has been minimized.

@duggelz

duggelz Aug 30, 2017

This renders as two lines of unformatted text in the preview, is that a quirk of the preview or a bug (or intentional)? It looks like this:

Note

This guide is written for Python 3, however, these instructions should work fine on Python 2.7.

This comment has been minimized.

@theacodes

theacodes Aug 30, 2017

Member

yeah github's preview is garbage

screen shot 2017-08-30 at 11 17 20 am

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 30, 2017

Really excited about this! If I can help accelerate this landing, let me know what needs done.

@ncoghlan

This looks good to me - I'd actually be fine with merging it as is, and then following up with additional PRs.

install the latest 3.x version from `python.org`_ or refer to the
`Installing Python`_ section of the Hitchhiker's Guide to Python.
.. Note:: If you're newcomer and you get an error like this:

This comment has been minimized.

@ncoghlan

ncoghlan Aug 31, 2017

Member

Typo: Missing "a" before newcomer

This comment has been minimized.

@theacodes

theacodes Aug 31, 2017

Member

Good catch, done.

File "<stdin>", line 1, in <module>
NameError: name 'python' is not defined
It's because this command is intended to be run in a *shell* (also called

This comment has been minimized.

@ncoghlan

ncoghlan Aug 31, 2017

Member

Might it be worth emphasising that this note also applies to all the other commands in the tutorial? For example:

It's because this command, and the other suggested commands in this tutorial, are intended ...

This comment has been minimized.

@theacodes
There's more resources you can look at to learn about installing and using
Python packages:
.. TODO Link to additional guides and resources.

This comment has been minimized.

@ncoghlan

ncoghlan Aug 31, 2017

Member

Maybe make these TODOs visible in the rendered version so there aren't just blank spaces while the link targets are being worked on?

This comment has been minimized.

@theacodes
If you find this approach isn't working well for you or your use case, you may
want to explore these other approaches:
.. TODO Link to alternatives

This comment has been minimized.

@ncoghlan

ncoghlan Aug 31, 2017

Member

Suggested initial set of links:

  • briefcase: briefcase is a tool that focuses on defining applications as Python packages, and then using that package definition to generate native application installers for popular client operating systems
  • pex: PEX is a single-file distribution format for executable Python virtual environments as zip archives containing a __main__.py file. Their main runtime requirement is that a suitable Python interpreter be available on the system running the application.
  • pipsi: Similar to pex, pipsi aims to run applications in self-contained virtual environments. Unlike pex, it generates these virtual environments dynamically inside the user's home directory for applications available as Python packages.
  • pew: the underlying virtual environment manager used by pipenv (this is a lower level tool that offers more flexibility in where, when, and how virtual environments are created and updated)
  • pip-tools: automated requirements.txt file management that distinguishes between abstract dependency combinations that are expected to work, and the specific pinned versions that will be tested and/or deployed (this is a lower level tool that offers more flexibility in when and how dependencies are installed and updated)
  • conda: conda is a "cross-platform platform" that manages the Python interpreter and external binary dependencies in addition to managing Python modules. It is most useful in situations where it is easier to ensure that the conda package manager is preinstalled than it is to ensure that a suitable Python version is available.

And then make a note that the above list only covers cross-platform tools, and there are also other platform specific options available if users are only interested in a limited set of platforms.

This comment has been minimized.

@theacodes

theacodes Aug 31, 2017

Member

I don't want to put all of this in this doc, but it does make sense for a general overview guide. Moved these to #367 (comment) for tracking. :)

@theacodes theacodes merged commit f3691de into pypa:master Aug 31, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@theacodes theacodes deleted the theacodes:new-tutorial-installing-packages branch Aug 31, 2017

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 31, 2017

💫🍰💫

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 31, 2017

Does this get auto-deployed?

@theacodes

This comment has been minimized.

Member

theacodes commented Aug 31, 2017

Yes - the Fastly config makes it take a second but it's now published here:

https://packaging.python.org/new-tutorials/installing-and-using-packages/

Feel free to share this link (we need feedback!), but keep in mind the URL is temporary. We'll eventually move this into /tutorials/installing-and-using-packages once I can ensure no content in the existing tutorial will be lost. :)

Thanks for reviewing @ncoghlan and @kennethreitz. :D

@kennethreitz

This comment has been minimized.

Contributor

kennethreitz commented Aug 31, 2017

Replicated on Python-Guide.org and Pipenv.org (with permission from @jonparrott):

Will link to this guide as the canonical source of truth, once the final URL has landed.

@qwcode

This comment has been minimized.

Contributor

qwcode commented Nov 27, 2017

what's the plan to make this the [alternative] official tutorial? as it is, still sitting under a unlinked "new-tutorials" segment?

@ncoghlan @jonparrott

@theacodes

This comment has been minimized.

Member

theacodes commented Nov 27, 2017

We had some contention on this, but the path forward is detailed in #394 (comment). I just have to find some time to work on it.

@ncoghlan

This comment has been minimized.

Member

ncoghlan commented Nov 27, 2017

@jonparrott If you're currently busy, perhaps it would be easier if I put a PR together, so you only needed to review it, rather than write it?

@theacodes

This comment has been minimized.

Member

theacodes commented Nov 27, 2017

Go for it. :) I'm always happy to review.

@ncoghlan

This comment has been minimized.

Member

ncoghlan commented Nov 29, 2017

New PR: #402

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment