New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add outline / rough draft of new Overview page #519

Merged
merged 24 commits into from Aug 12, 2018

Conversation

Projects
None yet
5 participants
@mahmoud
Copy link
Contributor

mahmoud commented May 31, 2018

Following up on @theacodes and my discussion at the PyCon 2018 packaging sprint, this PR brings in a new Overview document, meant to be linked from the packaging.python.org landing page. Also per Thea's instruction, I'm submitting the PR much closer to an "outline" state than a finished document.

This PR also addresses #310, providing much more direct coverage of packaging applications written in Python. This initial PR took a bit longer than expected because there are some intentional changes in direction from the original inspiration:

  1. More direct approach. Not assuming the audience has packaged Python before.
  2. Coverage of web, mobile, PaaS, and other frameworks.
  3. Brief explanation of tools -- (often smaller) applications with a developer audience
  4. Shorter sections so as to function more like a portal

TODOs for this document before the draft is final:

  • Flesh out and edit down sections for an even reading experience.
  • Documentation links for each technology.
  • Open-source project example links (the idea for which gave me a great idea for a survey we could run...)
  • Possibly import illustrations, if folks are interested

Some features I was planning for this doc may best be broken out into dedicated articles for each approach, e.g., "Packaging Python Applications with Snappy", just to keep it maximally browsable. Specifically I think a "Packaging Python Applications with Freezers" would be a great place for a table comparing the myriad options.

Anyways, hope you like the new direction, looking forward to continuing work on this!

@mahmoud mahmoud changed the title Add outline / rough draft of new overview page Add outline / rough draft of new Overview page May 31, 2018

@ncoghlan
Copy link
Member

ncoghlan left a comment

I really like the direction you're going with this - assorted more detailed comments in line :)

^^^^^^^^^^^^^^^^^^^^^^^^^^^

If your code consists of multiple Python files, it's usually organized
into a *package*. A package is any directory containing Python files,

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

We have both import package and distribution package defined in the glossary, so I'd suggest adjusting this section a bit to cover the ambiguity of the "package" term, and linking to those terms the first time they're used.

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

For sure! Definitely want lots of internal links, that's the reason for the season :)

Service platforms
*****************

If you're developing a "Platform-as-a-Service" or "PaaS" like Heroku

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

Missing "for" after "developing"

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

Haha, good catch. Developing a new PaaS would at the very least be a separate document.

* Heroku
* Google App Engine
* PythonAnywhere
* OpenShift

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

Zappa for AWS Lambda would also be worth mentioning here.

libraries and local services. Images can be built to include
everything.
Depending on a framework

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

"deployment system" might be a better phrase here (I initially thought you meant web framework)

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

I was kind of looking for a term that could encapsulate the conventions of Heroku/PaaS as well as stuff like Kivy/Beeware, where as long as you stay within the lines, stuff should just work.

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

I guess I could include a clarifications that not all frameworks include deployment systems, so evaluate carefully, especially if the option isn't on the list.

If you're developing software that will be deployed to machines you
own, users' personal computers, or any other arrangement, read on.

Web and mobile platforms

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

While technically accurate, I think mentioning "Web and ..." in the heading here is more confusing than helpful (since the previous section also mostly targets the web, just for backend network services rather than frontend single-page applications).

Alternatively, expanding it to Web browsers and mobile platforms would likely also work, as that makes it clearer that it isn't referring to backend web services the way the previous section does.

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

Yeah, web clients/browsers and mobile apps would definitely be the better explicit choice. I'll make that change.

************************

Python's advances are leading it into new spaces. These days you can
write a mobile app or web application frontend in Python. While the

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

Along a similar vein to tinkering with the section title, I think sticking a "single page" qualifier before "web application" will help distinguish the scope of this from more traditional server-side template rendering in a web framework.

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

Interesting, yeah, I'll try that out!

and Mac OS version for many years now.

* PEX
* zipapp (doesn't include library dependencies, requires Python 3.5+)

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

s/include/help manage/ (if you use pip install --target to add zip-compatible pure Python deps, zipapp will happily add them to the archive)

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

Yeah, the bullets in these sections are really anemic. I'll be sure to include more discussion of the pros and cons of each option. Even then, I might summarize by saying "minimal support" or something, because that is a really small subset of the libraries out there :)

installed, like Anaconda. Increasingly common in academic, analytical,
and other data-oriented environments. Also used in production services.

* conda/Anaconda

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

Possible alternative structure here: rather than saying "new Python ecosystem", perhaps say "language independent packaging ecosystem", and mention:

  • conda/Anaconda (cross-platform)
  • homebrew (Mac OS X only)
  • rpm (selected Linux distros only)
  • deb (selected Linux distros only)

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

Interesting. Yeah, I suppose there are Python applications out there that depend on RPM-packaged/managed libraries (python-*.i386.rpm, etc.). I'm not sure if I'd recommend it. I probably would want to give conda its own heading regardless, sort of like omnibus, below, because it does represent a pretty unique solution.


Embed your code on an Adafruit or a Micropython, or some other
hardware, and just ship it to the datacenter, or your users' homes,
and call it good.

This comment has been minimized.

@ncoghlan

ncoghlan Jun 1, 2018

Member

Hey, I resemble this remark :)

(DC fast chargers are a teensy bit bigger than a micro:bit, though)

This comment has been minimized.

@mahmoud

mahmoud Jun 1, 2018

Author Contributor

I need to find more projects like that to link to from here. I'm pretty confident we can find open-source exemplars of applications for all the solutions above, but do keep an eye out for good hardware applications.

@ncoghlan

This comment has been minimized.

Copy link
Member

ncoghlan commented Jun 1, 2018

If you have images you want to include, I think that could definitely be a good idea - I know a lot of folks found the layered diagrams useful for explain IaaS vs PaaS vs SaaS, and much of that applies here as well (at least on the application packaging side).

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jun 1, 2018

@ncoghlan Glad you like it, and thanks for the initial review! As far as pictures, I think you're right. A diagram is more helpful than the flavor illustrations of the original blog post. I'll probably make a version of this:

image

I'll ping back when I integrate these changes and make some progress on the TODOs.

@ncoghlan

This comment has been minimized.

Copy link
Member

ncoghlan commented Jun 2, 2018

If you want more examples of "Anaconda-like" situations, you might consider situations like ESRi's ArcGIS Pro, Enthought Canopy, VFX Platform, etc.

VFX Platform in particular is an interesting example, since it's a platform spec that agrees on common versions of various dependencies rather than a particular implementation: http://www.vfxplatform.com/

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jun 3, 2018

Just pushed a commit that addresses most of the points above. Still need to address my initial TODOs.

As far as Canopy and VFX Platform, I think the main thing I didn't find for those was an actual guide to making your own package for the platform. VFX is great, and kinda reminds me of manylinux, but without a reference/repo with the docker image for building and auditing.

Canopy is definitely like Anaconda in that it's a distribution, but all the docs I found were about using it as a consumer, not as a distributor of an original application.

Anaconda has multiple tutorials for building packages (applicable for both library and applications), which can, like OS package managers, depend on Postgres and other non-Python dependencies. Unlike OS package managers, it's OS agnostic and has a tutorial for smoothly transitioning from a Python-native ecosystem (PyPI), to the bigger pond of cross-language.

Like, I'm tempted to include it in a more comprehensive OS-native distribution section, but conda really is a pretty unique case, especially taking into account its particular mutual affinity with Python community.

@pradyunsg
Copy link
Member

pradyunsg left a comment

Minor comments inline.

Packaging libraries and tools
-----------------------------

You may have heard about PyPI and ``setup.py`` and wheel files. These

This comment has been minimized.

@pradyunsg

pradyunsg Jun 4, 2018

Member

nit: two "and", first can become a comma.

Packaging Python
================

Python is a general-purpose programming lanaguage, meaning you can use

This comment has been minimized.

@pradyunsg

pradyunsg Jun 4, 2018

Member

typo: language

check out the following frameworks, and refer to their packaging
guides:

* Kivy

This comment has been minimized.

@pradyunsg

pradyunsg Jun 4, 2018

Member

Should we link to these project's pages?

This comment has been minimized.

@mahmoud

mahmoud Jun 5, 2018

Author Contributor

Definitely! I actually have quite a few projects I still haven't mentioned, but when I get a spare hour or two, I plan on going through and hyperlinking everything :)

@ncoghlan

This comment has been minimized.

Copy link
Member

ncoghlan commented Jun 4, 2018

Fair points regarding conda vs the other alternatives I mentioned. While it's HPC focused rather than general user environment focused, I'd say https://spack.io/ is the closest thing I'm aware to a comparable system. (They submitted an entry for the "Other projects" page a while back: https://packaging.python.org/key_projects/#spack )

@theacodes
Copy link
Member

theacodes left a comment

Some top-level comments. This is looking amazing! Let me know once you're ready for final editorial review and we'll get it merged. :)

@@ -0,0 +1,256 @@
================
Packaging Python

This comment has been minimized.

@theacodes

theacodes Jun 8, 2018

Member

Could you make the title something like "An overview of Python packaging"? It makes it better for linking and SEO.

it for many things. You can build robots or server software or a game
for your friends to play.

For this reason, the first step in every Python project must be to

This comment has been minimized.

@theacodes

theacodes Jun 8, 2018

Member

Nit: this paragraph should merge with the one above (just delete the newlines).

It might seem strange to think about packaging before writing code,
but this process does wonders for avoiding headaches later on.

* Who are your software's users? Are they other developers doing

This comment has been minimized.

@theacodes

theacodes Jun 8, 2018

Member

Introduce this list with something like:

"Some of the questions you may need to ask to determine your packaging strategy are:"

multiple files?). It's easier to get incomplete transfers, and harder
to guarantee code integrity at the destination.

So long as your code contains nothing but pure Python code, and you

This comment has been minimized.

@theacodes

theacodes Jun 8, 2018

Member

Hmm, this is a bit weird. Even packages with native components have source distributions, and conversely, pure-Python packages can still have built distributions (wheels).

This comment has been minimized.

@mahmoud

mahmoud Jun 14, 2018

Author Contributor

Absolutely. I've added a graf to the (very underdeveloped) binary distribution section about how binary is best served up with source available, too. Still, because this is an overview, I'm going to leave some of the complexity (e.g., pure-Python wheels) offloaded to some of the other pages. I think next week sometime I'll be able to trawl the full guide for all of the crossrefs :)

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:

  1. If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's --only-binary option, whereas even pure Python sdists will fail to pass that check.
  2. However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.

So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:

  1. Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
  2. Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover

This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)

Python binary distribution
^^^^^^^^^^^^^^^^^^^^^^^^^^

Python's real power comes from its ability to integrate with the

This comment has been minimized.

@theacodes

theacodes Jun 8, 2018

Member

"Python's real power" can read as dismissive. I'd recommend "A great feature of Python is its ability to..."

This comment has been minimized.

@mahmoud

mahmoud Jun 14, 2018

Author Contributor

Totally. This section is very underdeveloped, but I've adjusted this in my most recent commit..

@pradyunsg

This comment has been minimized.

Copy link
Member

pradyunsg commented Jul 15, 2018

This PR has been idle for a month. Is there anything I can do to help get this moving again; by contributing content or otherwise?

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jul 15, 2018

Sorry @pradyunsg I promise I haven't forgotten, just running a bit late with some life stuff. Was going to work on it this weekend, but wedding planning flared up again. Thanks for the check-in!

@pradyunsg

This comment has been minimized.

Copy link
Member

pradyunsg commented Jul 22, 2018

No issues @mahmoud. :)

Thanks for making the time to work on this again.

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jul 26, 2018

I daresay this PR is finally done! I only imported diagrams, not illustrations, and there are a couple ideas for sections commented, but otherwise the overview reads and looks great on my local. Added a link to the home page, and I think this is ready to merge. Let me know if there are any requested changes. Otherwise, very much looking forward to seeing this go live!

@theacodes

This comment has been minimized.

Copy link
Member

theacodes commented Jul 26, 2018

* `constructor <https://github.com/conda/constructor>`_ - For command-line installers
* `py2exe <http://www.py2exe.org/>`_ - Windows only
* `py2app <https://py2app.readthedocs.io/en/latest/>`_ - Mac only
* `bbFreeze <https://pypi.python.org/pypi/bbfreeze>`_ - Windows, Linux, Python 2 only

This comment has been minimized.

@gtback

gtback Jul 26, 2018

Contributor

Trivial thing: should be https://pypi.org/project/bbfreeze/ (though this URL will redirect correctly)😁

P.S. Very excited about having this guide!

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jul 27, 2018

@gtback, dangit, I got all of them but one! Good catch.

Fixed, with some additional improvements as well. No TODOs left in the doc :)


This Python packaging superpower makes it possible for Pillow to be
a drop-in replacement for PIL, just by changing your project's
`q`install_requires`` or ``requirements.txt``.

This comment has been minimized.

@pradyunsg

pradyunsg Jul 27, 2018

Member

There sees to be a typo here.

This comment has been minimized.

@mahmoud

mahmoud Jul 27, 2018

Author Contributor

Fixed, thanks :)

@theacodes

This comment has been minimized.

Copy link
Member

theacodes commented Jul 27, 2018

Preview at d366f79 staged here: http://temp.theadora.io/pypug/overview.html if anyone wants to look at it.

@theacodes
Copy link
Member

theacodes left a comment

This mostly looks great except for one comment. Thank you so much for doing this!

looking for ways to package Python for a non-technical audience and/or
a production setting, skip ahead to :ref:`packaging-applications`.

Python modules

This comment has been minimized.

@theacodes

theacodes Jul 27, 2018

Member

This section reads a bit bare and doesn't quite mesh with the context. What are we trying to convey to the user here? What action do we expect them to take with this information?

This comment has been minimized.

@mahmoud

mahmoud Jul 27, 2018

Author Contributor

So this section is under two pressures:

  1. Python developers from their first pip install are steeped in setup.py and "for-other-developers" packaging. I vividly remember the times I really struggled to make the mental leap from "I should build an egg" to "I should build an RPM" for my production application. It wasn't until much later that I realized the deployment model and even the audience were completely different. So this section is trying to place these technologies in the decision tree, as early and separate as possible from the applications. Without it, the immediate question becomes "well, what about wheels/sdists/pip/eggs/etc."?
  2. The section is on the bare side (even shorter than the sections in the initial post) because while I want to acknowledge the technologies, the rest of the site is a much better guide. I wanted it to be more content-light and link-heavy.

If you think it could stand do be expanded a bit more without overshadowing more extensive writings on the rest of the guide, I'd be happy to do what I can. Alternatively, as someone who knows the guide much better than me, I would really welcome more links to things that need linking!

This comment has been minimized.

@theacodes

theacodes Jul 27, 2018

Member

Yeah, I'm thinking we can either (1) remove it, or (2) expand on it just a little:

In the most simple of cases such as a single Python file that only depends on the Python standard library, you can simply copy and use the script however you'd like. This is great for sharing simple scripts between people who both have compatible Python versions (such as via email, StackOverflow, or GitHub gists). There are even some entire Python libraries that offer this as an option, such as Bottle. However, this pattern doesn't hold well for projects that consist of multiple files, need specific additional Python or OS libraries, or need a specific version of Python.

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

Absolutely! Added, with a link to bottle, and boltons, only because I specifically talk about this in the docs.

@ncoghlan
Copy link
Member

ncoghlan left a comment

This is great, thank you :)

I think there's really only one substantive point which we still need to resolve, which is the information on distributing pure Python packages to other developers: I think we want to actively promote the use of universal wheel archives for pure Python projects, but even mentioning that as a potential question worth asking yourself will require a bit of restructuring of that entire section.

Beyond that, my comments are mainly copy-editing nits and a few minor technical corrections :)

* Who are your software's users? Will your software be installed by
other developers doing software development, operations people in a
datacenter, or a less software-savvy group?
* Is your software intended to run on servers, desktops, or embedded

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

Slight tweak to specifically mention mobile devices:

  • Is your software intended to run on servers, desktops, mobile clients (phones, tablets, etc), or embedded ...

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

sure!

looking for ways to package Python for a non-technical audience and/or
a production setting, skip ahead to :ref:`packaging-applications`.

Python modules

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.

If your code consists of multiple Python files, it's usually organized
into a directory structure. Any directory containing Python files,
provided one of those files is named ``__init__.py``, comprises an
:term:`import package`.

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

Since the introduction of namespace packages, this qualifier isn't necessary - __init__.py just marks the distinction between a self-contained packaged and a native namespace package, rather than being a pre-requisite for making an import package at all.

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

I've removed the qualifier, but it's kind of tricky. Maybe I'm getting philosophical here, but a directory without an expression of intent like __init__.py doesn't necessarily become an import package until someone imports it. And that was more a choice of the importer than the publisher.

I think I'll always prefer to the __init__.py for explicit package creation to avoid this sort of quandary. :)

This comment has been minimized.

@ncoghlan

ncoghlan Jul 30, 2018

Member

Yeah, I default to __init__.py as well, and there are still plenty of tools that only handle explicit import packages by default.

Perhaps we could say something like:

Any directory containing Python files can optionally be accessed at runtime as an :term:import package, rather than as a collection of independent scripts (and if one of the files in the directory is named __init__.py, it makes it explicit that the submodules are intended to be accessed via the package rather than directly).

multiple files?). It's easier to get incomplete transfers, and harder
to guarantee code integrity at the destination.

So long as your code contains nothing but pure Python code, and you

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:

  1. If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's --only-binary option, whereas even pure Python sdists will fail to pass that check.
  2. However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.

So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:

  1. Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
  2. Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover

This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)

a drop-in replacement for PIL, just by changing your project's
``install_requires`` or ``requirements.txt``.

Python binary distributions

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

Continuing the above discussion, I'm wondering if it may make sense to add a third subsection after this one: "Publishing binary distributions for pure Python projects".

Then the paragraph above about specific example external dependencies can move down into the second subsection (since we'll want folks to keep reading regardless of whether they have external dependencies or not), and the final subsection can adopt the perspective of "Even if you're publishing a pure Python sdist, you may also want to publish a wheel archive anyway, for these reasons..."

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

Yeah, I also want to recommend universal wheels for pure-Python packages. That said, in trying to "ramp" users based on their needs, I also don't want to overwhelm them. I've put a couple sentences in to tie the two together as best practice, but more technical discussions should probably be on the linked article (tutorials/packaging-projects.rst). Maybe an aside to explicitly recommend doing both could work?

This comment has been minimized.

@ncoghlan

ncoghlan Jul 30, 2018

Member

Yeah, it's a tricky balance. I think an aside probably isn't strong enough for this point, but I also don't think we really want to include in either of the existing subsections - hence the idea of make it a very short subsection in its own right.

This comment has been minimized.

@mahmoud

mahmoud Jul 30, 2018

Author Contributor

Hmm, I guess my main concern there is that right now the document is organized by "class of distributable artifact, and what it is capable of distributing". If I'm understanding correctly, this would add a section about wheels, which may be a confusing redundancy.

Another point is that sdist + wheel is absolutely a best practice on PyPI, but not necessarily so in other packaging workflows. I've used these formats "off PyPI" and always sdist only or wheel only. I'm not sure having both would have done much in those other packaging approaches. So in the interest of maintaining the introductory tone, I think it helps to introduce the formats and their intended purposes in this overview, then the best practices in the practical docs. (If someone manages to figure out how to push to PyPI from the overview alone, I'll be very surprised!)

This comment has been minimized.

@ncoghlan

ncoghlan Aug 4, 2018

Member

I don't see "class of distributable artifact" as the main organising principle in the current overview - I see it as being primarily organised by target audience, and then working back from that to the appropriate artifacts to use.

And when the target audience is "other Python developers", the appropriate artifacts to publish are both an sdist and a wheel archive.

The only time it makes sense to choose one or the other exclusively is when the target audience is "next stage in a specific automated build pipeline", and I don't think that's a scenario the overview really needs to emphasise.

At most, that use case feels like a potential side note along the lines of "Default to publishing both sdist and wheel archives, unless you're creating artifacts for a very specific use case where you know the recipient only needs one or the other".

This comment has been minimized.

@mahmoud

mahmoud Aug 11, 2018

Author Contributor

I liked that last line, incorporated it in one of the recent commits. :)

I have to say, when I wrote it, my target audience was squarely "intermediate developers", i.e., devs experienced enough to get something to a shippable state who are also smart enough to search out some prior art. I'm willing to bet most of them are already familiar with several formats at this point, just from package managers, app stores, pip, and so forth. So technologies became the organizing principle because it's about creating a taxonomy to avoid getting lost in a sea of apples and oranges.

I'd love to think that some folks might consult the guide before writing any code destined for production, but I'll still put my money on this not being their first rodeo. In fact, many are probably looking for ways to create a rational build pipeline, such as a wheel into a container, which is why I think this arrangement works better than a "wizard" approach of segmenting by application/developer type.

Anyways, I wish I had more weekends like this to go on, but I'm gettin married in under a month, so I think this is all I can muster for now. I hope it can do some good for now.


.. _depending-on-a-new-python:

Depending on a new Python ecosystem

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

It would be good to adjust the title here, as "new" is ambiguous: it could refer to "new-to-the-target-environment" or "new-to-the-world-at-large" (and neither is entirely accurate for the scope of the section anyway).

My suggestion would be "Depending on another software distribution ecosystem". That way if we later decide to add other options like Nix or homebrew in here, they can just be an extra paragraph at the end of this section, rather than needing their own section.

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

(working from bottom to top has its disadvantages, I see what you mean from down below)

would require only electricity.

Whereas the virtual machines described above are primarily reserved
for the tech-savvy, hardware appliances used by the most advanced data

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

Typo: missing "are" before "used"

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

fixed!

^^^^^^^^^^^^^^^^^^^^^^^^^

As mentioned in :ref:`depending-on-a-new-python` above, some operating
have package managers of their own. If you're very sure of the

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

Typo: missing "systems" in "operating systems"

This cross-ref will also need to change if that section title changes.

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

good catch! wrt the crossref, I think I made an explicit heading anchor ref thing in this case.

Debian, Ubuntu, etc.) or `RPM
<https://en.wikipedia.org/wiki/RPM_Package_Manager>`_ (for Red Hat,
Fedora, etc.), and use that built-in package manager to take care of
installation, and even deployment.

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

It would be good to directly mention FPM here, since that can take a fair bit of pain out of converting Python projects into Linux system packages: https://fpm.readthedocs.io/en/latest/source/virtualenv.html

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

Added

ride. This impression is mostly a byproduct of Python's
versatility. Once you understand the natural boundaries between each
packaging solution, you begin to realize that the varied landscape is
a small price Python programmers pay for using the most balanced,

This comment has been minimized.

@ncoghlan

ncoghlan Jul 28, 2018

Member

I'd tweak the praise of the language in this last sentence to be something unequivocally true. Something like:

... for using one of the most balanced, flexible, and broadly applicable languages available.

(The "one of" does the equivocation work there: "most" is debatable, "one of the most" is not)

This comment has been minimized.

@mahmoud

mahmoud Jul 29, 2018

Author Contributor

Heh, I'd take that debate anyday! But sure, not trying to make any enemies of anyone who read to the bottom of the page :)

mahmoud added some commits Jul 29, 2018

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Jul 29, 2018

OK, made some changes which I think mostly address the concerns raised.

@ncoghlan, regarding your big point, I'm OK expanding the packaging-libraries section somewhat, I just want to avoid having a big technical bump in the middle covering a topic that's really well-covered by links to the rest of the guide. Take a look at the sentences added in 8d3aae2, and if those aren't enough, would you mind writing a short aside (something I can put in a .. note:) as mentioned in the comment above?


.. _depending-on-a-new-ecosystem:

Depending on a new software distribution ecosystem

This comment has been minimized.

@ncoghlan

ncoghlan Jul 30, 2018

Member

"new" is still a bit ambiguous here ("new" in general? "new" to you? "new" to your users?).

Perhaps "a new" -> "another"?

This comment has been minimized.

@mahmoud

mahmoud Jul 30, 2018

Author Contributor

Oops, meant to fix that one before, fixed now!

@mahmoud

This comment has been minimized.

Copy link
Contributor Author

mahmoud commented Aug 11, 2018

Just finished adding a few more tweaks based on feedback, and I think in many ways this document is actually a lot more comprehensive and broadly-targeted than the original post, in all the best ways. I think it's safe to say at this point I think it is/I am done :)

For the moment, I've got a (my) wedding to attend to, so I don't think I'll have much more time to work on it. I think even for any of its remaining, arguable flaws, having the overview available on the primary site would help a lot more than it would hurt, so I would encourage publishing it in its current state and tweaking it further in the form of future PRs. Thanks again for all the feedback, everyone!

@pradyunsg
Copy link
Member

pradyunsg left a comment

This looks good to me. I agree that this should be good enough to publish as is; we can make any additional changes needed in the near future.

Thanks for working on this @mahmoud! ^.^

@theacodes

This comment has been minimized.

Copy link
Member

theacodes commented Aug 12, 2018

LGTM, too! @ncoghlan if you have any further concerns please let me know (or file bugs) and I'll be happy to fix them in follow-up PRs.

@mahmoud this is monumental work, thank you so much!

We can address any remaining issues in follow-ups.

@theacodes theacodes merged commit 4b5e860 into pypa:master Aug 12, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ncoghlan

This comment has been minimized.

Copy link
Member

ncoghlan commented Aug 13, 2018

8d3aae2 and 88a1bc9 cover my wheel vs sdist concerns without requiring major changes to that section, so +1 for that approach :)

Thanks again @mahmoud for this incredibly useful addition!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment