Priorities within CPython development for CZI proposal? #26
Comments
|
I'm just a lurker now, but why biomedical research? And some ideas for the brainstorm: In Python 3.10 we'll be able to use the full power of the PEG parser. But 3rd party tooling like Black doesn't have a PEG parser yet. We could work on a 3rd party PEG parser suitable to replace lib2to3. Improve mypy -- it may be losing much of its corporate funding, and we need to upgrade the type system to support numpy. More directly CPython: make it work properly on mobile and ensure that it keeps working there. ( And in web browsers for that matter.) Burn down the list of open PRs and issues, and design a new workflow to prevent them from getting so clogged again. (Is the bpo -> GitHub transition funded yet?) Modernize the buildbot farm (yes, we need OS diversity, but many of the hosts are ancient and too slow or memory starved to be useful). Maybe some mobile 'bots, too. Fund Serhiy for a year to make improvements across the board. Accelerate work on HPy. Incorporate several features from trio into asyncio (nurseries, multi-exceptions), and other asyncio work (better ways of connecting asyncio and threads?). Improve the docs (maybe migrate to markdown???). |
|
On 6/10/2020 10:39 PM, Guido van Rossum wrote:
More directly CPython: make it work properly on mobile and ensure that
it keeps working there. ( And in web browsers for that matter.)
+1. This is my favorite.
|
|
The reason that relevance to biomedical research is important for this particular question, right now is because the August 4th deadline is to ask for money from Chan Zuckerberg Initiative's Essential Open Source Software for Science program which "invites applications for open source software projects that are essential to biomedical research". The previous requests for applications explain:
The more persuasively that our grant application can say "here is how this will help the biomedical researchers who use Python," the more likely we are to get money from this program. I remember @freakboy3742's Language Summit presentation on Python on mobile this year -- Russell, if you can speak up with a paragraph about why this is particularly interesting from a biomedical research perspective, I'd love that! |
|
I'd vote strongly for this:
Something similar to the Django fellow, funded for a year as a test would be huge. We only then need to sell the larger story of Python being impactful to biomedical research, which is obviously true. I know there are other groups that have applied for this grant for this type of funding in the past, so I can definitely help share that knowledge from the Django & historical CZI grant side. 👍 |
Yes, it's funded. We are going through job applicants for the PM position now and we have an initial list of candidates to join the WG to manage the transition. |
I'll put in votes for these. |
|
And what are your own brainstorm ideas? |
|
Here's an attempt at a pitch for why "mobile python" is of interest to science/biomed Python is already a well established tool in science and biomedical research, where it is used for data analysis, visualization, machine learning and pattern recognition. Scientists and researchers also use Python to develop and maintain complex database-backed websites in support of their research goals. However, this usage is largely restricted to laptops and servers. The most widely used types of computing device - phones and tablets - are not currently well served by the Python ecosystem. At present, developing apps for phones and tablets generally requires specialist skills, and often requires mastering multiple programming languages. As a result, developing a mobile application to support their research isn't an option available to most researchers. A "mobile enabled" Python would enable scientists and researchers to leverage their existing programming skills to develop mobile applications. These applications could simply make existing data analysis and visualization techniques available on new platforms; or they could combine these techniques with the unique capabilities of mobile devices, such as photo capture, geolocation, and augmented reality. This would open up new opportunities for in-the-field data gathering and analysis that hasn't been previously possible. If I was to pitch potential projects for the grant with a mobile focus:
|
|
On Sat, Jun 13, 2020 at 9:33 PM Russell Keith-Magee < ***@***.***> wrote:
@brainwane <https://github.com/brainwane>
Here's an attempt at a pitch for why "mobile python" is of interest to
science/biomed
------------------------------
Python is already a well established tool in science and biomedical
research, where it is used for data analysis, visualization, machine
learning and pattern recognition. Scientists and researchers also use
Python to develop and maintain complex database-backed websites in support
of their research goals. However, this usage is largely restricted to
laptops and servers. The most widely used types of computing device -
phones and tablets - are not currently well served by the Python ecosystem.
At present, developing apps for phones and tablets generally requires
specialist skills, and often requires mastering multiple programming
languages. As a result, developing a mobile application to support their
research isn't an option available to most researchers. A "mobile enabled"
Python would enable scientists and researchers to leverage their existing
programming skills to develop mobile applications. These applications could
simply make existing data analysis and visualization techniques available
on new platforms; or they could combine these techniques with the unique
capabilities of mobile devices, such as photo capture, geolocation, and
augmented reality. This would open up new opportunities for in-the-field
data gathering and analysis that hasn't been previously possible.
------------------------------
If I was to pitch potential projects for the grant with a mobile focus:
- Officially integrating iOS and Android into the CPython build
- Developing iOS and Android wheel formats that are compatible with
mobile app distribution.
- Developing a "Minimal Viable Python" (a minimal Python install, with
standard library pieces being opt-in)
Want to use Python to create Android and iOS applications? Fund BeeWare
<https://beeware.org/>. They already enable this!
Mobile environments are non-traditional, locked down with foreign
language-runtime API surfaces that generally frown upon C and POSIX (for
good reasons). Outside the experience of most CPython core devs and not an
environment we'd collectively be easily able to support (meaning solving
that for the long term would need to be part of the project). I
don't believe there is even any need for pip and wheels there. That
distribution model is effectively forbidden by the platform. Any
application that wants to use Python *must* embed its own interpreter and
fully control its environment. I don't expect there to ever be a "install
this CPython app, download arbitrary code from the internet, pip install
wheels, edit+run code on a phone" story for CPython. It'd suck. No
application user would ever choose that so no-one trying to provide a
mobile app for their users would ever force that 1-star review experience
upon them.
Any goals around mobile support on the CPython side would make a lot more
sense if BeeWare's pain points were used to drive prioritized requirements.
…-G
|
|
(and yes i realize my reply is going to... someone from the beeware project
making those suggestions :)
…On Sun, Jun 14, 2020 at 2:48 PM Gregory P. Smith ***@***.***> wrote:
On Sat, Jun 13, 2020 at 9:33 PM Russell Keith-Magee <
***@***.***> wrote:
> @brainwane <https://github.com/brainwane>
>
> Here's an attempt at a pitch for why "mobile python" is of interest to
> science/biomed
> ------------------------------
>
> Python is already a well established tool in science and biomedical
> research, where it is used for data analysis, visualization, machine
> learning and pattern recognition. Scientists and researchers also use
> Python to develop and maintain complex database-backed websites in support
> of their research goals. However, this usage is largely restricted to
> laptops and servers. The most widely used types of computing device -
> phones and tablets - are not currently well served by the Python ecosystem.
>
> At present, developing apps for phones and tablets generally requires
> specialist skills, and often requires mastering multiple programming
> languages. As a result, developing a mobile application to support their
> research isn't an option available to most researchers. A "mobile enabled"
> Python would enable scientists and researchers to leverage their existing
> programming skills to develop mobile applications. These applications could
> simply make existing data analysis and visualization techniques available
> on new platforms; or they could combine these techniques with the unique
> capabilities of mobile devices, such as photo capture, geolocation, and
> augmented reality. This would open up new opportunities for in-the-field
> data gathering and analysis that hasn't been previously possible.
> ------------------------------
>
> If I was to pitch potential projects for the grant with a mobile focus:
>
> - Officially integrating iOS and Android into the CPython build
> - Developing iOS and Android wheel formats that are compatible with
> mobile app distribution.
> - Developing a "Minimal Viable Python" (a minimal Python install,
> with standard library pieces being opt-in)
>
> Want to use Python to create Android and iOS applications? Fund BeeWare
<https://beeware.org/>. They already enable this!
Mobile environments are non-traditional, locked down with foreign
language-runtime API surfaces that generally frown upon C and POSIX (for
good reasons). Outside the experience of most CPython core devs and not an
environment we'd collectively be easily able to support (meaning solving
that for the long term would need to be part of the project). I
don't believe there is even any need for pip and wheels there. That
distribution model is effectively forbidden by the platform. Any
application that wants to use Python *must* embed its own interpreter and
fully control its environment. I don't expect there to ever be a "install
this CPython app, download arbitrary code from the internet, pip install
wheels, edit+run code on a phone" story for CPython. It'd suck. No
application user would ever choose that so no-one trying to provide a
mobile app for their users would ever force that 1-star review experience
upon them.
Any goals around mobile support on the CPython side would make a lot more
sense if BeeWare's pain points were used to drive prioritized requirements.
-G
|
|
I know absolutely nothing about the complexities of developing for mobile, but FWIW I can freely pip install in the Python that comes with Termux, for Android. Not everything works, but everything I've tried will install. Further, as long as I So, maybe this is irrelevant to the current conversation, but superficially it seems to argue against @gpshead's position. |
|
@gpshead I completely agree that the workflow for using Python on a mobile device would be different. However, that doesn't mean that wheels aren't possible, or wouldn't be useful. If anything, they're more useful on mobile platforms because of the way mobile apps need to operate. Going into specifics will likely rathole this entire discussion. I'm happy to elaborate; let me know where the better place would be. Suffice to say that the three points I listed are the pain points that BeeWare has with CPython at present. At the core of all of them is elevating "interpreter embedded in an app sandbox" as a first-class distribution story for Python - i.e., a Python installation story that doesn't include or involve |
|
On Sun, Jun 14, 2020 at 3:53 PM Russell Keith-Magee < ***@***.***> wrote:
@gpshead <https://github.com/gpshead> I completely agree that the
workflow for using Python on a mobile device would be different. However,
that doesn't mean that wheels aren't possible, or wouldn't be useful. If
anything, they're *more* useful on mobile platforms because of the way
mobile apps need to operate.
Going into specifics will likely rathole this entire discussion. I'm happy
to elaborate; let me know where the better place would be.
Suffice to say that the three points I listed *are* the pain points that
BeeWare has with CPython at present. At the core of all of them is
elevating "interpreter embedded in an app sandbox" as a first-class
distribution story for Python - i.e., a Python installation story that
*doesn't* include or involve python.exe. And while that story is the
*only* way Python works on mobile, it's also a useful story for
standalone app distribution on desktop platforms.
+1 anything that makes a first class distribution story out of embedding
more usable without the current pains involved in setting up and
maintaining that kind of thing would indeed be great. I expect there are
multiple intersecting stories of exact needs based on what various
platforms/environments expect or already provide or don't provide.
…-gps
|
|
My guess was that wheels for mobile platforms would be useful to app
developers to assemble the bundle for their application, not for end users.
|
|
@gvanrossum Yes, that's the intended use case. End-user wheel installs may not even be possible. I believe Apple's App Store guidelines would reject an app that allowed the installation of wheels (and especially binary wheels) after the app has gone through review. |
|
Ideas:
|
|
Quick update: we discussed potential ideas to submit proposals for. We want to wait until after our next meeting next week when everyone is able to attend before we publicly state what we think the priorities should be. Feel free to continue to discuss things until then. |
|
Thanks @brettcannon! Project Funding WG has a work-in-progress list of funders. So if core developers and the Steering Council come up with some great ideas that aren't well suited to a CZI application, it's worth collecting those because some might be well-suited to a Mozilla Open Source Support Award application, or a Comcast Innovation Fund application, etc. |
|
Also, @brettcannon, I presume that once the Steering Council publishes the Vision Deck that was mentioned in a past update, that would be helpful for this discussion and related funding-seeking discussions. Will we be seeing that soon? |
|
The vision deck was scrapped for plans to present at PyCon US on the topic, which then got dashed due to funding concerns for the PSF. I suspect, though, the list we present back to you all will encompass what we would have put in that document and presentation. |
|
I consider that the performance of the Python runtime matters to ensure that Python will remain relevant in 5 or 10 years. I identified three projects which are realistic:
|
|
@brettcannon wrote on June 15th:
Thanks! Should we expect the Steering Council's public statement of priorities soon? In order to submit a good proposal by August 4th, I figure we need either of these 2:
(I'm saying "thing" instead of "project" because of ideas like "hire a person for a year to do general code review/issue wrangling" which is a fundable thing but not a project.) |
|
Unfortunately we weren't able to get to this topic today as something more pressing came up and took up the whole meeting. I'll start an email thread to see what we can pull together. And I am going to take from that last comment, @brainwane , that you are after a short list by July 7. |
|
@brettcannon thanks for the update.
Yes, I think that would be great, thanks. In case this helps: I have faith that, whatever the topic area is, as long as CPython maintainers want it and it's applicable to biomedical research, we can find a way to scope work and make it feasible and plausible as a proposal. |
|
@brettcannon should we expect a short list early this week? Thanks! |
|
Just finished our meeting and the two things we would suggest proposing are:
Let us know if you need any more clarifications on those. |
|
Thanks @brettcannon -- very helpful! We now have three weeks to try to find proposal-writers, write the proposal, edit, get Board approval, and submit. I hope that the Project Funding Working Group's members can do a lot of the lifting on writing this proposal, but that's not certain. I'm going to close this issue now because the Steering Council has set its priorities for this, but anyone who's interested in writing even a few paragraphs about why this is important and how much work it would take, please reply to this issue to volunteer. |
Is something like pants pex in the same vein as shiv? Looks like pex has the ability to generate python executables, but not sure if these executables fit the "first-class distribution story" described above. |
|
shiv is a modernization of pex. At my job, we were using pex for tarball distributions (not really single file executables, see below), but pex had lots of performance problems, mostly due (IMHO) to its backward compatibility requirements. shiv dropped all that, supporting only Python 3 and using modern techniques and libraries (e.g. importlib.resources instead of pkg_resources) to get good performance. As nice as shiv is, I wouldn't classify it as a "single file executable". Both pex and shiv are fundamentally tarballs with a special shebang line that Python knows how to execute, but 1) you still have to have the Python binaries installed out of band; 2) you still have to unpack the tarball to be able to import extension module shared libraries (since I want something like what PyOxidizer does, where you don't have to install anything else out-of-band or otherwise, and you don't have to unpack anything the first time you run it. It's just an executable that you can ship around and users wouldn't even have to know it was written in Python! |
|
Sad news fam, our proposal was not accepted :( |
TL;DR: what are a few priority areas for CPython development where one full-time person or a few part-time people, working for one year, could make a big difference? Let's decide by July 7th and apply for CZI money.
Chan Zuckerberg Initiative's Essential Open Source for Science grant is going to open their next funding cycle to applications soon -- June 16th to August 4th, per the request for applications.
The PSF can apply for $50k-$250k USD for a 1-year project. The PSF has successfully gotten one of these grants already: $200,000 to improve the user experience and debuggability of pip (more details), and we've been welcomed to apply again.
I think the PSF should consider proposing at least one project that improves CPython. In my experience, projects need at least a few weeks to put together a good application, sometimes a month. So I would love to use this issue to generate some ideas, and then, by July 7th, have a consensus, aided by the Steering Council, on what to pursue. Then the new Project Funding Working Group (currently being formed) can help with advising proposal-writers and helping them get Board and/or Ewa's approval, and help get the proposal submitted before August 4th.
I suggest a few criteria:
My three suggestions to kick off discussion are:
(Budget estimate: If we ask for and get $250K, and the PSF takes 15% overhead, we get $212,500. If we assume that PSF hires contractors for this work at an hourly pay rate at USD$115-$200 per hour, that's enough for about 1000 to 1800 hours of work. CZI funds one-year projects, so, at that pay rate, this ends up being the range between one half-time person and one full-time person.)
The text was updated successfully, but these errors were encountered: