Release Windows Store app containing Python

[zooba]

BPO	34977
Nosy	@pfmoore, @vsajip, @vstinner, @tjguk, @jkloth, @ned-deily, @zware, @zooba, @ethanhs, @miss-islington, @tirkarthi
PRs	bpo-34977: Add Windows App Store package #10245 [3.7] bpo-34977: Add Windows App Store package (GH-10245) #10677 Revert "bpo-34977: Add Windows App Store package (GH-10245)" #11019 [3.7] Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" #11021 bpo-34977: Add Windows App Store package #11027 [3.7] bpo-34977: Add Windows App Store package (GH-11027) #11028 bpo-34977: Use venv redirector instead of original python.exe on Windows #11029 [3.7] bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029) #11030 bpo-34977: Remove unused preprocessor definition #11092 [3.7] bpo-34977: Remove unused preprocessor definition (GH-11092) #11093 bpo-32907: pathlib: Fix test_resolve_common failure on Windows #5812

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2018-12-12.23:45:02.538>
created_at = <Date 2018-10-13.22:43:47.118>
labels = ['3.8', 'type-feature', '3.7', 'OS-windows']
title = 'Release Windows Store app containing Python'
updated_at = <Date 2019-10-18.00:21:36.712>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2019-10-18.00:21:36.712>
actor = 'izbyshev'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2018-12-12.23:45:02.538>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2018-10-13.22:43:47.118>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 34977
keywords = ['patch']
message_count = 42.0
messages = ['327679', '327691', '328146', '328147', '330166', '330359', '330452', '330643', '331265', '331266', '331267', '331300', '331301', '331304', '331310', '331311', '331316', '331321', '331322', '331328', '331350', '331352', '331356', '331357', '331358', '331363', '331368', '331377', '331387', '331517', '331518', '331519', '331520', '331521', '331530', '331534', '331553', '331560', '331566', '331571', '331572', '331573']
nosy_count = 11.0
nosy_names = ['paul.moore', 'vinay.sajip', 'vstinner', 'tim.golden', 'jkloth', 'ned.deily', 'zach.ware', 'steve.dower', 'ethan smith', 'miss-islington', 'xtreak']
pr_nums = ['10245', '10677', '11019', '11021', '11027', '11028', '11029', '11030', '11092', '11093', '5812']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34977'
versions = ['Python 3.7', 'Python 3.8']

[zooba]

The Windows Store can now install unrestricted apps, which means we can provide the Python interpreter in there.

Advantages:

• far more reliable installation
• can have specific executables on PATH
• automatic updates
• other apps can act as extensions (with permission), so things like Mu could also become store apps in the future

Disadvantages:

• only --user installs for pip will work
• harder to find/modify installed files

I think for a (very significant) subset of our users, this will be a much better experience than downloading our current installer. It also has the advantage of making "Python" appear in searches in the start menu, which will link directly to a one-click "Install" button.

The biggest issue is likely to be pip not installing with --user by default, as well as subsequent issues with apps that require users to navigate to their %AppData% directories manually (since these will be redirected to different locations).

But until we get a package available for testing, it will be hard to figure out what needs fixing. Given this is just distribution and not a new platform, I plan to enable it for Python 3.7 (probably 3.7.2, but it may be possible to bundle 3.7.1 as-is for testing).

[tirkarthi]

+1 for reliable installation. There were some cases (bpo-33220, bpo-30944) where the installer was detected as a Virus by some vendors which would be reduced since this is installed from the App store as a verified app. I hope this will also reduce the scenarios where users need to resort to bug tracker with log files when there are installation failures [1] providing a more smooth one click install.

There is a long standing open issue [2] to make pip install --user as a default which I hope gets resolved and there is an issue with using pip install --user inside a virtualenv which seems to block the earlier issue [3]

Thanks a lot for this!

[1] https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=windows+msi+install&submit=search&status=-1%2C1%2C2%2C3

[2] pypa/pip#1668

[3] pypa/pip#5702

[zooba]

I'm 99% sure that with bpo-34725 I'll be able to make "pip --user" the default when using this install, as well as making the commands globally available (i.e. "python[3[.8]]", "pip[3[.8]]", "idle[3[.8]]" will work everywhere).

[zooba]

(And I'll deal with that 1% at the PyPA sprints next weekend. We are due for a new pip insertion anyway, so I guess that'll come for 3.7.2.)

[zooba]

For people watching the bug, this is ready for review.

Most of the change is adding a "layout" script (roughly the equivalent of "make install" for Windows, but with more flexibility to generate different structures, precompile stuff, etc.)

About the only CPython change is to venv. I added a new launcher variation that looks for pyvenv.cfg and runs the python.exe listed in that, so now we don't have to copy any DLLs into the venv at all (it's needed because the old one would try to LoadLibrary DLLs inside the store package, which is not allowed, but it also fixes my #1 complaint about venv which is that upgrading the base Python breaks all venvs).

There are also a few tweaks to the startup process, including extending the existing __PYVENV_LAUNCHER__ env variable from macOS to Windows as well, though it's handled in a different location. And a couple of minor tweaks to tests - issues discovered now that it's pretty easy to generate a "real" layout to run tests in.

I can break this up into a couple of PRs if that's preferred, though it doesn't really gain anything. If you're not interested in the new layout script, stop reading the PR after you finish launcher.c :)

[zooba]

If anyone would like to try this out early, here's some instructions for a build I just made.

Enable sideloading apps on your machine (see https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development - may already be done, sideload or developer mode should do)

In a regular (non-admin) PowerShell prompt in a temporary directory, run these commands:

iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34540/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
Expand-Archive .\python37.zip .
Add-AppxPackage appx\python.msix

(The first may take a while as it downloads approx. 80MB. I'm intending to leave the test suite out of the final package, which should bring it down closer to 40MB.)

Now play with it:
$> python.exe
$> python3.exe
$> python3.7.exe
$> pip.exe
$> pip3.exe
$> pip3.7.exe
$> idle.exe
$> idle3.exe
$> idle3.7.exe
(Or press Win+S and search for new entries)

To remove it:

Remove-AppxPackage (Get-AppxPackage *Python.3.7)
(delete the downloaded files)

Right now, the py.exe launcher can't find it, nor can any tools that rely on enumerating the registry - directly requesting the reg keys will return the right values though. VS Code seems to pick it up just fine though, I guess because it's on PATH.

[zooba]

For my own reference, some issues I found installing this package on a clean machine:

• requires Developer Mode (until I get the sccd file signed - just sideloading is not sufficient)
• requires vcruntime140.dll to be included in the package (workaround is to install the redistributable from https://visualstudio.microsoft.com/downloads/#title-39324)
• venv needs to treat .pdb files as binary

[zooba]

Updated testing instructions - this one should work with just sideloading permissions I think.

iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34760/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
Expand-Archive .\python37.zip .
Add-AppxPackage appx\python.msix

To remove:
Remove-AppxPackage (Get-AppxPackage *Python.3.7)
(delete the downloaded files)

[zooba]

New changeset 468a15a by Steve Dower in branch 'master':
bpo-34977: Add Windows App Store package (GH-10245)
468a15a

[zooba]

New changeset 2532091 by Steve Dower in branch '3.7':
[3.7] bpo-34977: Add Windows App Store package (GH-10245)
2532091

[zooba]

Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build. As soon as we snap for the RC I'll kick off an update and make the store page public, and hopefully can promote it enough to get eyes on it.

Unfortunately, I discovered as part of a test submission that the minimum Windows version matters, and it's a version that hasn't been rolled out fully yet because of some bugs, so there may not be that many people who can use it this first time. But that will improve over time, and I'm sure I can find enough people to flush out issues before the next release (anyone on the Windows Insiders program should be fine).

[vstinner]

Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build.

I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437. It doesn't seem like a good idea to push such change between a release candidate and a final version :-(

[vstinner]

New changeset cb0b78a by Victor Stinner in branch 'master':
Revert "bpo-34977: Add Windows App Store package (GH-10245)" (GH-11019)
cb0b78a

[vstinner]

I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437.

I reverted the change in master and I will merge PR 11021 once tests pass to revert in 3.7 as well.

It doesn't seem like a good idea to push such change between a release candidate and a final version :-(

I started a discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2018-December/155932.html

[vstinner]

New changeset 783b794 by Victor Stinner in branch '3.7':
Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)
783b794

[vstinner]

Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)

That should repair buildbots, so I remove "release blocker" priority.

[zooba]

We haven't done RC yet - why is pushing this now so bad? Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.

[vstinner]

We haven't done RC yet - why is pushing this now so bad?

I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-) By the way, as I wrote in my email, I would prefer to see the venv+tests change pushed in separated change (and explain why you do these changes).

I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.

Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

Do you mean that buildbots must upgrade to VS 2017? Can Python 3.6 and 3.7 be compiled without any issue on VS 2017? Maybe send an email to buildbot-status if you want to see more buildbots running VS 2017 rather than VS 2015.

--

There was a small bug in your PR, in PC/launcher.c, fixed by commit 4c49da0 (bpo-35436):

+ if (executable == NULL) {
+ error(RC_NO_MEMORY, L"A memory allocation failed");
+ }

[vstinner]

Since the first attempt broke all Windows buildbot workers, it would be nice to use a custom build before merging a new attempt :-) See instructions in the devguide:
https://devguide.python.org/buildbots/?highlight=custom#custom-builders

Note: Pablo used buildbot-custom to test a revert of the App Store change.

[zooba]

> We haven't done RC yet - why is pushing this now so bad?
I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours

I don't dispute that policy, but you said it shouldn't go in between RC and a release, which it isn't.

nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-)

More nitpicking: the venv changes only affect Windows. Which makes it purely Windows and tests, exactly as I said. Adding smiles doesn't save you from having to read :)

Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

Sure. You can start by reading this thread, but the summary is that App Store apps (regardless of platform) generally have to go through an approval process before they can be installed. Without going through this process, they can't be tested fully in context. I believe we should make a true release and put that through the process, rather than me just doing one out of my branch.

I provided instructions for developer testing above, which covers about 80% of what needs to be checked. The rest can only be tested by real use of a proper release.

[terryjreedy]

Various questions and comments:

0. Steve: For more than one reason, I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org. Can you assure us that the latter are not going away? I would have the same concern about adding something to the Apple/Mac store.

1. Ned: Since enhancements are not normally backported, it would be nice to have your approval, in whatever form it took, recorded here and not just in private email. I give a reason below.

2. Installers are always experimental until tested by multiple users on multiple machines. Even then, installation specific issues can arise. As I remember, the current installer, when new in 3.5, needed a shakedown period. One installer-specific issue was the initial working directory. To have a Windows Store Python ready for 3.8, we need to start now.

There was once an issue where one of the installers failed to include one idlelib file. I believe it was help.html. I since added idle_test/test_help.py, which would now fail with something like "No such file or directory: 'C:\\Programs\\Python37\\Lib\\idlelib\\help.html'".)
Repository testing does not test the actual binaries that users install, in their production context.

Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

3. Anyone: Though it is not essential that I know, I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there. My search only came up with age restrictions, which python does not need (in my opinion).

4. While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces. They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear). I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.

[zooba]

Thanks Terry, great questions.

I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org. Can you assure us that the latter are not going away?

Absolutely. The current installer is not going anywhere, and there's no point considering the possibility until we've completely dropped support for Windows 10 1803 and earlier (which was only released this year, so dropping support is a long way out yet).

Installers are always experimental until tested by multiple users on multiple machines. Even then, installation specific issues can arise. ... To have a Windows Store Python ready for 3.8, we need to start now.

Thanks for the vote of support for this point.

Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

Also, I've been testing IDLE regularly, since one of the advantages of being in the Store is that we can safely put idle[3[.7]].exe on PATH. It definitely works :)

I've also set up tests that run from an "installed" layout, rather than the source tree, and those are all passing as well (with one of the trivial test fixes in the PR).

I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there.

In general, all app stores disallow their apps from doing things that may break or corrupt the machines they are installed on. Apps are self-contained and can only operate outside of their boundaries through carefully designed permissions/capabilities (which often prompt the user before continuing).

Recently, Microsoft added support for "full trust" apps, which can bypass many of these restrictions. The aim is to allow traditional applications to swap their MSIs for being installed through the store, which can then ensure trustworthiness of at least the installers. (For example, a Store app is *never* going to install something into System32 or modify operating system files, whether it's full trust or not.) Full trust allows us to do things like launching non-app store processes and creating raw sockets through legacy APIs, so we can very easily move Python into a full trust package rather than having to rewrite it using approved APIs (a project that's been attempted a few times, but always loses low-level functionality that is considered a core part of Python - hence my long-term desire to make the low-level parts optional and have higher-level abstractions be the guaranteed parts).

While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces. They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear). I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.

I agree that it's valuable and I've done it internally plenty of times. Unfortunately, no matter how much people repeatedly say that a PR for CPython is part of a chain, there always seems to be pushback from people who ignore that and won't approve a critical prerequisite because there's "no need".

There's also the point that many of the changes were not known to be needed before the others were made. The pre-squash history is very interleaved between the various changes, and extracting them into separate PRs that totally stand alone is nearly as much work as solving the issues in the first place. I made the call this time to keep them all together and direct reviewers' attention towards the relevant parts. It turns out that didn't attract reviewers, so I'll come up with a new strategy next time I want to get people's attention on a change (maybe Buzzfeed style headlines? :) )

[pfmoore]

> Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

I'd like to expand on this question a little. The first thing I'll
note is that I'm not aware of any documentation on the Windows Store
installer, and how it works. It's possible there's a docs patch in
here somewhere, but to be honest, there are 8 PRs linked to this issue
and I don't have the time to check them all right now.

How will the Windows Store install work alongside the PSF one? Will it
work the same, in that neither will go on PATH unless the user
explicitly requests it? If I have PSF Python 3.7.1 and Store Python
3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
influence it to choose the other one? Same question for 3rd party
tools like "virtualenv -p 3.7" - I know it's not up to us to deal with
virtualenv issues, but conversely I do think we have a responsibility
to not change assumptions that 3rd party tools rely on. In terms of
PEP-514, will the Store installer use a different tag, or some other
way of having registry entries independent of the PSF installer?

Basically, I'm not clear on how having the Store install and the PSF
install simultaneously present will work - very few tools in my
experience really support PEP-514 (not even the py launcher) - and
frankly, if we're going to break tool assumptions that there's only
one official "3.7" present at any time, we're on very shaky ground
claiming that PEP-514 warned them if we haven't even updated our own
supplied tool! (I know that there's lots of ways people can have
multiple versions right now - PSF plus Anaconda, for example - but
it's fine to say "you're mixing distributions, you're on your own" in
that case. If the PSF is publishing two versions of 3.7, which can be
simultaneously installed, we can't really use that argument.

Paul

[zooba]

The first thing I'll note is that I'm not aware of any documentation
on the Windows Store installer, and how it works.

Right now, correct.

How will the Windows Store install work alongside the PSF one?

It will go into PATH under the same names as on other platforms (python, python3, python3.7, and same for pip and idle). The Windows Store makes it possible to do this reliably, so it's just on by default (and configurable through OS-provided UI for selecting the app to respond to a certain name).

If you have already installed it traditionally and enabled the PATH modification, that will take precedence. So you'll get the MSI installed Python. If you've modified PATH by hand, it depends on where you put it.

If I have PSF Python 3.7.1 and Store Python
3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
influence it to choose the other one? Same question for 3rd party
tools like "virtualenv -p 3.7"

Right now, py.exe will not detect it. This is an unavoidable limitation right now (specifically, enumeration of classic app compatibility keys does not work, though direct access does), but it may become avoidable in the future (and being able to say "Python needs it" is actually going to be useful for removing the limitation).

As usual, our recommended way to influence choice of Python is to specify it. "virtualenv -p 3.7" will find a traditionally installed Python first, but if it's not there and the store one is it *might* find the store one (it depends whether it requests the specific registry key or tries to enumerate all of them - again, I've got people looking into enabling enumeration, but we're talking about adding another feature to Windows specifically for Python, and those take time).

Basically, I'm not clear on how having the Store install and the PSF
install simultaneously present will work - very few tools in my
experience really support PEP-514 (not even the py launcher) - and
frankly, if we're going to break tool assumptions that there's only
one official "3.7" present at any time, we're on very shaky ground
claiming that PEP-514 warned them if we haven't even updated our own
supplied tool! (I know that there's lots of ways people can have
multiple versions right now - PSF plus Anaconda, for example - but
it's fine to say "you're mixing distributions, you're on your own" in
that case. If the PSF is publishing two versions of 3.7, which can be
simultaneously installed, we can't really use that argument.

In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.

[zooba]

How will the Windows Store install work alongside the PSF one?

Oh, and since I didn't mention it in my direct answer, the Store install goes to a totally different location (per machine, no elevation required), as do packages installed using pip (automatically per user). Nothing is shared between the two installs at all.

[pfmoore]

Thanks for the clarification.

In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

Agreed, it does sound like it's probably the correct approach. But I'd
strongly recommend that as much of the behaviour as possible gets
documented as part of this change. There are a *lot* of projects out
there that write code for Windows based on a relatively superficial
understanding of all of the complexities that come into play here -
most registry scanning code is based on guesswork and reverse
engineering of how things worked before PEP-514 (I should know, I
wrote rather a lot of it!!!) and plays fast and loose with even things
like 32-bit/64-bit registry overlaying (it's late and I'm tired, sorry
for not using the accurate term). Settig projects like virtualenv,
tox, nox, pew, pipenv etc up for new bug reports associated with store
installs, without any sort of guidance as to what's going on in the
official docs, seems unfriendly at best. (I'm thinking of a section in
"Python Setup and Usage" covering the store installer - that in
addition to the changes in
https://docs.python.org/3.7/using/windows.html#installation-steps
needed to mention that there's now five installers available).

A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.

The "pythonup" project (https://github.com/uranusjr/pythonup-windows)
creates versioned copies of python - python3.7.exe. Again, it's a
non-standard project, so not technically our problem, but I'd be
interested in how it would interact with the Store python.

I do appreciate that we need to get the store installer out there in
order to test things like this, and that initially at least there
aren't going to be a huge number of people using the store installer
over the traditional one (and likely vanishingly small numbers of
people using both in parallel). But conversely, there's a class of
problems that will be reported to other projects rather than to us,
and I think we should be trying to support those other projects in
dealing with this change, not simply leaving it up to them to find out
about it (I doubt they'd even know to ask "did you use the store
install or the traditional one" when diagnosing a problem).

In general, I don't think we've done that great a job of publicising
the various install options on Windows - we still get pip users trying
to install packages into the embedded distribution, which they are
using as if it were a standalone interpreter, and as far as I can tell
essentially no-one knows about the nuget packages for Python. Let's
not repeat the mistakes we've made previously with this new
distribution option.

[zooba]

In general, I don't think we've done that great a job of publicising
the various install options on Windows - we still get pip users trying
to install packages into the embedded distribution, which they are
using as if it were a standalone interpreter, and as far as I can tell
essentially no-one knows about the nuget packages for Python. Let's
not repeat the mistakes we've made previously with this new
distribution option.

Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?

So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.

[pfmoore]

Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

Sorry about that - I wasn't trying to make you feel guilty! And yes,
the embeddable package docs are very good. The problem with that one
is mostly that people don't *read* the docs (but having them to point
to makes responding to pip issues much easier, so you have my
gratitude there :-)

I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?

So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.

In which case, let's have a placeholder note in the docs, saying that
the Windows Store installer exists, but it's experimental and may have
unexpected interactions with other tools that expect a standard Python
install. Users trying the Windows Store installer should be prepared
for such issues, and we'd appreciate reports so that we can address
them. Add that to the release announcement as well for extra
visibility.

[zooba]

In which case, let's have a placeholder note in the docs, saying that
the Windows Store installer exists, but it's experimental and may have
unexpected interactions with other tools that expect a standard Python
install. Users trying the Windows Store installer should be prepared
for such issues, and we'd appreciate reports so that we can address
them. Add that to the release announcement as well for extra
visibility.

Sure, but first I need PR 11029 (and 11030 the backport) approved, so that I can rebase and work on redoing the other half of the change. Victor seems to have disappeared for the weekend, but I made all his requested changes.

[zooba]

I'm just waiting on Victor to sign off on PR 11029 (or someone else to do it) so I can merge it and rebase PR 11027.

Hopefully then someone will review it today so that we don't have to hold up 3.7.2rc1 any longer.

[vstinner]

Sorry, I don't have the bandwidth to provide a proper review.

[zooba]

Okay, I'll merge the first part now so I can rebase the main one and *hopefully* someone will be willing to review it now it's smaller. Otherwise, I think we're best to go with buildbots and all the testing I've already done.

This is my sole job today, so I'll keep an eye on the buildbots. I'll also get the docs updated as Paul suggested for the main PRs.

[zooba]

New changeset 1c3de54 by Steve Dower in branch 'master':
bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
1c3de54

[zooba]

New changeset b264c60 by Steve Dower in branch '3.7':
[3.7] bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
b264c60

[jkloth]

See also bpo-35450: venv module doesn't create a copy of python binary by default

[zooba]

Paul (and anyone else) - the below link should go directly to just the commit with the docs update. I did a slight rearrangement of the install docs to make the options clearer, and wrote up *just* enough info on nuget to help people use it right (I hope). Similarly for the Store package.

Any comments appreciated. Still aiming to get this in for 3.7.2rc1 today.

7d7ddbc

I've also kicked off a custom buildbot run against the 3.7 version of the change. The version in master ran earlier and was clean. Plus I'm running test builds on the release machine, so it should be ready to go.

[zooba]

Thanks, Paul. Appreciate it :)

Doing a final call before I hit merge in an hour or so. Right now:

• full release build works
• full test pass on all CI + custom buildbots work
• prior test package made it to the store
• docs clearly show it may change again

Any last pushback?

[zooba]

New changeset 0cd6391 by Steve Dower in branch 'master':
bpo-34977: Add Windows App Store package (GH-11027)
0cd6391

[zooba]

New changeset 0e4ad88 by Steve Dower in branch '3.7':
bpo-34977: Add Windows App Store package (GH-11027)
0e4ad88

[zooba]

Luckily, just noticed that I introduced a bug as part of splitting up the original PR into two.

It wasn't going to have an impact until the package was properly installed from the store, but without the proper directory resolution in there (since the variable I removed is never defined now), in some scenarios it wouldn't be possible to launch Python from a process outside of the app package, such as a virtual environment.

Testing in a proper app container is difficult, but I'll see if I can come up with a way to do it. It might require a custom buildbot, so I'll see whether there's a way I can do that (historically we've not been able to wire internal work machines - including Azure hosted ones - into other CI systems, but I'll try again).

[zooba]

New changeset d5a6a38 by Steve Dower in branch 'master':
bpo-34977: Remove unused preprocessor definition (GH-11092)
d5a6a38

[miss-islington]

New changeset 9bb306d by Miss Islington (bot) in branch '3.7':
bpo-34977: Remove unused preprocessor definition (GH-11092)
9bb306d