Should we (partially) revert using platformdirs by default?

[Gitznik]

How would this feature be useful?

In #1001 platformdirs was introduced, to by default store the pipx data in the canonical location per OS. Since then, a couple of issues were raised:

Windows

The data is now stored in AppData. This led to the following issues that did not have a good resolution (yet):
#1168, #1203 (thanks @dechamps, @guahki for doing a ton of work here)

The jist of it is, that the AppData folder is subject to path redirection due to sandboxing. This, in combination with the microsoft store installation of Python, means that the --python flag is no longer compatible with other installed microsoft store python versions.

On top of this being basically not fixable, it's also according to @dechamps and @guahki investigation impossible to replicate in an automated test.

MacOS

The data is now stored in the Application Support folder by default. This led to the following issues that did not have a good resolution (yet):
#1198, #1212

The whitespace in the folder messes up shebang functionality in

• using a pipx installed application in a shebang (e.g. aws cli)
• using scripts/gui-scripts

While our documentation states that applications installed via pipx should declare an entrypoint, that is not impacted by a space in the path, a lot of real applications don't do this and rely on scripts instead. All of these scripts don't work in the new default location. There is also no way for us to fix this, aside from us advising our users to set a different PIPX_HOME.

Linux

All good here

Describe the solution you'd like

Not really a solution, but I'd like to have an open discussion on whether we should (partially?) revert using platformdirs by default. I'd especially be interested in the factual benefits we see in using the platformdirs locations by default, aside from following the recommendations of the OS.

I see 2 possible solutions:

1. Stick to using platformdirs by default

IMO in this case we should very clearly document the limitations of using the default settings and try to catch as many possible issues as possible, advising users that run into them to change PIPX_HOME.

For most users, not being able to use --python would probably be fine, especially in combination with being able to just fetch missing python versions instead of using the microsoft store versions (#1242).

Having all application scripts broken on mac seems like a bigger issue to most users.

On windows a big issue I see here is that our default behavior would rely on functionality we can't really test at all, which is the microsoft store python version weirdness in combination with the AppDirs sandboxing.

2. Change the default behavior on Windows and MacOS

Have default values that lead to the most robust experience we're aware of, which is the old paths we were using. We can have a new env variable that enables the platformdirs paths for users that are happy with the limitations that come with it.

[Gitznik]

IMO we should revert the behavior on mac and windows. The original reason for introducing it was to adhere to the XDG specs on linux. AFAIK no mac or windows user was complaining about the install location on their OS, so introducing platformdirs for them was solving an issue that did not exist and overall made the experience on those operating systems worse.

[guahki]

I think it would be sensible to revert to the old locations on Windows and MacOS. As you said: the original reason for change does not really affect Windows and MacOS in the first place.

The only thing to think about is the way to handle people using the current location. Should there be a fallback mechanism (maybe with a warning?), ...

[Gitznik]

Yeah we can just reuse the existing fallback code for users that still had the old locations and turn it around. The old locations would be the default locations again, so there's no more fallback for that needed.

[Gitznik]

As there does not seem to be anyone speaking out against reverting, I've turned this into an issue and will close this discussion for now.
#1257

[jaraco]

@zooba FYI, based on the summary above and linked analyses, it's not viable for Python applications to store app data (or maybe just virtualenvs) in AppData. The issue seems to be related to issues deleting files in use. I haven't looked deeply into it, but I thought I'd bring it to your attention in case you have any better recommendations than to simply install in ~/.local or wished to register some of these issues with Windows itself.

[zooba]

From a quick skim, I'm guessing this is the problem of paths being redirected inside the Python "app" but not from outside, so whenever Python writes a path to it somewhere else, it's going to have to pre-resolve it (inside the sandbox) first.

What that basically means is that it's not viable for Python apps to share app data with other applications in AppData. And that's considered "correct" by Windows (though there's a growing sense of it also being considered "annoying to non-Windows devs", which may result in the data model eventually being broken to make things easier for non-Windows devs... I'd be more of a fan of this if it was ever reciprocated 😉 ).

So the choices are basically:

• store shared data outside of AppData - I'm not a fan of .local as it's very likely to collide, but ~\pipx would be fine, as would ~\Documents\pipx\... if it makes sense to have these migrate automatically between user machines
• only access the data via Python - so any paths that get stored have to be used through Python, and are only useful to other apps to pass back into Python

I see there was a change from os.system("rmtree <dir>") to shutil.rmtree("<dir>"), which moves the resolution inside the Python process and so will get redirection correct. I forget the exact resolution order as it's changed a couple times, but passing an already-redirected path back into Python should be handled fine.

[Gitznik]

Unfortunately, as we support using different python versions via the python flag (including ms store installed ones, which are stored under AppData, and executed via a subcommand), I think number 1 is not really viable for us.

Can either of you give me some context as to why .local\pipx is not a good location on Windows? We would have another layer of backwards compatibility to maintain if we chose to use another path as default, so I'm interested if that change seems worth the effort.

[zooba]

It's just that .local is a completely made up location with no meaning, whereas generally you'd name the directory after your tool/company. And a dot prefix is equally meaningless, apart from sometimes making it impossible to interact with the directory.

The user profile directory is already basically inaccessible to users - they'll usually have convenient access to ~\Desktop, ~\Documents, ~\Pictures or ~\Videos, but not anything else under ~. So you aren't hiding it any better with the dot or the unrelated name, you're just being extra inconvenient.

The exception I'd make here is if ".local" is your registered trademark or registered business name 😉 That's the kind of reason you'd use that name. Otherwise, you're just advertising that you don't use Windows and don't care to learn anything about it, which may or may not be true, but I'm sure you'd prefer not to advertise it! 😆

[Gitznik]

Gotcha, thanks for this. The more you know 👍