Let's move PSModulePath out of the documents folder

[JohnLudlow]

Summary of the new feature/enhancement

The documents folder has always been where modules user-scoped are kept, and that's always been ok, but I recently discovered some problems with this as I've been getting more familiar with Azure and my work laptop maps Documents to OneDrive. This can be problematic for a few reasons, such as OneDrive not syncing correctly or trying to restore a bunch of files from a module I just uninstalled. This is particularly an issue with larger modules (or groups of modules) such as Az.*

We can certainly wish OneDrive was better at its job, but really these files shouldn't be there - they are not documents. The user is not expected to edit them or look at them. They are more akin to application files or application data. There's no value to them being in OneDrive other than to transfer them to another machine, and that can be better achieved by having a script in OneDrive that calls PSDepend or PowerShellGet to install modules on that other machine.

I have updated my profile script to remove this path but it reappears after running install-module.

Proposed technical implementation details (optional)

1. Remove Documents\PowerShell\Modules from the $env:PSModulePath default. Select a new default path such as ~\PowerShell\Modules or $env:localappdata\PowerShell\Modules
2. Offer a cmdlet or documentation to move the existing modules to the new location, and optionally update the value of the environment variable.
3. Additionally, offer to move the existing modules and update the value of the environment variable during the install. (That way the user has the choice to make the install take care of it or do it later at a time of their choosing).

[jeroenlandheer]

A few extra things to consider when putting this up for triage:

• When I install a module on one machine and I am later at a different PC, OneDrive updates the modules folder... but Upgrade-Module doesn't work anymore, because it says the module wasn't installed with Install-Module ... (which it was, just not at the machine I'm working on.)
• If your module folder is big loading PowerShell can take a (very) long time because it will wait until OneDrive has pulled all the files to your machine.

I think this change should happen... hopefully soon. I think storage for these modules should be in the %userpfile%\AppData\Local folder, this is the place for things that should not get synced with other machines.

PS: I create PowerShell modules... I have a lot of them... and they are all on my OneDrive now. 😒

[iSazonov]

@jeroenlandheer Do you mean Update-Module? If so you couls discuss in PowerShellGet repository.

[JohnLudlow]

@iSazonov Would the PowerShellGet guys be able to change the default value of $env:PSModulePath and update the installer to optionally update it on upgrade?

Though there should probably be a second issue in PowerShellGet to respect the value, not overwrite it.

[iSazonov]

@JohnLudlow Any changes related to PSModulePath are breaking changes. We need to keep backward compatibility.

[jeroenlandheer]

@iSazonov Sorry, that was a typo. This should indeed go there too, but don't you think they will choose the location of the modules where they install them... based on whatever this teams says it should be? (AFAIK there's no way to customize this with settings, but that's something for another day.)

If paths for loading the modules is the primary function of PSModulePath, this issue can be solved quite easily:

1. Designate a folder somewhere in %LocalAppData%\...
2. Tell others that installing modules in this folder is best practice.
3. Put both paths in PSModulePath (i.e. %LocalAppData%\.. and %userprofile%\Documents\...)
4. Ensure the new location is added to new installations in the PSModulePath environment variable.

No need to move stuff around, new things get installed in %LocalAppData%\.. and old modules that are still in the user's documents folder keep working. Maybe this doesn't even require a code change in PowerShell, just some changes in the installer.

And now we're on this subject, in Windows installations, modules which are installed on the local machine are installed by default in the Program files\PowerShell\Modules folder. I think the correct place for this should be %AllUsersProfile%\... (often called C:\ProgramData)

AFAIK the functionality of PSModulePath itself doesn't change with all of this, so this is technically not a breaking change.

Hope this helps.

[JohnLudlow]

@jeroenlandheer Indeed, and if I do want to move the older modules to the new folder (I probably would, in my case), there are some options:

• Offer to do that in the installer
• Offer a tool (such as a cmdlet) to do it post-install
• Offer documentation that says "you can copy everything from <old location> to <new location>"

Any of those would work for me, but some people may appreciate it being done for them.

And you're absolutely correct IMHO about the Windows Modules. The only modules that should be there are ones that are installed as part of PowerShell and shouldn't be removed.

[SeeminglyScience]

AFAIK the functionality of PSModulePath itself doesn't change with all of this, so this is technically not a breaking change.

While I agree that it would be hard to claim this is explicitly a breaking change of an API, it would still be disruptive. There has never been a supported way of determining the module paths of different scopes, e.g. no PSModuleInfo.AllUsersModuleRepository or similar. Any tools looking to interact with the module directories explicitly has been required to hard code the path, so any location change here will break these tools.

(Note that I'm not necessarily advocating against it, but it's something that needs to be considered)

[rjmholt]

Related: #7082 (comment)

[kilasuit]

Previously discussed in #8069 too

[rjmholt]

So we reviewed this on the @PowerShell/powershell-committee today, but also this is something I've personally done some thinking on in the past.

The mangling on particular locations on the PSModulePath is something we've seen users impacted by repeatedly. For example: PowerShell/vscode-powershell#2824 (comment).

Another important point is that the CurrentUser and AllUsers module paths are actually configurable in the powershell.config.json:

PowerShell/src/System.Management.Automation/engine/Modules/ModuleIntrinsics.cs

Lines 1304 to 1305 in e2c23fc

	string allUsersModulePath = PowerShellConfig.Instance.GetModulePath(ConfigScope.AllUsers);
	string personalModulePath = PowerShellConfig.Instance.GetModulePath(ConfigScope.CurrentUser);

So there is a concept of both the default CurrentUser module path and the actual one.

However, there's currently no way to communicate the actual one, so the default is the implicit contract between PowerShell's module system (Import-Module/Get-Module/command lookup) and PowerShellGet. I was hoping to improve that in #7082, but even without that the contact being implicit means there's room to get it wrong.

Today it seems that both PowerShell 7 and PowerShellGet agree that the default CurrentUser module path lies under the MyDocuments path, but evidence suggests that this wasn't always the case.

The big issue now is that OneDrive configurations override the MyDocuments folder and cause this unexpected migration.

But additionally, the actual CurrentUsers module path could be overridden with a configuration. And this is something that things like the PowerShell Azure Functions worker would want to do given the radically different environment in which it operates. Except that there's no way for PowerShellGet to pick this up.

In terms of changing the default PSModulePath, there have been a lot of proposals to fix PSModulePath in various ways. Changing the CurrentUser path away from MyDocuments is going to break PowerShellGet, since it depends on the same location being used. However, the MyDocuments path clearly presents problems, and both the proposed alternatives ($env:LOCALAPPDATA\PowerShell\Modules and $HOME\PowerShell\Modules) make sense.

So in terms of recommendations:

• The Engine Working Group has also been asked to review this, to go into a bit more technical depth.
• An API of some kind that gives both the default and actual module path components is probably a good direction to go in.
• In terms of backward compatibility, tooling can look for this API and fall back to the traditional defaults.
• The committee didn't decide on whether or not the default path should be changed. We want a chance for the Engine WG to discuss this, and also want to talk to the OneDrive team to understand things a bit better here.

[JohnLudlow]

@rjmholt That's a great summary. Thanks.

We'll see what the Engine Working Group say

[JamesWTruher]

With behavior of OneDrive the WG agrees that we need to come up with a proper solution, however the scenarios have a high level of complexity which is exacerbated by the need that other PS tools also need to be able to use whatever solution is made available.

The WG proposes that PS should provide an API which allows tools to query for the default location of modules. The tool chain also needs to be able to query PS so the values are consistent.

We propose that an RFC is needed to define the behaviors, not only for PowerShell proper but for tools like PowerShellGet.

We think at least the following requirements need to be addressed in the RFC:

• where is the default location for PSModulePath (system and/or user)
• how may this default location be modified
• how is the API used by PS and the other tools
• how are misconfigurations handled (non-existing paths, multiple paths, etc)

[XVII]

+1. Nuget doesn't have this problem because it uses Profile Root\.nuget -- though perhaps long term the %LOCALAPPDATA% would make more sense for the user modules for PowerShell.

It's like syncing a .git repo in OneDrive! Unnecessary.

Whichever solution comes up, it should be configurable for devices, i.e. via InTune or PowerShell so we can make it consistent for all our developers.

[mikhey]

Would a dot folder work here possibly? i.e. ~/.pwsh/ OR ~/.pwsh-modules I'm thinking of about Powershell running on other systems, Linux MacOS etc - if it could be consistent across platforms that would help "yugely" too. Well, I think at least.

I came across this as I'm looking to considerably cut down my documents folder when looking at OneDrive backup and I was hoping there was a way I could have PSGet use another folder. I'm still looking into what my options for this are - not sure if just changing PSModulePath env variable alone would do the trick.

[JohnLudlow]

@mikhey Changing PSModulePath works until you do Install-Module which breaks it again

A ~/.pswh folder (or whatever it gets called) would work, just as long as it's not somewhere OneDrive will try and mess with it

[rkeithhill]

My preference is ~\.pwsh (or ~\.powershell) on Windows. The Microsoft guidance is to use ~\AppData\Local\Microsoft\PowerShell or some folder like that under ~\AppData\Local. My problem with that is AppData is hidden which doesn't make for a great user experience when you tell folks to edit their ~\AppData\Local\Microsoft\PowerShell\profile.ps1 script. If they try to navigate to that file via File Explorer they're likely to get tripped up by the "missing" AppData dir.

Also, I notice that many other tools on Windows are now using ~\.<tool> folders:

[mikhey]

@JohnLudlow yeah, I was thinking of making it a symlink, but OneDrive doesn't like that for the backups. After I realized that I had much more space than I thought it did, I just let it go to OneDrive.

While it may have become a non-issue for me regarding OneDrive, I do feel it is something that should be addressed.

This has probably been mentioned before, but off the top of my head a few things to look at for an implementation could be:

• Configurable use profile file to explicitly set primary user modules install path - for backward compatibility/support
• A PowerShell environment variable or method that can be queried to locate user modules (cross module dependency)
• Upon first run of PowerShell or installing/updating/removing a module, the user should be offered to move modules to the "Out of Documents folder" location to the new "preferred"/standard user profile location.

[rjmholt]

Would a dot folder work here possibly?

As @rkeithhill said, those dot folders pollute $HOME, and even though they're not displayed, they still need to be processed by programs that examine the Home directory, creating a sort of tragedy-of-the-commons problem where more and more applications putting their configurations and data in a directory based in the Home directory adds up to slower filesystem performance for unrelated tasks.

PowerShell actually already conforms to the XDG base directory specification on *nix, which is a standard intended to solve this problem. PowerShell data files, including modules, are stored in $env:XDG_DATA_HOME/powershell, which defaults to $HOME/.local/share/powershell.

The issue then is that Windows doesn't really offer a clear answer here, because its philosophy is to separate users from configurers/application installers.

Worth noting is that the XDG standard is configurable with a sensible default, and I imagine we want the same, we just need to weigh configurability against complexity (mainly for anyone developing against our decision) and performance.

[dafzor]

@rkeithhill and dot files are supposed to also be hidden, but Windows is not linux so dot files are left always visible.

So I'd like to add my voice here in asking to please respect the platform conventions on Windows and use the appropriate AppData folder, and in the same way use XDG on Linux and whatever macOS specifies for it's convention.

As for accessibility for profile editing, I think that's already covered by using <editor of choice> $PROFILE which will work regardless of platform.

At most adding a $PROFILEHOME as an alias for (Get-ChildItem $PROFILE).Directory.FullName if the user wants to open the folder for any reason?

[codaamok]

I'm just stopping by to express my frustration with how things are today. I'm happy to read about positive discussions for change, though!

Trying to update PSModulePath in your profile or via registry (like the docs suggest) is pointless, because as previously mentioned, as soon as you use Install-Module, the env var is clobbered again within the session with the undesired OneDrive Documents folder. You'll also notice that Install-Module seems to ignore PSModulePath, too, as it wants to always search in MyDocuments\[Windows]PowerShell:

To workaround this, you need to update your MyDocuments (aka 'Personal') folder to no longer be in the OneDrive folder. This seems to work well for me:

1. Right-click My Documents (on the desktop), and then click Properties.
2. In the Target box, type the new path to the My Documents folder, or click Move to browse to the folder and if needed, create a new My Documents folder. If the path you type does not exist, you are prompted to confirm that you want to create a new folder.
3. Click Yes to move files from the old My Documents location to the new location, or click No if you do not want to move these files. Note that if you click No, the files in the old location are not deleted, but they are no longer visible from My Documents after the change.

Figured I'd share this in case anyone needed it /shrug

[billti]

Just to plus one on the pain and urgency here. I just lost half a day trying to figure out exactly the issue @jeroenlandheer mentioned back on June last year, namely trying to Update-Module on a machine fails because I installed the module on a difference machine, and both are sync'ing my documents via OneDrive.

Beside the issue above this causes, I'm now syncing over 900MB of PowerShell files to my OneDrive account, and I only have two modules installed! (The 'Az' and the 'Microsoft.Graph' modules - which I suspect are in common usage).

I'm probably just going to stop using OneDrive or install for AllUsers, but it was time consuming and frustrating to figure this out.

[ohadschn]

Can we include the scripts folder in this discussion too? Current behavior <modulespath>\..\Scripts seems like a hardcoded quick and dirty fix, an afterthought.

I think it should be treated separatly from the modules folder, at least have the ability to.

Maybe it could have it's own environmental variable, at least optionally?

• PSScriptPath?
• Default to PSModulePath\..\Scripts if PSScriptPath isn't set?

Agreed - without this, Controlled Folder Access would still block the installation of all modules that make use of that folder (for example, https://github.com/PowerShell/SecretManagement).

[bwilsonms]

@SteveL-MSFT since this didn't make 7.4, is it on deck for evaluation for 7.5? Looking forward to the roadmap :)

As mentioned, DevDrive is a thing now so if we want fancy buzzwordy reasons to make people happy, being able to set the default to D:.powershell\Modules would be nice :)

Not every powershell users needs a dev drive however, there are several people tasked with just running scripts all day, everyday.

[JustinGrote]

@bwilsonms of course, that would be just one of many reasons why this would be desirable.

[RobertKlohr]

It's now 2024.

@SteveL-MSFT, @JohnLudlow, @iSazonov, @rkeithhill, Et al. How do we keep this moving forward? It's been almost 3 years and while there seems to be consensus on the need, there does not seem to be any tangible progress toward fixing the problem.

This issue was tagged with milestone, 7.4 but that has passed. There were also several past comments about needing an RFC, but I never saw where one was started (What does RFC even mean in the context of a PowerShell setting?), and if it did can some point the way to where that work is being done?

I know this a considered a breaking change, but it can also alleviate a previous breaking change introduced by Controlled Folder Access (see post here) that many of us are having to work around on a daily basis.

We need this in 2024 and ideally in 7.5.

[JustinGrote]

As a workaround, ModuleFast installs to your LocalAppData by default and automatically sets up everything for you to make it work.

https://github.com/justingrote/modulefast

GitHub

GitHub - JustinGrote/ModuleFast: A "fast and loose" way to install modules from Powershell Gallery quickly. Meant for CICD, not production

A "fast and loose" way to install modules from Powershell Gallery quickly. Meant for CICD, not production - GitHub - JustinGrote/ModuleFast: A "fast and loose" way to install mo...

[ctmal956]

As it stands now I consider this a breaking experience as it is. We have our Documents folders redirected and scripts take forever to finish because of the constant searching for modules in our network folders.
Other than scripts I specifically need it for I'll continue to use the older versions of Powershell that I don't see this issue with.
As for keeping modules in the Documents folder...for the life of me I can't see how modules are considered Documents or how the Documents folder is an appropriate location for them. And not giving an option to change this behavior? This makes it a deal breaker in my opinion.

[JustinGrote]

As it stands now I consider this a breaking experience as it is. We have our Documents folders redirected and scripts take forever to finish because of the constant searching for modules in our network folders. Other than scripts I specifically need it for I'll continue to use the older versions of Powershell that I don't see this issue with. As for keeping modules in the Documents folder...for the life of me I can't see how modules are considered Documents or how the Documents folder is an appropriate location for them. And not giving an option to change this behavior? This makes it a deal breaker in my opinion.

It comes from very old decision way back in old PowerShell before OneDrive was even a concept. Can't predict the future :)

[FaffeF]

I think this is some sort of Microsoft sickness. Some decision is made early on (place stuff in "system32" folder). Then things change, or this is no longer a great idea (stuff should be in "system64" logically). But since other developers didn't actually read Microsoft's best practice guide, they hardcode the value. Then Microsoft for some reason goes out of their way to retain compatibility with all Windows versions ever released, making things ever more unintuitive.

Just make the change when it's needed.

Maybe get some guys from the Microsoft naming department, they seem to like making changes? Though then you'll probably run in to the "path to long" problem...

[danielniccoli]

As it stands now I consider this a breaking experience as it is. We have our Documents folders redirected and scripts take forever to finish because of the constant searching for modules in our network folders. Other than scripts I specifically need it for I'll continue to use the older versions of Powershell that I don't see this issue with. As for keeping modules in the Documents folder...for the life of me I can't see how modules are considered Documents or how the Documents folder is an appropriate location for them. And not giving an option to change this behavior? This makes it a deal breaker in my opinion.

It comes from very old decision way back in old PowerShell before OneDrive was even a concept. Can't predict the future :)

Which is why it is so important to stick to the OS guidelines and put that stuff in LOCAL APP DATA, instead of dreaming up new locations that WILL break things when new features are introduced, like OneDrive.

[KirillOsenkov]

Is there a way for a user to opt-out of OneDrive being added to PSModulesPath every time? Can we add a setting somewhere? It should be easy to add a setting and keep the old behavior by default.

[JustinGrote]

It's not OneDrive that gets added, it's the Documents folder path that onedrive is redirecting. The "simple" solution is to not redirect your My Documents to OneDrive, and just use OneDrive separately as a separate folder, there aren't any "good" alternatives that don't result in a lot of errors and I've tried a lot (symlinking, hardlinking, regedits, excludes, you name it).

[KirillOsenkov]

Can we add an option that adds a custom user-specified directory instead of Documents?

[KirillOsenkov]

or better yet doesn't add anything at all and relies purely on the PSModulesPath environment variable?

[JustinGrote]

That already exists, unfortunately it has to be done in a powershell.config.json file at startup in the same directory as pwsh.exe and cannot be configured at runtime.
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_powershell_config?view=powershell-7.4#psmodulepath

I requested that path be exposable but that was closed in favor of this issue:
#18613

And PowerShellGet Install-Module and Install-PSResource are hard-coded still to install to Documents despite this setting, those have to be updated to do it as well.

My ModuleFast module installs to localappdata by default and updates your PowerShell path and profile accordingly, so it is an alternative option in the meantime:
https://github.com/justingrote/modulefast

about PowerShell Config - PowerShell

Configuration files for PowerShell, replacing Registry configuration.

GitHub

GitHub - JustinGrote/ModuleFast: A "fast and loose" way to install modules from Powershell Gallery quickly. Meant for CICD, not production

A "fast and loose" way to install modules from Powershell Gallery quickly. Meant for CICD, not production - JustinGrote/ModuleFast

[KirillOsenkov]

is there a bug to support this setting in Install-Module?

[JustinGrote]

is there a bug to support this setting in Install-Module?

PowerShell/PSResourceGet#1494

[fsackur]

I have successfully used symlinks to solve OneDrive for hundreds of users.

Documents was redirected to OneDrive by policy. We created a folder in the user profile, moved modules there incrementally, and symlinked modules individually into Docs/blah/Modules. Modules can be moved on the fly, even if there are locked DLLs, with [IO.File]::Move (or Move-Item in PSv7). You also need to update the InstalledLocation in PSGetModuleInfo.xml if you want Update-Module to work.

[JustinGrote]

@fsackur pretty sure onedrive still complains of a sync conflict when you do this and there was no way even with exclusions to suppress it, unless something's been updated since I last attempted.

[mikhey]

@rkeithhill I agree it becomes an issue for users that don't have the setting to see hidden folders. That said the %LOCALAPPDATA% can be used in the File Explorer address bar and should work (in theory™).

I'm liking the ~/AppData/Local or ~/AppData/Roaming folders as it may line up with the ~/.local folder on Linux machines essentially. Ultimately, the sooner this can be configurable either via environment variable and/or config file would be fantastic!

[pl4nty]

OneDrive follows symlinks/junctions without errors on the latest versions (tested on MSIT Fast, Insider, and prod), but the status on files/folders is syncing even when sync has completed. I've been storing PowerShell and WindowsPowerShell on a Dev Drive using junctions for a few months.

There's a new setting to exclude folders by name and ADMX/ADMLs, but it doesn't activate even when using the example values. The file-based exclusion setting works with similar values. ProcMon shows OneDrive.exe loading EnableODIgnoreFolderListFromGPO values at startup, and I can see the codepath when decompiled, but symbols aren't available so I haven't debugged any further.

[JustinGrote]

RIght, it's that constant "syncing" status that I'm referring to, it's not seamless for casuals.

[JustinGrote]

@rkeithhill I agree it becomes an issue for users that don't have the setting to see hidden folders. That said the %LOCALAPPDATA% can be used in the File Explorer address bar and should work (in theory™).

I'm liking the ~/AppData/Local or ~/AppData/Roaming folders as it may line up with the ~/.local folder on Linux machines essentially. Ultimately, the sooner this can be configurable either via environment variable and/or config file would be fantastic!

I would argue ideally users don't need to ever see the modules folder except for advanced troubleshooting, similar to the registry, and only interface with it using the install/update/uninstall commands indirectly.

[ALIENQuake]

OneDrive follows symlinks/junctions without errors on the latest versions (tested on MSIT Fast, Insider, and prod), but the status on files/folders is syncing even when sync has completed. I've been storing PowerShell and WindowsPowerShell on a Dev Drive using junctions for a few months.

@pl4nty Can you please elaborate? Which Windows/OneDrive version are you using?

[danielniccoli]

I would argue ideally users don't need to ever see the modules folder except for advanced troubleshooting, similar to the registry, and only interface with it using the install/update/uninstall commands indirectly.

Factually incorrect. AppData is and has been the location to store application data and configuration (besides the registry) for over 30 years. It's only today that more and more inexperienced Windows developers starting abusing the profile folder to place configuration data, because they're used to macOs or Linux. As a sysadmin, I've migrated countless user profiles to new computers over the past decades and can tell you from experience, that most software (or at least the ones adhering to the standard) put their settings into AppData (or the registry).

AppData: "The directory that serves as a common repository for application-specific data for the current […] user."

https://learn.microsoft.com/en-us/dotnet/api/system.environment.specialfolder?view=net-6.0#system-environment-specialfolder-applicationdata

Furthermore:

https://learn.microsoft.com/en-us/dotnet/api/system.environment.specialfolder?view=net-6.0

[JustinGrote]

Factually incorrect. AppData is and has been the location to store application data and configuration (besides the registry) for over 30 years.

Powershell is an application. Modules are application data. Therefore that's where modules should go (Appdata) instead of documents, and "allusers" modules should go into ProgramData

I don't see anything here that counters that point. Maybe you misunderstood what I was saying?

[pl4nty]

@ALIENQuake currently OneDrive 24.086.0428.0003 on Windows 10.0.22631.3527

[ALIENQuake]

@pl4nty Well, ive tried and I'm still seeing "The 'ExampleFolderJunction' in Documents is junction point ..." error which prevents me from syncing Documents folder. Same for latest insider. Thanks for help anyway, I won't nagg you about this anymore.

[danielniccoli]

Factually incorrect. AppData is and has been the location to store application data and configuration (besides the registry) for over 30 years.

Powershell is an application. Modules are application data. Therefore that's where modules should go (Appdata) instead of documents, and "allusers" modules should go into ProgramData

I don't see anything here that counters that point. Maybe you misunderstood what I was saying?

I agree that modules, like other application data, should be placed within (Local)AppData or ProgramData. I misunderstood your argument as advocating for placing these files inside a dot-folder within the user profile.