Enable discovery of compatible Windows PowerShell modules in PSCore6 #130

SteveL-MSFT · 2018-06-15T00:44:03Z

No description provided.

…n PowerShell Core 6

markekraus · 2018-06-15T10:10:50Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+> The module '{0}' could not be imported as it is not declared as compatible.  Use `-SkipCompatibilityCheck` switch to override this check.
+
+`Import-Module` will support `-SkipCompatibilityCheck` switch to try and import the module regardless of `CompatiblePSEditions`.
+Since module auto-discovery does not expose cmdlets from modules not declared as compatible, explicit `Import-Module` with


As much as I hate to suggest an automatic preference variable, one likely needs to be added to allow auto-discovery to import "incompatible" modules.

Since PowerShell Core shipped without this being enforced, code has been written for Core which is absolutely compatible, but lacks CompatiblePSEditions. Additionally, plenty of modules originally written to target Windows PowerShell "just work" in PowerShell Core those modules may no longer be maintained or are maintained by a 3rd party who may not have updating their module manifest anywhere in their priority stack. That code may also be under license which forbids the user from making modifications as a work-around.

Given that there is plenty of compatible code this change will make "incompatible" simply because it lacks CompatiblePSEditions in the manifest, there should be a way for users to enable auto-discovery to bypass compatibility checks that does not require adding a ton of Import-Module -SkipCompatibilityCheck calls.

I agree, this setting is needed. But maybe it should be a more permanent one, for example in powershell.config.json.
Also, there can be a cmdlet, something like Set-ExecutionPolicy. Say, Set-ModuleAutoDiscoveryPolicy.

It needs to be toggleable at run time and possibly even scoped, so I disagree with putting it in powershell.config.json. I don't think it is permanent global state. It may only be needed in a single script of many that runs on a system. To me, that json file should have very very very compelling reasons to be expanded until there is a proper definition of everything that can and should go in it. and even then, it should be avoided unless it is a permanent global setting.

Set-ModuleAutoDiscoveryPolicy and Get-ModuleAutoDiscoveryPolicy would be a decent alternative to an automatic preference variable.

Perhaps I need to have a more explicit statement, but the intent is that the enforcement of CompatiblePSEditions is only for the System32 path.

ah.. yes after reading it your this more clearly stated I can see that now, definitely did not read like that was the case before that though. All good then. If the intent is only to CompatiblePSEditions on the system32 path, then there is no need for my suggestion.

Looks like I forgot to merge @joeyaiello's feedback which had additional language to clarify this. Please review again.

Yes, much clearer and acceptable as-is.

Just wanted to throw this out there:

The RFC seems to sort Modules into 3 categories:

Modules only compatible with WinPS

Modules only compatible with PSCore

Modules compatible with both WinPS and PSCore

...but I think there will be a fourth category for awhile...

Modules that explicitly need both WinPS and PSCore (i.e. some operations occur in PSCore and some occur in WinPS via the WindowsCompatibility Module).

For this fourth category, I'm thinking...

They should also go under the PSCore System32 $env:PSModulePath.

CompatiblePSEditions should be set to 'Core' (since the Modules would be using the WindowsCompatibility from a pwsh shell)

There should be another property that explicitly describes the need for both WinPS and PSCore in order for the Module to work properly.

WindowsCompatibility module works with any Windows PowerShell module from PSCore6 irrespective of the CompatiblePSEditions value. This RFC is about "natively" supporting Windows PowerShell modules while WindowsCompatibility achieves support via implicit remoting. I think we should keep those two things separate. Over time as more Windows PowerShell modules are ported, the need for WindowsCompatibility module will be diminished and eventually unnecessary.

rjmholt · 2018-06-15T15:58:45Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+### PowerShell Core PSModulePath
+
+Modules in the PowerShell Core PSModulePaths are implicitly deemed compatible even if the manifest does not


Does "PowerShell Core PSModulePaths" here refer to:

All paths in $env:PSModulePath (i.e. configured by the user)?

The default PowerShell Core module paths - being the ones that are part of $env:PSModulePath in a fresh PowerShell Core installation?

(2), but without the System32 directory proposed to be added in this RFC?

For 2/3, the best documentation I can find for what these paths actually are is in the code in ModuleIntrinsics.cs, which lists the "PersonalModulePath", "SharedModulePath" and "PSHomeModulePath". Are those the module paths to declare implicitly compatible?

Would it be possible to enumerate the specific paths that are:

to be module paths by default, and

to be implicitly compatible module paths

in this RFC (in the same way as the Windows PowerShell module paths above)?

PowerShell Core PSModulePaths refers to all paths that are part of $env:PSModulePath except System32. If the user changed the system PSModulePath env var, we should respect that. There is a separate PR that @BrucePay is authoring for PSModulePath6 that is outside the scope of this RFC.

The tl;dr; version is that we are special casing the System32 path in PSCore6. The existing PSModulePath should behave as it behaves today.

So any path found in PowerShell Core's $env:PSModulePath should be considered implicitly compatible?

rjmholt · 2018-06-15T18:14:42Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+The module manifest (since Windows PowerShell 5) may contain a `CompatiblePSEditions` property to indicate if it is compatible with
+Windows PowerShell (value of `Desktop` even on Windows Server) or PowerShell Core (value of `Core`).
+Currently, this property is only documentation and not observed by PowerShell.
+The proposal is to modify PowerShell Core 6.1 to respect this property.


What is the proposed behaviour for modules that don't have a manifest? I know the RFC is primarily to address published modules, but the handling of any PSModuleInfo will be affected by this.

I'm thinking that the manifest-default behaviour of assuming CompatiblePSEditions = @("Desktop") would be a serious break for simple .psm1s. So some choices there are maybe:

Set CompatiblePSEditions = null in the PSModuleInfo and require code paths to handle that (and not treat it like the manifest default).

Set CompatiblePSEditions = @("Core"), on the basis that we assume only compatibility with the loading runtime.

Set CompatiblePSEditions = @("Desktop", "Core"), on the basis that .psm1s without a manifest are informal enough that we take a "we won't get in your way; it should be easy to do simple things" policy.

iSazonov · 2018-06-18T06:05:09Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+### Windows PSModulePath
+
+Windows PowerShell, by default, has module paths in:


What paths do PowerShell Core use? Seems we need reword the line.

This section is only about Windows PSModulePath. PSCore6.1 change is the next section. This section is giving some contextual information.

iSazonov · 2018-06-18T06:06:32Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+  2. Program files: $env:ProgramFiles\WindowsPowerShell\Modules
+  3. System32: $env:Windir\System32\WindowsPowerShell\v1.0\Modules
+
+The proposed change is to add *only* the `System32` path to `PSModulePath` on PowerShell Core 6.


Looks like all modules in the folder is compatible with PowerShell Core 6. Seems the line should be moved in ### PowerShell Core PSModulePath paragraph.

I can see how it may be confusing to cover PSCore6 in this section, so it may be better to move this line to the next section.

So I'd change the section header to "Proposed change in PSModulePath".

iSazonov · 2018-06-18T06:07:23Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+The proposed change is to add *only* the `System32` path to `PSModulePath` on PowerShell Core 6.
+
+Modules in the user profile should be re-installed under PowerShell Core 6.


Why? Will it re-installed to another path?

I can add more context here if needed, but the point is that if you run install-module from Windows PowerShell, it installs to the Windows PowerShell location and won't be found by PSCore6 (unless Add-WindowsPSModulePath is used)

If a module was installed in $home\Documents\WindowsPowerShell\Modules where it will be after re-install?

I'll replace the word under with using. The idea is that you would use install-module from PSCore6 and thus it would be in the right location.

iSazonov · 2018-06-18T06:09:16Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+The proposed change is to add *only* the `System32` path to `PSModulePath` on PowerShell Core 6.
+
+Modules in the user profile should be re-installed under PowerShell Core 6.
+Duplicate modules are something that can be addressed in PowerShellGet and outside the scope of this RFC.


Seems we need define search path priority - set the order of viewing paths.

I can add a section lower that clarifies the default paths in PSCore6 along with their order.

Yes, maybe create a table to show paths and proposed change.

iSazonov · 2018-06-18T06:10:26Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+Modules in `Program Files` that may be compatible with PowerShell Core 6 is addressed below.
+
+Users who want to search the entirety of the Windows PowerShell `PSModulePath` should use `Add-WindowsPSModulePath` cmdlet
+from the `WindowsCompatibility` module.


What is motivation of the difference?

The focus of this RFC is specifically the System32 folder and not all the Windows PSModulePaths (line 15 above). I can add more text to clarify this if needed.

Please add.

iSazonov · 2018-06-18T06:12:02Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+### PowerShell Core PSModulePath
+
+Modules in the PowerShell Core `PSModulePath` are implicitly deemed compatible even if the manifest does not
+declare compatibility.


All modules in RSAT will be PowerShell Core 6 compatible?

Perhaps eventually, but certainly not initially.

iSazonov · 2018-06-18T06:15:42Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+If the module manifest contains `CompatiblePSEditions` with value `Core`, then that module is treated as compatible.
+If the module manifest contains `CompatiblePSEditions` with only the value of `Desktop`, then that module is treated as incompatible.
+If the module manifest does not contain `CompatiblePSEditions`, then that module is treated the same as `Desktop` which is incompatible.


This conflict with line 51:

Modules in the PowerShell Core PSModulePath are implicitly deemed compatible even if the manifest does not declare compatibility.

This section is specifically only about modules under System32 path

I don't understand - in line 40 we say that we add system32 path to PSModulePath.

iSazonov · 2018-06-18T06:20:02Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+## Alternate Proposals and Considerations
+
+Open issue: Should `CompatiblePSEditions` column only be added in `ModuleInfo` table format if used with `-SkipCompatibilityCheck`?


I believe we should show this always - we should inform users what modules is still not compatible to exclude hidden errors.

We can see what the user feedback is once @rjmholt has some working code checked in. My only concern is taking up too much column space. Although in table formatting, we could call it PSEdition and perhaps just D or C instead of Desktop and Core if we need to save space.

rjmholt · 2018-06-18T21:50:37Z

2-Draft-Accepted/RFCNNNN-PSCore6-and-Windows-Modules.md

+
+`Get-Module -ListAvailable` will support `-SkipCompatibilityCheck` switch to list all modules in `PSModulePath`.
+
+If `Import-Module` is used with a module that is not declared in the manifest as compatible, it will throw a terminating error:


What is the prescribed behaviour for non-psd1 modules -- ones that have no manifest?

What should the PSModuleInfo.PSEdition fields for those modules say?

Since this only applies to modules in System32, there shouldn't be any such modules. If there are, we should add a module manifest. However, if we missed any then I would expect it to be the same as having a module manifest without CompatiblePSEditions property.

Maybe I've missed something...

I'm thinking that Import-Module and Get-Module can be called on any module file, even ones not under System32. They can both produce a PSModuleInfo object and share code paths with manifest-described modules.

So with the new behaviour described in this RFC, Import-Module ./my/unpubslished/module.psm1 will have to make some decision on PSEdition compatibility settings. It seems tempting to say that no CompatiblePSEditions defaults to @("Desktop") in general, but that will mean breaking all psm1s in Core.

So we need to have a separate behaviour for standalone psm1s that the module cmdlets should follow, like just assuming total compatibility.

So the questions are, for a psm1:

Should we always assume compatibility?

What should PSModuleInfo.CompatiblePSEditions contain?

Does -SkipCompatibilityCheck have any function in that scenario?

Yes, you found a case not covered explicitly in the RFC which is modules that in PSModulePath. The intent is that CompatiblePSEditions is only enforced for modules under System32 in the section on line 63. So in all other cases, it is implicitly deemed compatible.

Ah!

I was reading the RFC (in the original version) as:

Modules in the PowerShell Core PSModulePaths are implicitly deemed compatible even if the manifest does not declare compatibility... If the module manifest contains CompatiblePSEditions with value Core, then that module is treated as compatible. If the module manifest contains CompatiblePSEditions with only the value of Desktop, then that module is treated as incompatible.

(meaning that modules not on the PowerShell Core PSModulePath are not implicitly compatible)

But are you saying that it's more specific to the System32 path:

All modules not found on the System32 path are implicitly compatibility-whitelisted. The compatibility of modules found on the System32 path will depend on their respective CompatiblePSEditions field.

Is there also intent for gallery module compatibility checking? Or only if they are installed to the System32 location?

kilasuit · 2018-06-27T12:31:49Z

My only issue is that until there is an update of the Modules housed in the System32 pushed out to users then this change is basically of no use to anyone because none of the modules contain this manifest property and personally don't think that modules should be getting installed here anymore. These should be left well alone and be Legacy modules.

Also any updates to the manifest of these modules would end up BREAKING module loading on pre v5 systems as the CompatiblePSEditions field was added in v5.1 timeline for Nano Server / Pwsh implementations going forward.

So for those that decide to install Pwsh on Windows 8/2012 and below where they are by default have a default version of PowerShell that is lower than PowerShell v5.1 this will cause more headaches as shown below with a manually edited version of the ActiveDirectory Module Manifest to include the CompatiblePSEditions Manifest Property - therefore you need to rethink this for Back Compat reasons.

What would make more sense to me would be to get these modules that are included in the System 32 PSModulePath getting released to the PowerShell Gallery and control them being updated from there with the changes as expected otherwise you need to issue Windows updates

All in all i don't think this is a well thought out RFC but this more stems from the poor implementation of the CompatiblePSEditions property in PS v5.1 which totally broke downlevel deployment of modules that contain this property.

SteveL-MSFT · 2018-06-27T17:18:59Z

@kilasuit we are not making any changes to downlevel Windows. The changes to the modules and thus the module manifests are only going to be in the latest Win10/WinServer builds. Releasing Windows in-box modules not owned by the PowerShell team is up to the feature owners. Modules in PSGallery are perfectly fine not using CompatiblePSEditions property in the module manifest if they need to support PSv4 or lower. This RFC explicitly states that the enforcement is only for modules in System32 path, so it should not impact any PSGallery modules.

…SCore6-and-Windows-Modules.md

SteveL-MSFT · 2018-07-26T00:00:55Z

@PowerShell/powershell-committee voted on this an accepted this RFC

SteveL-MSFT and others added 3 commits June 14, 2018 13:42

draft of enabling discover of compatible Windows PowerShell modules i…

3c3b7ec

…n PowerShell Core 6

feedback for CompatiblePSEditions RFC

8f228a9

Update RFCNNNN-PSCore6-and-Windows-Modules.md

f59e1d9

SteveL-MSFT added the Review - Open for Comments label Jun 15, 2018

markekraus reviewed Jun 15, 2018

View reviewed changes

rjmholt reviewed Jun 15, 2018

View reviewed changes

rjmholt mentioned this pull request Jun 15, 2018

Include a CompatiblePSEditions field in the module manifest PowerShell/PSReadLine#713

Closed

iSazonov reviewed Jun 18, 2018

View reviewed changes

address feedback for CompatiblePSEditions RFC

a418431

SteveL-MSFT force-pushed the CompatiblePSEditions branch from 01d1f6b to a418431 Compare June 18, 2018 21:41

rjmholt reviewed Jun 18, 2018

View reviewed changes

add clarity on applicability to only system32

72ecfda

rjmholt mentioned this pull request Jun 26, 2018

Check CompatiblePSEditions manifest field for modules loaded from Windows PowerShell $PSHOME PowerShell/PowerShell#7183

Merged

11 tasks

rename -SkipCompatibilityCheck to -SkipEditionCheck

2430094

SteveL-MSFT force-pushed the CompatiblePSEditions branch from 9c62d28 to 2430094 Compare June 27, 2018 22:26

SteveL-MSFT added Review - Committee and removed Review - Open for Comments labels Jul 2, 2018

rjmholt mentioned this pull request Jul 16, 2018

Get-Module -ListAvailable hides edition-incompatible modules but lists otherwise incompatible modules PowerShell/PowerShell#7298

Closed

SteveL-MSFT added 2 commits July 25, 2018 16:59

Update RFCNNNN-PSCore6-and-Windows-Modules.md

a6298ea

Update and rename RFCNNNN-PSCore6-and-Windows-Modules.md to RFC0025-P…

3fde0f1

…SCore6-and-Windows-Modules.md

SteveL-MSFT merged commit 4e3d880 into PowerShell:master Jul 26, 2018

SteveL-MSFT removed the Review - Committee label Jul 26, 2018


		### PowerShell Core PSModulePath

		Modules in the PowerShell Core PSModulePaths are implicitly deemed compatible even if the manifest does not


		### Windows PSModulePath

		Windows PowerShell, by default, has module paths in:


		The proposed change is to add only the `System32` path to `PSModulePath` on PowerShell Core 6.

		Modules in the user profile should be re-installed under PowerShell Core 6.


		## Alternate Proposals and Considerations

		Open issue: Should `CompatiblePSEditions` column only be added in `ModuleInfo` table format if used with `-SkipCompatibilityCheck`?


		`Get-Module -ListAvailable` will support `-SkipCompatibilityCheck` switch to list all modules in `PSModulePath`.

		If `Import-Module` is used with a module that is not declared in the manifest as compatible, it will throw a terminating error:

Enable discovery of compatible Windows PowerShell modules in PSCore6 #130

Enable discovery of compatible Windows PowerShell modules in PSCore6 #130

Uh oh!

Conversation

SteveL-MSFT commented Jun 15, 2018

Uh oh!

markekraus Jun 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sethvs Jun 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markekraus Jun 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pldmgg Jun 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjmholt Jun 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markekraus Jun 15, 2018 •

edited

Loading

sethvs Jun 15, 2018 •

edited

Loading

markekraus Jun 15, 2018 •

edited

Loading

pldmgg Jun 16, 2018 •

edited

Loading

rjmholt Jun 15, 2018 •

edited

Loading

rjmholt Jun 20, 2018 •

edited

Loading