PS Community Call for September 17, 2020

[joeyaiello]

Please add topics/questions for the September 17, 2020 Community Call here.

Agenda:

• PowerShell 7.1 preview 7
• Ignite 2020
• PSReadline list view
• Gallery statistics
• PowerShellGet update
• SecretManagement update
• Notebooks
• Q&A

[sdwheeler]

EDIT - Docs announcement - I will be doing a presentation to RTPUG on WEDNESDAY evening 8pm EDT. I will be talking about PowerShell-Docs and how to contribute.

This is happening tonight 9/16, so this will be too late for the community call.

[KirkMunro]

Discussion topic: separation of native cmdlets into modules that are compatible with Windows PowerShell 5.1 as well as PowerShell 7.x.

Specifically, I'm wondering about the ConvertTo-Json and ConvertFrom-Json cmdlets, the need to move those forward to .NET Core JSON APIs, the need to provide SDK-level access to JSON conversion APIs so that .NET code that uses S.M.A. can do conversions without having to invoke PowerShell, and the desire to address the issues with the JSON cmdlets that would otherwise be breaking if those changes were to be made in the native, built-in cmdlets we have today.

Relevant discussion: tail end of the conversation on this open PR: https://ucclearly.myjetbrains.com/youtrack/dashboard?id=e5b14068-e378-437b-baa4-7199fc8f9a17.

[TylerLeonhardt]

the need to provide SDK-level access to JSON conversion APIs so that .NET code that uses S.M.A. can do conversions without having to invoke PowerShell

@KirkMunro I believe we added this in 6.2 because of Azure Functions

[rjmholt]

I believe we added this in 6.2 because of Azure Functions

In https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/JsonObject.cs.

Example of usage: https://github.com/PowerShell/PowerShell/blob/3effa204103460c996a8612aa70718fdf924047d/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommand.cs#L115-L125

[vexx32]

Would be good to document this usage somewhere for cmdlet authors to reference, I think. Had no idea we could do that. 👀

[KirkMunro]

@TylerLeonhardt and @rjmholt That's great for .NET Core, but that doesn't help Windows PowerShell, right? Although I suppose I could have a separate DLL for .NET Core that wraps those APIs and exposes them to my Windows PowerShell .NET Framework executable.

Even with that in place though, it doesn't speak to the desire to move forward to .NET Core JSON APIs, how to deal with case-insensitive JSON that contains multiple entries with the same value but different cases (as can happen from a simple Get-ADUser -filter * -properties * | ConvertTo-Json), right? Or maybe it does address some of my issues if it exposes the -AsHashtable functionality via the SDK. I'll have to take a closer look.

[rjmholt]

Would be good to document this usage somewhere for cmdlet authors to reference, I think. Had no idea we could do that. 👀

Needs to go in the XML comments so it can appear in the dev documentation.

That's great for .NET Core, but that doesn't help Windows PowerShell, right?

Yeah, this API is only available in PS 7+. Windows PowerShell has half the API (not such which) but is missing the other half, and in fact I think uses a totally different JSON engine.

To pick apart your question a bit better so I understand it:

• You'd like to convert JSON in SDK code, away from the pipeline (so no PowerShell invocation required)
• You want to customise some behaviour, like case-insensitive key handling
• Also, what happens when the underlying JSON API changes

So to answer as best I can given my understanding:

• PowerShell's own JSON conversion APIs are provided not to convert arbitrary JSON, but to convert JSON in a PowerShell-specific way. There are other, lower-overhead ways to convert JSON, but they won't handle PSObjects or special keys in objects the way PowerShell does.
• The behaviour of those conversion APIs is tuned to PowerShell's conversion cmdlets. They don't seek to offer configuration, but instead align with the Convert*-Json cmdlets.
• For your own custom JSON conversion needs, you should prefer a configurable API that isn't coupled to PowerShell, like the Newtonsoft library or the new .NET JSON conversion library.
• As for moving to .NET's new JSON library in our own implementation, our intent is very much for users not to know or care. Newtonsoft and System.Text.Json are both already shipped in PS 7. Which one is used by PowerShell's own JSON conversion logic should be transparent to consumers.

Let me know if that makes sense or doesn't address what you're looking for

[KirkMunro]

• PowerShell's own JSON conversion APIs are provided not to convert arbitrary JSON, but to convert JSON in a PowerShell-specific way. There are other, lower-overhead ways to convert JSON, but they won't handle PSObjects or special keys in objects the way PowerShell does.

That's fine, I'm only doing conversion on PowerShell object data that is returned from a script. I have a runtime that runs PowerShell, and then I send the results off via a REST API call. Those results need to be converted into JSON first, and compressed. Then we decompress and store them. So this is exactly what I need, not some general purpose JSON conversion that isn't about PSObjects, special keys, etc.

• As for moving to .NET's new JSON library in our own implementation, our intent is very much for users not to know or care. Newtonsoft and System.Text.Json are both already shipped in PS 7. Which one is used by PowerShell's own JSON conversion logic should be transparent to consumers.

Right, end users shouldn't know or care, but if the backend library is more performant, and possibly more forgiving/inline with PowerShell that prefers case insensitive over case sensitive information, as someone who works with a runtime that executes PowerShell, I do have some preferences over what is used on the back end, especially when it comes to performance.

Thanks @rjmholt for those answers. My interest in a cross platform standalone JSON library may become significantly less important if I can plug in these APIs and cherry pick the newer functionality that is in the SDK that powers those cmdlets in PowerShell 7.x.

[rjmholt]

So this is exactly what I need, not some general purpose JSON conversion that isn't about PSObjects, special keys, etc.

Understood.

Right, end users shouldn't know or care, but if the backend library is more performant, and possibly more forgiving/inline with PowerShell that prefers case insensitive over case sensitive information, as someone who works with a runtime that executes PowerShell, I do have some preferences over what is used on the back end, especially when it comes to performance.

Absolutely. And it's something we've been discussing on and off. Certainly if we were making the move today we'd be using System.Text.Json. The concern is that making the move without breaking small edge cases is hard. On that note, the case sensitivity isn't an implementation detail of the JSON engine, it's a deliberate choice for PowerShell, since PowerShell tends to think case-insensitively -- so moving over to a new engine, we'd keep that behaviour. If you needed to change that behaviour, you'd need to move away from PowerShell's own JSON APIs to get the specific behaviour you're looking for.

a cross platform standalone JSON library

For this goal, I would say you've got a couple of possibilities:

1. Wrap calls to Convert*Json with the PowerShell.AddCommand().Invoke() API
2. Wrap a runtime-multiplexed call to the appropriate underlying PowerShell API
3. Implement your own conversion logic based on a cross-platform library

(2) tends to be what we do in PowerShellEditorServices. For example, the debugging APIs introduced (with your help @KirkMunro) in PS 7 allow us to do new things in the debugger (for example set a breakpoint while script is running), but don't exist in older versions. So at load-time we do some reflection to discover those APIs and if they exist we call them, but if not we fall back to calling the appropriate cmdlet (in this case Set-PSBreakpoint). For JSON cmdlets, I would look for the corresponding API and if it doesn't exist, fall back to calling Convert*-Json on a pipeline.

[iSazonov]

The discussion with @KirkMunro was started in #11198 and I hope MSFT team accept my suggestion PowerShell/PowerShell#11198 (comment) (in short - to use .Net Runtime dev model) - It's incredible but it's easier to invest in .Net today than in PowerShell.

We're establishing Working Groups to decentralize management of Area-* labeled PRs and issues

What is a progress in the intention?
I would like to invest more in file system provider, web and json cmdlets (for which we have many issues), engine optimizations but now it just doesn't make sense...

[ThomasNieto]

• @mklement0's issue/comment on removing the requirement for curly braces when using null-conditional operator. Do not require { ... } around variable names for null-conditional member access PowerShell#11379 (comment)

• .NET MAUI (cross-platform GUI) could allow for PowerShell's GUI cmdlets to support Mac/Linux.

• PowerShell's updatable help is a powerful but underutilized feature. One of the blockers is hosting the HelpInfo.xml and cab files on a web server. Could PowerShell Gallery be updated to allow module authors to host these files?

[mklement0]

Discuss the path toward resolution of PowerShell/PowerShell#1995 (broken passing of empty arguments and arguments with embedded double quotes to native executables), specifically:

• The state of the PSEscapeDoubleQuotedStringForNativeExecutables experimental-feature PR WIP: Try to manage quotes in strings when calling native executables PowerShell#13482, with respect to:

  • still not passing empty arguments through (as observed by @Artoria2e5): WIP: Try to manage quotes in strings when calling native executables PowerShell#13482 (comment)
  • not containing important accommodations for calls to batch files and msiexec-style CLIs on Windows: Arguments for external executables aren't correctly escaped PowerShell#1995 (comment)

• The need to widely publicize this experimental feature once implemented, to gather important feedback whether fixing the problem by default is acceptable, despite breaking existing workarounds, so as to put an end to 15 years of quoting hell.

[JustinGrote]

Q: What is the plan for Powershell Standard with .NET 5 since the .NET Standard approach is changing?
https://devblogs.microsoft.com/dotnet/the-future-of-net-standard/

Also, I feel like PSStandard still includes a ton of DLLs that have to be manually removed for Windows Powershell 5.1 for those modules to import correctly with Add-Type, it seems like an additional privateassets dependency list may be required to cut down on the assemblies generated for a binary/hybrid Powershell Module. Any potential for optimizing some of that?

[JustinGrote]

Q: Any updates on Assembly Load Context, etc. to work with side-by-side issues between modules? I imagine system.text.json is going to become the new common problem here. Maybe add something to import-module that if multiple modules are added simultaenously, discover from the manifests the highest version assembly and load that first so the others will binding redirect up in .NET core?

[SteveL-MSFT]

Thanks everyone. Listen to the recording for answers/discussion to these questions.

[ili101]

Thanks everyone. Listen to the recording for answers/discussion

@SteveL-MSFT Can you share the recording?
Thank you

[mklement0]

In case there are others (like me until now) who are unfamiliar with the process: The recordings and transcripts are usually posted at https://aka.ms/PSCommunityCall, which, however, hasn't happened yet for the September call.

[vexx32]

@ili101 Prior to those going up you can also view the most recent recording via Teams itself from the https://aka.ms/JoinPSCall link, which will have the prior meeting's recording available until they update that shortlink for the next month's meeting.

[mklement0]

Thanks, @vexx32, good to know.

However, it would still be beneficial to have the YouTube link posted soon after and within a predictable time frame:

• It is a permalink.

• It allows linking to specific portions of the recording, which can be helpful in discussions on issues.