ConvertFrom-Yaml, ConvertTo-Yaml

[DarwinJS]

Would be great to support Yaml natively.

This was also mentioned by @fabiendibot on #3046

It would also be nice if the CMDLets had the goal of cleanly handling conversion of objects that came from XML as it seems like it would be a frequent use case. Maybe some good tests around this conversion?

[ArieHein]

We had a similar discussion from DSC aspect,
allowing us to change json based configuration files, we wanted to have options for modifying xml based files, YAML based files ,INI based files supporting RegEx swaps from within Text Manipulation cmdlets.

Lack of existing support in PS means we have to work hard to get such ability.
It has been on hold pending community contribution, but if it was baked into PS, it would make it much easier for the DSC part as well.

[lzybkr]

When you say natively, do you mean like XML or JSON?

The current thinking is that YAML should not be baked into PowerShell at all, instead it should be a separate module that you can update without picking up a new version of PowerShell.

If YAML were baked into PowerShell like XML, that would be impossible (think [xml]"b")

If we went the JSON route, you'd have cmdlets to work with YAML - so not really baked into PowerShell, but you'd still have the drawbacks of needing to update PowerShell to get YAML updates.

[joeyaiello]

@lzybkr I know we said we didn't want to bring in a new library, but I think this is something we might need to reassess. Ideally, we should also ship the module on the Gallery, but I think a TON of modern scenarios require YAML now.

Maybe not in 6.0 timeframe, but we should talk about it.

[DarwinJS]

@ArieHein - I have some simple functions that save and retrieve a hash array to the registry. Only handle REG_SZ - but for a simple set of settings it is sufficient - let me know if you want a copy.

I mispoke when I said "native" - I primarily meant "built-in" - it wouldn't bother me if they were shipped-in script modules that could be updated.

[iSazonov]

Our first discussion #2109

[DarwinJS]

@iSazonov - ah yes I see!

I noticed the reference to AWS support of YAML on the thread - I have been converting some templates and have found this to helpful: https://github.com/awslabs/aws-cfn-template-flip

[joeyaiello]

@iSazonov thanks for the pointer, I couldn't find it for some reason. Remember it well, though.

In re-reading that original thread, I think we should definitely implement the cmdlets at some point in the future, and ship them in the Gallery. Based on their quality, and people's perceived usefulness (along with some refactoring work we hope to do after 6.0.0), we can make the in-box vs. Gallery-only call.

[iSazonov]

We have Yaml in https://github.com/PowerShell/platyPS

[MattTunny]

yeah this would be awesome to have, ended up using https://github.com/awslabs/aws-cfn-template-flip to convert

[iSazonov]

@MattTunny Welcome to contribute! :-)

[josepmv]

There's a Windows Server user voice openned to vote for it :-)

https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/11088495-out-of-the-box-support-for-yaml-like-csv-xml-j

[Satak]

This should definitely be part of the native PS 6.1 library. So many things these days are in YAML.

[bergmeister]

There are now psyaml and powershell-yaml modules on the PSGallery but both are not even able to round-trip a YAML file from a VSTS build definition. I don't mind if the module is baked into PowerShell or is a module from the PSGallery.

[BrucePay]

I wonder if the core problem here is the clunky way we deploy modules. Today, you have to find, trust and install a module before you can use it. Compare this with the (apparently) slick way that Javascript does var m = require('mymodule'). Maybe we should have some way to do what DSC does but for native PowerShell. In DSC, when a module is referenced in a configuration, it's automatically downloaded and installed on the target node with no manual effort. Making critical but non-core modules available that way should eliminate the "it should be part of core" arguments. And for nodes that are disconnected from the net, we could have a tool that bundled the dependencies in a script into an archive which is then deployed to the target. This is how the Azure DSC resource extension works - there is a tool that scans a script to figure out the required modules then builds a zip file containing everything that is needed and publishes it to a blob. The Azure resource extension then pulls this blob, installs the modules and runs the script.

[bgshacklett]

For something that is this important, I really don't ever want to depend on a third-party library unless I have some way of vendoring it. It's way too easy for third party developers to potentially break entire ecosystems (see https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/).

Broader issues aside, there is currently no good YAML module for PowerShell, as @bergmeister pointed out. This is a must for a language which is heavily focused towards automation. YAML based configuration files are hugely popular now and it's very hard to avoid them even if you don't have to contend with the opinions of a team to do so. Think of the reasoning behind including XML and JSON as core parts of the language. The case for YAML really isn't so different.

[BrucePay]

@bgshacklett From what I've heard from the Puppet guys, there are just no good YAML parsers :-)

[iSazonov]

Is platyPS parser good enough?

@vors Is there simple way to reuse platyPS YAML parser in PowerShell Core repo?

[josepmv]

I prefer the idea of a separate official module in PowerShell Gallery like @lzybkr says because it would be possible to use it in older powershell versions and it could have its own releases. That would be like the sqlserver module. @BrucePay if it were a page in PowerShell Gallery with Microsoft own modules, it would be easier to find and everybody would know that they can trust them.

But I would understand if it were backed into Powershell as XML and JSON.

The important thing is that it exists ConvertFrom-YAML and ConvertFrom-YAML official functions because YAML is a widely used format for configuration files and it shouldn't be a third-party module, as @bgshacklett point out.

I made a blog entry testing and comparing the two modules I've found to work with YAML files: PSYaml and powershell-yaml.

They have different behaviours because internally they're using different objects:

module	mappings	sequences
PSYaml	OrderedDictionary	Array
powershell-yaml	Hastable	List

I think we need a standard ConvertFrom-YAML and ConvertFrom-YAML.

[gaelcolas]

Actually, ConvertFrom-Yaml in powershell-yaml uses OrderedDictionary when converting with the -ordered parameter.
I've been using this module successfully for a while (in my Datum module for DSC Configuration data, and with kitchen yamls), but don't have a vsts build definition to test with.

Bear in mind that the right way to call it is: get-content -Raw MyFile.yml | ConvertFrom-Yaml -Ordered (people often miss the -Raw).

I wonder why we'd need a Microsoft official module, putting even more overhead on MSFT and reinventing the wheel... Maybe trying to contribute to an existing one first, add tests to avoid regression, open issues to make sure the owner knows the problems is a better approach...
You know what happens when you're trying to create a standard out of the 99 existing implementations...

And yes it would be better outside the language, I agree that the dependency management could be better, bundling everything in PS is not a solution though.
The broad npm issue is also a failure in process. Fork and re-publish fixed it in no time, building apps out of latest version of the internet was the reason it broke so many live apps.

[markekraus]

I agree with @gaelcolas I think this is better with everyone working with the owners of an existing community module to raise and ensure quality.

I'll just add that tests for such a project should include working with a large number of real-world YAML files for things like AppVeyor, Travis CI, VSTS, AWS CloudFormation, etc. For my own experience with YAML deserilization, I have had little success with one solution working universally and have ultimately had to reinvent the wheel several times. In that sense, I agree with @BrucePay "there are just no good YAML parsers".

[iSazonov]

We are talking about this platyPS module because it is already actively used in PowerShell Help environment. I guess no one from MSFT can tell how good this module is because of Code of Conduct. They can either silently reject it or improve it.
And although we've been talking about this a long time ago I don't see how we could use the components of this module here in a simple way.
Maybe @adityapatwardhan and @SteveL-MSFT will open their plans and timeline especially as the new Help RFC is already in the experiment stage.

[SteveL-MSFT]

My personal view is that I would rather see more community modules succeed and become de facto standard than requiring "official" modules from Msft.

[markekraus]

@iSazonov It is one thing to have a solution that works for serializing/deserializing a well defined schema. It is quite another to have a solution that works in general with all schemas that are compliant YAML.

[iSazonov]

I understand the desire of MSFT to reuse community projects to cut costs. But the situation is, in fact, that MSFT may not make use of so many community projects:

• many have bad code, have no trust
• many projects are one person

MSFT has published Powershell specifications more 10 years ago, but nobody ported it yet until MSFT did.
The OpenSSL Project has existed for many years but nobody ported it to Windows while MSFT has not done this.
MSFT revealed many thousands of API interfaces, but how many of them were ported to Unix?
The interesting thing about why the company launched its project .Net Core rather than reuses Mono?
PowerShell is already a year and a half is an open source project, but I see that in this repository only one person from the community makes systematic contribution in the code @markekraus and only one person makes systematic analysis @mklement0.
I don't think that if we divide the project into parts, then we got more contributions.
I don't think the situation will change tomorrow. I wouldn't count on it.

[wasker]

IMHO, YAML is as popular as JSON and CSV, and not having inbox converters for YAML in PowerShell is kind of sad. Having inbox YAML converters will also ensure that their behavior is on par with JSON converters, which is not the case with community modules.

Don't get me wrong - I appreciate that folks create modules for community, but in the current state of the world, YAML conversion is table stakes - we don't expect folks to download 3rd-party modules for JSON conversion.

[marktoman]

@dirkslab You could use https://github.com/cloudbase/powershell-yaml

GitHub**cloudbase/powershell-yaml**PowerShell CmdLets for YAML format manipulation. Contribute to cloudbase/powershell-yaml development by creating an account on GitHub.

The problem is that unlike ConvertFrom-Json, this module deserializes to Hashtable instead of a proper object (or PSCustomObject), which doesn't allow using the pipeline in the following way:

$obj | ? Categories -gt 0 | % Name

My current workaround:

'...' | ConvertFrom-Yaml | ConvertTo-Yaml -JsonCompatible | ConvertFrom-Json

Suboptimal, to say the least.

GitHub

cloudbase/powershell-yaml

PowerShell CmdLets for YAML format manipulation. Contribute to cloudbase/powershell-yaml development by creating an account on GitHub.

[mklement0]

@marektoman, your example should work with a hashtable:

PS> @{ one = 1; two = 2 } | ? one -eq 1 | % two
2

Also note that you can cast hashtables to [pscustomobject] (albeit only one by one - see #13838):

PS> @{ one = 1; two = 2 }, @{ one = 10; two = 20 } | % { [pscustomobject] $_ }

one two
--- ---
  1   2
 10  20

[marktoman]

@marektoman, your example should work with a hashtable:

PS> @{ one = 1; two = 2 } | ? one -eq 1 | % two
2

Also note that you can cast hashtables to [pscustomobject] (albeit only one by one - see #13838):

PS> @{ one = 1; two = 2 }, @{ one = 10; two = 20 } | % { [pscustomobject] $_ }

one two
--- ---
  1   2
 10  20

Good point, the behavior is unrelated to hashtable, but it is likely related to the way each module does output.

When I run

$yamlArray | ConvertFrom-Yaml | ? Name -eq 'All'

, it doesn't work. However, this does:

($yamlArray | ConvertFrom-Yaml) | ? Name -eq 'All'

[dirkslab]

Thanks @marektoman , my solution was to remove the extra yaml module overhead and settle somewhere in between yaml and json.

I did something similar as you when requiring an pscustomobject:
$obj = Get-Content -Path .\deployment-config.yml -raw | ConvertFrom-Yaml | convertto-json -Depth 100 | ConvertFrom-json -Depth 100

The direction I have chosen to go with:
I have replaced all my yaml config with .psd1 and using Import-PowerShellDataFile

Not as clean as yaml, more than sufficient and easy to read though. Also eliminated any and all module issue.

Examples:
Sharing project with other devs that are domain policy restricted on installing random and unsigned modules, not an issue anymore.
I can now . browse path before I convert it to a string
adding comments are easy

My personal goals were to need little to no 3rd party modules for my ps project. I am now back on track and it's hassle free.

[mklement0]

@marektoman : The (...) forces enumeration, which in turn suggests that when ConvertFrom-Yaml outputs an array of hashtables that it does so as a single object, which contravenes the expected streaming (one-by-one) pipeline behavior. It is what '[ 1, 2 ]' | ConvertFrom-Json used to do (see #3424), and still does in Windows PowerShell.

The price you pay for the (...) workaround is that all output must be collected in memory first, but I suspect that won't be a problem in practice.

[marktoman]

@dirkslab Good idea for that use case. My use case is a shell independent tool that produces single output, which is both human and machine readable.

@mklement0 I see, thanks for all the info.

[Vaccano]

It is absolutely CRAZY that there is not first class support for yaml in PowerShell. I am just stunned.

The Kubernetes movement alone should be more than enough to get this prioritized.

[autosvet]

We should admit that YAML is used practically in many/most of cloud, CI, configuration, orchestration tools that IT pros, devs and DevOps need to use. So if PowerShell want to stay (most :D) valuable in that world for each it was designed, created and tailored specifically it should support it out of the box.

[dzmitry-lahoda]

can i donate 10 USD for the feature? I know there are services where donation is done until some dev implements

[Vaccano]

This page talks a lot about the powershell-yaml module as an option to get YAML support.

Does this library work on Linux with Powershell Core? or it just a Windows only thing. (I noticed that its dependencies include .Net Framework, so I am worried.)

[gaelcolas]

@Vaccano it works well on Linux.

[jdhitsolutions]

My initial thinking is that YAML support should be standard in PowerShell 7.x. YAML is such an integral language to today's enterprises that its omission is glaring considering support for XML and JSON. I have no problem with a separate module that can also be installed from the gallery, but it should be included by default in a new PowerShell 7.x installation. I know there are organizations that won't consider anything open source, and certainly nothing that doesn't originate or is supported by Microsoft. Given the value of PowerShell as a cloud language, native YAML support seems like the right decision.

I am also less concerned about making the module downlevel compatible. If someone needs YAML conversion support for Windows PowerShell, there appear to be a number of options in the PowerShell Gallery.

[redstrike]

As a normal user, I've installed PowerShell Core and use it in my work.
Our projects' ps1 scripts are not compatible with PowerShell Core. I have switched to the legacy PowerShell to run them.
And it's difficult to install a new module in this constrained work environment.
I don't know how to write PowerShell scripts, and now I have to live with the old PowerShell.
I feel sad because the new shiny PowerShell Core that I'm in love, is not usable in even a simple real-world usage.
Maybe, I have to rewrite those *.ps1 files by using a serious programming language.

[joeyaiello]

@PowerShell/wg-powershell-cmdlets has been discussing this issue, and despite the extremely high demand (and my own personal desire to have strong, canonical YAML cmdlets), we're unfortunately going to have to close it (for now) as Won't Fix. The reasoning is two-fold:

• While the PowerShell package is already fairly large, we're doing our best not to add new dependencies to the package so that we don't continue to grow our size unfettered, and because we believe that pulling modules from the PowerShell Gallery (or an internal trusted Gallery, having been vetted appropriately by an enterprise) should be a standard part of a PS7 workflow.
• As a team, we simply don't have the capacity to build a .NET YAML parser from scratch, nor do we have the capacity to vet, support, and service a 3rd party library for which the security implications are high-stakes.

If .NET were to officially support a YAML library, we (as Microsoft) could potentially look into supporting a YAML module on the Gallery. But until that point, the existing community modules will have to suffice.

[potatoqualitee]

Thanks, Joey. I'd like to file a Feature Request in dotnet but don't understand it well enough to know which repo to add the issue. Can you please suggest which repo would be most appropriate within https://github.com/dotnet ?

GitHub

.NET Platform

Home of the open source .NET platform. .NET Platform has 201 repositories available. Follow their code on GitHub.

[potatoqualitee]

(or could a Microsoft person/PowerShell team member place to the request as it may get a higher priority?)

[iSazonov]

See dotnet/runtime#28414

[potatoqualitee]

Thank you @iSazonov. That doesn't look promising :/ Seems that we do need an advocate from the PowerShell team to create the request, eh?

[iSazonov]

I guess Microsoft has internal conclusion that it is low priority and small area.

You can ask PowerShell team online today on Community Call PowerShell/PowerShell-RFC#299

[msftbot]

This issue has been marked as won't fix and has not had any activity for 1 day. It has been closed for housekeeping purposes.

[Vaccano]

This makes me sad.

Powershell says ".Net has to do it for us to do it"
.Net says "It is up to Powershell if they want to do it"

I wonder if some of the higher ups at Microsoft realize the runaround that is going on here.

YAML is important. Having it from a signed Trusted Source is critical for most companies. Powershell has built in restrictions making it hard to use things not signed by trusted sources.

I give up. It seems that the enough dust has been kicked up to obscure this issue. I imagine at some point someone high up at Microsoft will say "What? No YAML support in PowerShell?" Fingers will be pointed, blame laid, excuses made (etc). Then it will get done. But no amount of requests by the community will get this through.

[awakecoding]

Why has this been closed as WONTFIX? Seriously, YAML is so common these days, all the cloud-native stuff uses it. One can do so much with just vanilla PowerShell before they need to import a module, and not being to deal with YAML shouldn't be one of them. The built-in JSON cmdlets is one of the reasons why JSON is to popular in PowerShell: it's there, it's standardized, and it works. I am sure I am not alone defaulting to JSON just for that reason, and thinking twice about using YAML if I don't really need it, as it makes my scripts less portable or standalone.

[JustinGrote]

@awakecoding joey's explanation is here: #3607 (comment)

The Json library, even though it was Newtonsoft, was "blessed" by Microsoft as the official JSON library till system.text.json came around.

It's a bummer for sure, I would love native YAML support as well, but I personally think the reasoning is sound. I think energies should be focused on more easily pulling down dependencies quickly for a script/module.