Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConvertFrom-Yaml, ConvertTo-Yaml #3607

Closed
DarwinJS opened this issue Apr 20, 2017 · 88 comments
Closed

ConvertFrom-Yaml, ConvertTo-Yaml #3607

DarwinJS opened this issue Apr 20, 2017 · 88 comments

Comments

@DarwinJS
Copy link
Contributor

@DarwinJS DarwinJS commented Apr 20, 2017

Would be great to support Yaml natively.

This was also mentioned by @fabiendibot on #3046

It would also be nice if the CMDLets had the goal of cleanly handling conversion of objects that came from XML as it seems like it would be a frequent use case. Maybe some good tests around this conversion?

@ArieHein
Copy link

@ArieHein ArieHein commented Apr 20, 2017

We had a similar discussion from DSC aspect,
allowing us to change json based configuration files, we wanted to have options for modifying xml based files, YAML based files ,INI based files supporting RegEx swaps from within Text Manipulation cmdlets.

Lack of existing support in PS means we have to work hard to get such ability.
It has been on hold pending community contribution, but if it was baked into PS, it would make it much easier for the DSC part as well.

@lzybkr
Copy link
Member

@lzybkr lzybkr commented Apr 20, 2017

When you say natively, do you mean like XML or JSON?

The current thinking is that YAML should not be baked into PowerShell at all, instead it should be a separate module that you can update without picking up a new version of PowerShell.

If YAML were baked into PowerShell like XML, that would be impossible (think [xml]"b")

If we went the JSON route, you'd have cmdlets to work with YAML - so not really baked into PowerShell, but you'd still have the drawbacks of needing to update PowerShell to get YAML updates.

@joeyaiello
Copy link
Contributor

@joeyaiello joeyaiello commented Apr 20, 2017

@lzybkr I know we said we didn't want to bring in a new library, but I think this is something we might need to reassess. Ideally, we should also ship the module on the Gallery, but I think a TON of modern scenarios require YAML now.

Maybe not in 6.0 timeframe, but we should talk about it.

@DarwinJS
Copy link
Contributor Author

@DarwinJS DarwinJS commented Apr 20, 2017

@ArieHein - I have some simple functions that save and retrieve a hash array to the registry. Only handle REG_SZ - but for a simple set of settings it is sufficient - let me know if you want a copy.

I mispoke when I said "native" - I primarily meant "built-in" - it wouldn't bother me if they were shipped-in script modules that could be updated.

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Apr 21, 2017

Our first discussion #2109

@DarwinJS
Copy link
Contributor Author

@DarwinJS DarwinJS commented Apr 21, 2017

@iSazonov - ah yes I see!

I noticed the reference to AWS support of YAML on the thread - I have been converting some templates and have found this to helpful: https://github.com/awslabs/aws-cfn-template-flip

@joeyaiello
Copy link
Contributor

@joeyaiello joeyaiello commented Apr 21, 2017

@iSazonov thanks for the pointer, I couldn't find it for some reason. Remember it well, though.

In re-reading that original thread, I think we should definitely implement the cmdlets at some point in the future, and ship them in the Gallery. Based on their quality, and people's perceived usefulness (along with some refactoring work we hope to do after 6.0.0), we can make the in-box vs. Gallery-only call.

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Oct 8, 2017

@MattTunny
Copy link

@MattTunny MattTunny commented Oct 25, 2017

yeah this would be awesome to have, ended up using https://github.com/awslabs/aws-cfn-template-flip to convert

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Oct 25, 2017

@MattTunny Welcome to contribute! :-)

@josepmv
Copy link

@josepmv josepmv commented Dec 1, 2017

@SteveL-MSFT SteveL-MSFT added this to the 6.1.0-Consider milestone Dec 12, 2017
@Satak
Copy link

@Satak Satak commented Mar 13, 2018

This should definitely be part of the native PS 6.1 library. So many things these days are in YAML.

@bergmeister
Copy link
Contributor

@bergmeister bergmeister commented Apr 24, 2018

There are now psyaml and powershell-yaml modules on the PSGallery but both are not even able to round-trip a YAML file from a VSTS build definition. I don't mind if the module is baked into PowerShell or is a module from the PSGallery.

@BrucePay
Copy link
Collaborator

@BrucePay BrucePay commented Apr 24, 2018

I wonder if the core problem here is the clunky way we deploy modules. Today, you have to find, trust and install a module before you can use it. Compare this with the (apparently) slick way that Javascript does var m = require('mymodule'). Maybe we should have some way to do what DSC does but for native PowerShell. In DSC, when a module is referenced in a configuration, it's automatically downloaded and installed on the target node with no manual effort. Making critical but non-core modules available that way should eliminate the "it should be part of core" arguments. And for nodes that are disconnected from the net, we could have a tool that bundled the dependencies in a script into an archive which is then deployed to the target. This is how the Azure DSC resource extension works - there is a tool that scans a script to figure out the required modules then builds a zip file containing everything that is needed and publishes it to a blob. The Azure resource extension then pulls this blob, installs the modules and runs the script.

@bgshacklett
Copy link

@bgshacklett bgshacklett commented Apr 24, 2018

For something that is this important, I really don't ever want to depend on a third-party library unless I have some way of vendoring it. It's way too easy for third party developers to potentially break entire ecosystems (see https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/).

Broader issues aside, there is currently no good YAML module for PowerShell, as @bergmeister pointed out. This is a must for a language which is heavily focused towards automation. YAML based configuration files are hugely popular now and it's very hard to avoid them even if you don't have to contend with the opinions of a team to do so. Think of the reasoning behind including XML and JSON as core parts of the language. The case for YAML really isn't so different.

@BrucePay
Copy link
Collaborator

@BrucePay BrucePay commented Apr 24, 2018

@bgshacklett From what I've heard from the Puppet guys, there are just no good YAML parsers :-)

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Apr 25, 2018

Is platyPS parser good enough?

@vors Is there simple way to reuse platyPS YAML parser in PowerShell Core repo?

@josepmv
Copy link

@josepmv josepmv commented Apr 25, 2018

I prefer the idea of a separate official module in PowerShell Gallery like @lzybkr says because it would be possible to use it in older powershell versions and it could have its own releases. That would be like the sqlserver module. @BrucePay if it were a page in PowerShell Gallery with Microsoft own modules, it would be easier to find and everybody would know that they can trust them.

But I would understand if it were backed into Powershell as XML and JSON.

The important thing is that it exists ConvertFrom-YAML and ConvertFrom-YAML official functions because YAML is a widely used format for configuration files and it shouldn't be a third-party module, as @bgshacklett point out.

I made a blog entry testing and comparing the two modules I've found to work with YAML files: PSYaml and powershell-yaml.

They have different behaviours because internally they're using different objects:

module mappings sequences
PSYaml OrderedDictionary Array
powershell-yaml Hastable List

I think we need a standard ConvertFrom-YAML and ConvertFrom-YAML.

@SteveL-MSFT SteveL-MSFT removed this from the 6.1.0-Consider milestone Apr 28, 2018
@SteveL-MSFT SteveL-MSFT added this to the 6.2.0-Consider milestone Apr 28, 2018
@gaelcolas
Copy link

@gaelcolas gaelcolas commented May 2, 2018

Actually, ConvertFrom-Yaml in powershell-yaml uses OrderedDictionary when converting with the -ordered parameter.
I've been using this module successfully for a while (in my Datum module for DSC Configuration data, and with kitchen yamls), but don't have a vsts build definition to test with.

Bear in mind that the right way to call it is: get-content -Raw MyFile.yml | ConvertFrom-Yaml -Ordered (people often miss the -Raw).

I wonder why we'd need a Microsoft official module, putting even more overhead on MSFT and reinventing the wheel... Maybe trying to contribute to an existing one first, add tests to avoid regression, open issues to make sure the owner knows the problems is a better approach...
You know what happens when you're trying to create a standard out of the 99 existing implementations...

And yes it would be better outside the language, I agree that the dependency management could be better, bundling everything in PS is not a solution though.
The broad npm issue is also a failure in process. Fork and re-publish fixed it in no time, building apps out of latest version of the internet was the reason it broke so many live apps.

@markekraus
Copy link
Collaborator

@markekraus markekraus commented May 2, 2018

I agree with @gaelcolas I think this is better with everyone working with the owners of an existing community module to raise and ensure quality.

I'll just add that tests for such a project should include working with a large number of real-world YAML files for things like AppVeyor, Travis CI, VSTS, AWS CloudFormation, etc. For my own experience with YAML deserilization, I have had little success with one solution working universally and have ultimately had to reinvent the wheel several times. In that sense, I agree with @BrucePay "there are just no good YAML parsers".

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented May 2, 2018

We are talking about this platyPS module because it is already actively used in PowerShell Help environment. I guess no one from MSFT can tell how good this module is because of Code of Conduct. They can either silently reject it or improve it.
And although we've been talking about this a long time ago I don't see how we could use the components of this module here in a simple way.
Maybe @adityapatwardhan and @SteveL-MSFT will open their plans and timeline especially as the new Help RFC is already in the experiment stage.

@SteveL-MSFT
Copy link
Member

@SteveL-MSFT SteveL-MSFT commented May 2, 2018

My personal view is that I would rather see more community modules succeed and become de facto standard than requiring "official" modules from Msft.

@markekraus
Copy link
Collaborator

@markekraus markekraus commented May 2, 2018

@iSazonov It is one thing to have a solution that works for serializing/deserializing a well defined schema. It is quite another to have a solution that works in general with all schemas that are compliant YAML.

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented May 2, 2018

I understand the desire of MSFT to reuse community projects to cut costs. But the situation is, in fact, that MSFT may not make use of so many community projects:

  • many have bad code, have no trust
  • many projects are one person

MSFT has published Powershell specifications more 10 years ago, but nobody ported it yet until MSFT did.
The OpenSSL Project has existed for many years but nobody ported it to Windows while MSFT has not done this.
MSFT revealed many thousands of API interfaces, but how many of them were ported to Unix?
The interesting thing about why the company launched its project .Net Core rather than reuses Mono?
PowerShell is already a year and a half is an open source project, but I see that in this repository only one person from the community makes systematic contribution in the code @markekraus and only one person makes systematic analysis @mklement0.
I don't think that if we divide the project into parts, then we got more contributions.
I don't think the situation will change tomorrow. I wouldn't count on it.

@wasker
Copy link

@wasker wasker commented Oct 29, 2020

IMHO, YAML is as popular as JSON and CSV, and not having inbox converters for YAML in PowerShell is kind of sad. Having inbox YAML converters will also ensure that their behavior is on par with JSON converters, which is not the case with community modules.

Don't get me wrong - I appreciate that folks create modules for community, but in the current state of the world, YAML conversion is table stakes - we don't expect folks to download 3rd-party modules for JSON conversion.

@marktoman
Copy link

@marktoman marktoman commented Dec 21, 2020

@dirkslab You could use https://github.com/cloudbase/powershell-yaml

GitHub**cloudbase/powershell-yaml**PowerShell CmdLets for YAML format manipulation. Contribute to cloudbase/powershell-yaml development by creating an account on GitHub.

The problem is that unlike ConvertFrom-Json, this module deserializes to Hashtable instead of a proper object (or PSCustomObject), which doesn't allow using the pipeline in the following way:

$obj | ? Categories -gt 0 | % Name

My current workaround:

'...' | ConvertFrom-Yaml | ConvertTo-Yaml -JsonCompatible | ConvertFrom-Json

Suboptimal, to say the least.

GitHub
PowerShell CmdLets for YAML format manipulation. Contribute to cloudbase/powershell-yaml development by creating an account on GitHub.

@mklement0
Copy link
Contributor

@mklement0 mklement0 commented Dec 21, 2020

@marektoman, your example should work with a hashtable:

PS> @{ one = 1; two = 2 } | ? one -eq 1 | % two
2

Also note that you can cast hashtables to [pscustomobject] (albeit only one by one - see #13838):

PS> @{ one = 1; two = 2 }, @{ one = 10; two = 20 } | % { [pscustomobject] $_ }

one two
--- ---
  1   2
 10  20

@marktoman
Copy link

@marktoman marktoman commented Dec 21, 2020

@marektoman, your example should work with a hashtable:

PS> @{ one = 1; two = 2 } | ? one -eq 1 | % two
2

Also note that you can cast hashtables to [pscustomobject] (albeit only one by one - see #13838):

PS> @{ one = 1; two = 2 }, @{ one = 10; two = 20 } | % { [pscustomobject] $_ }

one two
--- ---
  1   2
 10  20

Good point, the behavior is unrelated to hashtable, but it is likely related to the way each module does output.

When I run

$yamlArray | ConvertFrom-Yaml | ? Name -eq 'All'

, it doesn't work. However, this does:

($yamlArray | ConvertFrom-Yaml) | ? Name -eq 'All'

@dirkslab
Copy link

@dirkslab dirkslab commented Dec 21, 2020

Thanks @marektoman , my solution was to remove the extra yaml module overhead and settle somewhere in between yaml and json.

I did something similar as you when requiring an pscustomobject:
$obj = Get-Content -Path .\deployment-config.yml -raw | ConvertFrom-Yaml | convertto-json -Depth 100 | ConvertFrom-json -Depth 100

The direction I have chosen to go with:
I have replaced all my yaml config with .psd1 and using Import-PowerShellDataFile

Not as clean as yaml, more than sufficient and easy to read though. Also eliminated any and all module issue.

Examples:
Sharing project with other devs that are domain policy restricted on installing random and unsigned modules, not an issue anymore.
I can now . browse path before I convert it to a string
adding comments are easy

My personal goals were to need little to no 3rd party modules for my ps project. I am now back on track and it's hassle free.

@mklement0
Copy link
Contributor

@mklement0 mklement0 commented Dec 21, 2020

@marektoman : The (...) forces enumeration, which in turn suggests that when ConvertFrom-Yaml outputs an array of hashtables that it does so as a single object, which contravenes the expected streaming (one-by-one) pipeline behavior. It is what '[ 1, 2 ]' | ConvertFrom-Json used to do (see #3424), and still does in Windows PowerShell.

The price you pay for the (...) workaround is that all output must be collected in memory first, but I suspect that won't be a problem in practice.

@marktoman
Copy link

@marktoman marktoman commented Dec 22, 2020

@dirkslab Good idea for that use case. My use case is a shell independent tool that produces single output, which is both human and machine readable.

@mklement0 I see, thanks for all the info.

@Vaccano
Copy link

@Vaccano Vaccano commented May 21, 2021

It is absolutely CRAZY that there is not first class support for yaml in PowerShell. I am just stunned.

The Kubernetes movement alone should be more than enough to get this prioritized.

@autosvet
Copy link

@autosvet autosvet commented Jun 5, 2021

We should admit that YAML is used practically in many/most of cloud, CI, configuration, orchestration tools that IT pros, devs and DevOps need to use. So if PowerShell want to stay (most :D) valuable in that world for each it was designed, created and tailored specifically it should support it out of the box.

@dzmitry-lahoda
Copy link

@dzmitry-lahoda dzmitry-lahoda commented Jun 5, 2021

can i donate 10 USD for the feature? I know there are services where donation is done until some dev implements

@Vaccano
Copy link

@Vaccano Vaccano commented Jun 11, 2021

This page talks a lot about the powershell-yaml module as an option to get YAML support.

Does this library work on Linux with Powershell Core? or it just a Windows only thing. (I noticed that its dependencies include .Net Framework, so I am worried.)

@gaelcolas
Copy link

@gaelcolas gaelcolas commented Jun 11, 2021

@Vaccano it works well on Linux.

@jdhitsolutions
Copy link
Collaborator

@jdhitsolutions jdhitsolutions commented Jun 14, 2021

My initial thinking is that YAML support should be standard in PowerShell 7.x. YAML is such an integral language to today's enterprises that its omission is glaring considering support for XML and JSON. I have no problem with a separate module that can also be installed from the gallery, but it should be included by default in a new PowerShell 7.x installation. I know there are organizations that won't consider anything open source, and certainly nothing that doesn't originate or is supported by Microsoft. Given the value of PowerShell as a cloud language, native YAML support seems like the right decision.

I am also less concerned about making the module downlevel compatible. If someone needs YAML conversion support for Windows PowerShell, there appear to be a number of options in the PowerShell Gallery.

@redstrike
Copy link

@redstrike redstrike commented Jul 9, 2021

As a normal user, I've installed PowerShell Core and use it in my work.
Our projects' ps1 scripts are not compatible with PowerShell Core. I have switched to the legacy PowerShell to run them.
And it's difficult to install a new module in this constrained work environment.
I don't know how to write PowerShell scripts, and now I have to live with the old PowerShell.
I feel sad because the new shiny PowerShell Core that I'm in love, is not usable in even a simple real-world usage.
Maybe, I have to rewrite those *.ps1 files by using a serious programming language.

@joeyaiello
Copy link
Contributor

@joeyaiello joeyaiello commented Aug 18, 2021

@PowerShell/wg-powershell-cmdlets has been discussing this issue, and despite the extremely high demand (and my own personal desire to have strong, canonical YAML cmdlets), we're unfortunately going to have to close it (for now) as Won't Fix. The reasoning is two-fold:

  • While the PowerShell package is already fairly large, we're doing our best not to add new dependencies to the package so that we don't continue to grow our size unfettered, and because we believe that pulling modules from the PowerShell Gallery (or an internal trusted Gallery, having been vetted appropriately by an enterprise) should be a standard part of a PS7 workflow.
  • As a team, we simply don't have the capacity to build a .NET YAML parser from scratch, nor do we have the capacity to vet, support, and service a 3rd party library for which the security implications are high-stakes.

If .NET were to officially support a YAML library, we (as Microsoft) could potentially look into supporting a YAML module on the Gallery. But until that point, the existing community modules will have to suffice.

@potatoqualitee
Copy link

@potatoqualitee potatoqualitee commented Aug 19, 2021

Thanks, Joey. I'd like to file a Feature Request in dotnet but don't understand it well enough to know which repo to add the issue. Can you please suggest which repo would be most appropriate within https://github.com/dotnet ?

GitHub
Home of the open source .NET platform. .NET Platform has 201 repositories available. Follow their code on GitHub.

@potatoqualitee
Copy link

@potatoqualitee potatoqualitee commented Aug 19, 2021

(or could a Microsoft person/PowerShell team member place to the request as it may get a higher priority?)

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Aug 19, 2021

@potatoqualitee
Copy link

@potatoqualitee potatoqualitee commented Aug 19, 2021

Thank you @iSazonov. That doesn't look promising :/ Seems that we do need an advocate from the PowerShell team to create the request, eh?

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Aug 19, 2021

I guess Microsoft has internal conclusion that it is low priority and small area.

You can ask PowerShell team online today on Community Call PowerShell/PowerShell-RFC#299

@msftbot
Copy link

@msftbot msftbot bot commented Aug 20, 2021

This issue has been marked as won't fix and has not had any activity for 1 day. It has been closed for housekeeping purposes.

@msftbot msftbot bot closed this Aug 20, 2021
@Vaccano
Copy link

@Vaccano Vaccano commented Aug 20, 2021

This makes me sad.

Powershell says ".Net has to do it for us to do it"
.Net says "It is up to Powershell if they want to do it"

I wonder if some of the higher ups at Microsoft realize the runaround that is going on here.

YAML is important. Having it from a signed Trusted Source is critical for most companies. Powershell has built in restrictions making it hard to use things not signed by trusted sources.

I give up. It seems that the enough dust has been kicked up to obscure this issue. I imagine at some point someone high up at Microsoft will say "What? No YAML support in PowerShell?" Fingers will be pointed, blame laid, excuses made (etc). Then it will get done. But no amount of requests by the community will get this through.

@awakecoding
Copy link
Contributor

@awakecoding awakecoding commented Aug 24, 2021

Why has this been closed as WONTFIX? Seriously, YAML is so common these days, all the cloud-native stuff uses it. One can do so much with just vanilla PowerShell before they need to import a module, and not being to deal with YAML shouldn't be one of them. The built-in JSON cmdlets is one of the reasons why JSON is to popular in PowerShell: it's there, it's standardized, and it works. I am sure I am not alone defaulting to JSON just for that reason, and thinking twice about using YAML if I don't really need it, as it makes my scripts less portable or standalone.

@JustinGrote
Copy link
Contributor

@JustinGrote JustinGrote commented Aug 24, 2021

@awakecoding joey's explanation is here: #3607 (comment)

The Json library, even though it was Newtonsoft, was "blessed" by Microsoft as the official JSON library till system.text.json came around.

It's a bummer for sure, I would love native YAML support as well, but I personally think the reasoning is sound. I think energies should be focused on more easily pulling down dependencies quickly for a script/module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet