Add Parameter to ConvertTo-Json to ignore unsupported properties

[mike-the-automator]

I'd like to propose an additional parameter for the ConvertTo-Json cmdlet that specifies the caller would like the cmdlet to ignore properties of the InputObject that cannot be converted to JSON successfully. Currently, the behavior of the cmdlet is to throw an exception with a message similar to the following

"ConvertTo-Json : The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or "deserialization of a dictionary. Keys must be strings.

My proposal is that if the -SkipIncompatibleProperties (parm name feedback appreciated :-) ) parameter is specified, then the cmdlet would silently skip over these properties and not throw any exceptions.

I've been frustrated by this error several times in my scripting in situations where I don't really care about the particular property that triggered the error; I would just like a quick and easy serialized representation of the object. In the most recent instance, I needed a way to record exception info in a VARCHAR column in SQL Server. The property of the System.Exception that triggered this error was the "Data" field which I didn't really care about, but it prevented me from easily using the cmdlet to serialize the exception object.

As an aside, it looks like there is a proposal by @KirkMunro to move the JSON cmdlets to their own module. I understand it may make sense to wait on this feature until the module split occurs.

[bergmeister]

@mike-the-automator Does -ErrorAction SilentlyContinue solve your specific case(s)? If not then it might be the better solution to rather tweak the error behaviour instead of adding a new parameter.

[mike-the-automator]

Thanks for your response. I tested -ErrorAction SilentlyContinue and that did not seem to solve my problem.

Transcript:

PowerShell v6.0.0-rc.2
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

PS C:\Program Files\PowerShell6RC2\6.0.0-rc.2> 1/0
Attempted to divide by zero.
At line:1 char:1
+ 1/0
+ ~~~
+ CategoryInfo          : NotSpecified: (:) [], RuntimeException
+ FullyQualifiedErrorId : RuntimeException

PS C:\Program Files\PowerShell6RC2\6.0.0-rc.2> $j = ConvertTo-Json $Error[0] -ErrorAction SilentlyContinue
ConvertTo-Json : The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
At line:1 char:6
+ $j = ConvertTo-Json $Error[0] -ErrorAction SilentlyContinue
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (System.Collecti...tionaryInternal:ListDictionaryInternal) [ConvertTo-Json], InvalidOperationException
+ FullyQualifiedErrorId : NonStringKeyInDictionary,Microsoft.PowerShell.Commands.ConvertToJsonCommand

PS C:\Program Files\PowerShell6RC2\6.0.0-rc.2> $j -eq $null
True

In the scenario I envision, when the new parameter is supplied there would be no error thrown and the value of the variable $j would be a JSON representation of the exception objects, ignoring any fields that would have triggered the InvalidOperation exception.

[mike-the-automator]

@bergmeister I have mulled over the suggestion to just change the behavior for -ErrorAction SilentlyContinue and my impression is that would be a counter intuitive way to achieve my goal. It would be a breaking change for users because a scenario that used to generate an Exception would no longer do so.

Also, silently discarding data without being explicitly told to do so feels wrong. It almost feel more like an occasion to use -Force as more of a "No, really this is what I want to do" kind of parameter.

Another means to get the result I want would be to just create a cmdlet to strip non-JSON compliant fields of an object that would result in this error and put it in front of ConvertTo-Json in the pipeline, e.g.

$j = $Error[0] | Remove-JsonIncompatibleProperties | ConvertTo-Json

I could always create a convenience wrapper-function to eliminate the need for a long pipeline (I suck at naming things). Given that perspective I can understand if the consensus is that the complexity/utility ratio doesn't pan out for this feature request, but I'd like to know what others think.

[FireInWinter]

I think it is a separate issue that ErrorAction is ignored and that it still raises an exception, but I think the correct functionality of ErrorAction with SilentlyContinue should be to return no data, not to strip out some data.

I think either Force or SkipIncompatibleProperties would be a better solution as you are explicitly telling it to get rid of data.

[mike-the-automator]

@FireInWinter Created a separate issue as requested. I think we are in agreement that the behavior requested in this issue should be implemented as a new parameter for the reasons already stated.

I'm interested in contributing and would be happy to work on a PR for either issue, but I'd rather not invest time in this feature request if it has a low probability of being merged. I'm not asking for a guarantee that my PR will be accepted, but it would be nice to know if I'm barking up the wrong tree here.

[bergmeister]

Can a maintainer please comment on the design/issue. @iSazonov @lzybkr ?

[SteveL-MSFT]

My preference would be to have a -SkipUnsupportedTypes switch. -Force is too overloaded to me. cc @joeyaiello @HemantMahawar for suggestions on the name of the switch.

[mklement0]

Generally, the use case you're describing is better handled using serialization based on CLIXML rather than JSON.

What makes this more cumbersome is the fact that you must export to a (temporary) file first, using Export-CliXml, and also read from a file, using Import-CliXml; however, a new pair of ConvertTo-CliXml / ConvertFrom-CliXml cmdlets that directly output / read the XML as a strings are being worked on - see #3898

[SteveL-MSFT]

@mklement0 agree that if the use case is to serialize and deserialize later without care of the serialization format, CliXml is a better solution. However, I can see cases where you want specifically json (more human readable than xml...)

[mklement0]

@SteveL-MSFT:

I merely restricted my comment to pointing out the CLIXML alternative, because I don't know what the right answer to the specific problem at hand is, but here are a few thoughts:

A simpler way to reproduce the problem, using a hashtable with a non-string key:

@{ 1 = 'one' }  | ConvertTo-Json

The error message suggests that an explicit decision was made not to support non-string keys and to fail in that event.

The alternatives - to be activated via new switch(es) - are:

• simply apply .ToString() to non-string keys and serialize nonetheless

• as suggested, skip unserializable properties.

Given that JSON is not suitable for general-purpose serialization, however, I wonder if this is really necessary.
Generally, you will know what properties should be serialized ahead of time, and can filter out unsupported properties then, in the simplest case with Select-Object -Property ...

[iSazonov]

I wonder if this is really necessary.

I wonder too.
JSON is well standardized. I would have looked at what this standard says.

[markekraus]

I wonder if this is really necessary.

Let's not let convenience become irrelevant in this project.

I have use cases for this myself that I have had to work around. I have object A that has nested property B that cannot be serialized, but I need to upload this to a generalized logging endpoint. that expects a JSON object. I don't care about property B even in the context of PowerShell or .NET. I have to create Object C that has everything but property B and then I can serialize it. Or I have to use a half dozen other work arounds in this scenario including creatining my own JSON serialize or a proxy class.

[SteveL-MSFT]

This reminds me that in my HttpListener module, I had to work around JSON serialization limitations by manually stripping types that were not supported.

[B-Art]

I would suggest a function which converts all none-string values to string values before piping it to convertto-json.
You can put a select statement in between for your final json output.

@{ 1 = 'one' }.getenumerator() | select key,value | ConvertTo-Json
Works....

[mklement0]

@B-Art:

This issue is about ignoring values that can't be converted to JSON.

By contrast, you're looking to for an opt-in method for including such values by converting dictionary keys to strings, so I suggest you create a new issue with your proposal.

As an aside, re your example: It isn't the select call that makes the command succeed, it is using .GetEnumerator(), because the latter sends a [System.Collections.DictionaryEntry] instance (rather than the whole hashtable) through the pipeline, which is itself not a dictionary, so the to-JSON conversion succeeds - however, that is not the same as what @{ '1' = 'one' } | ConvertTo-Json would give you, i.e., the hashtable with stringified keys.

[heinrich-ulbricht]

Greeting from 5 years later where I need this functionality in a PowerShell Azure Function.

Specifically I'm trying to convert the output of "Get-Error" to JSON to push it to a logging endpoint. Fails with ConvertTo-Json: The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings..

[jelleholtkamp]

Would like to see this functionality as well. Need it in an Azure Pipeline for a similar scenario as @heinrich-ulbricht, to convert an error record to JSON for consumption further in the pipeline.

[B-Art]

1/0
$Error[0].InvocationInfo | ConvertTo-Json
gives:
{ "MyCommand": null, "BoundParameters": {}, "UnboundArguments": [], "ScriptLineNumber": 1, "OffsetInLine": 1, "HistoryId": -1, "ScriptName": "", "Line": "1/0", "PositionMessage": "At line:1 char:1\r\n+ 1/0\r\n+ ~~~", "PSScriptRoot": "", "PSCommandPath": null, "InvocationName": "", "PipelineLength": 0, "PipelinePosition": 0, "ExpectingInput": false, "CommandOrigin": 1, "DisplayScriptPosition": null }
AND
$Error[0].ToString() | ConvertTo-Json
"Attempted to divide by zero."
OR
$Error[0].Exception.InnerException | Convertto-Json
{ "TargetSite": null, "Message": "Attempted to divide by zero.", "Data": {}, "InnerException": null, "HelpLink": null, "Source": null, "HResult": -2147352558, "StackTrace": null }

[DougChandler]

+1 for a parameter to -SkipUnserializableTypes. I am trying to serialize an exception to send to Log Analytics for further investigation and hitting the exception thrown by Convertto-json.

[iRon7]

@mike-the-automator,

The property of the System.Exception that triggered this error was the "Data" field which I didn't really care about, but it prevented me from easily using the cmdlet to serialize the exception object.

I understand the need of a solution for the fact that a single item/property causes the whole ConvertTo-Json cmdlet to fail but I rather than a quick and dirty workaround, I would prefer it to (optionally) serialize as expected and convert keys that are not strings, like e.g. an integers (or anything else?) to type cast to a string (`"$Key") considering that PowerShell is a loosely language and this does already works for:

[PSCustomObject]@{ 1 = 'one' }  | ConvertTo-Json
{
  "1": "one"
}

And would of cause fail for something like:

[PSCustomObject]@{ 1 = 'one'; '1' = 'one2' }  | ConvertTo-Json
ParserError:
Line |
   1 |  [PSCustomObject]@{ 1 = 'one'; '1' = 'one2' }  | ConvertTo-Json
     |                                ~~~
     | Duplicate keys '1' are not allowed in hash literals.

but that is the actual error I would like to be able suppress...

[B-Art]

Something strange happens the other way around:
'{ 1: "one", "1": "one2" }' | Convertfrom-Json
Will give:

1
-
one2

Or:
'{ 1: "one", "1": "one2" }' | Convertfrom-Json -AsHashtable
Will give:

Name                           Value
----                           -----
1                              one2

So the first key will be overwritten by the second. No errors...?

I would prefer an error, because the JSON is incorrect and the conversion should throw an error?
Or the key already exists and there cannot be another key with the same name?

I did not expect this without an error.

Another experiment:

(" 1 = one `n '1' = one2 " | ConvertFrom-StringData).GetType()

Gives:

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Hashtable                                System.Object

And:

(" 1 = one `n '1' = one2 " | ConvertFrom-StringData)

Gives:

Name                           Value
----                           -----
1                              one
'1'                            one2

Converting it to JSON:

(" 1 = one `n '1' = one2 " | ConvertFrom-StringData) | ConvertTo-Json

Gives 2 Key Value pairs

{
  "1": "one",
  "'1'": "one2"
}

And this works too:

[PSCustomObject]@{ 1 = 'one'; "'1'" = 'one2' }  | ConvertTo-Json

Gives:

{
  "1": "one",
  "'1'": "one2"
}

[sean-sauve]

Has anyone asked why ConvertTo-Json considers a ListDictionaryInternal as invalid? It's entirely possible to convert the keys in a ListDictionaryInternal to JSON if GetEnumerator() is run on it first.

Transcript:

PS>1 / 0
RuntimeException: Attempted to divide by zero.
PS>$e = $error[0]
PS>$e

RuntimeException: Attempted to divide by zero.
PS>$e | ConvertTo-Json
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.

TerminatingError(ConvertTo-Json): "The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings."

ConvertTo-Json: The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
PS>$e.Exception.Data | ConvertTo-Json

TerminatingError(ConvertTo-Json): "The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings."

ConvertTo-Json: The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings.

PS>$e.Exception.Data.GetEnumerator() | % { $_ | ConvertTo-Json -Depth 2 } | Out-Null
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.

[B-Art]

This will give some insight and possibly a solution:
https://blog.ironmansoftware.com/convertto-json-memory/