Add parameterset to New-Object to accept hashtable and emit PSCustomObject

[TobiasPSP]

Summary of the new feature / enhancement

PowerShell is a object-oriented shell. Programmers are familiar with this concept but admins and occasional scripters often are not. This audience primarily uses simple data types and arrays vs. objects.

By providing a new cmdlet Convert-ObjectToArray, the complexity of generating own objects would be significantly reduced. For example, a user could calculate a number of results, then "wrap" them easily into an object and return structured info from own functions:

$date = Get-Date
$lottery = 1..49 | Get-Random -Count 7
$env:username, $date, $lottery | Convert-ArrayToObject -PropertyName UserName, Drawing, Result

Result:

UserName Drawing             Result             
-------- -------             ------             
tobia    30.07.2021 10:07:00 {10, 48, 42, 11...}

Experienced scripters would benefit as well. Often system calls return unstructured array information. With Convert-ArrayToObject, it would intuitively be possible to impose a structure on that data.

The following example queries the eventlog for the newest five installed updates. The actual update names and other event information is stored inside "Properties" as a simple array.

By using Convert-ArrayToObject, it would be trivial to impose a meaningful object structure to this array and then emit the result as object:

$filter = @{
    LogName = 'System'
    Level = 4,5
    Id = 19
    ProviderName = 'Microsoft-Windows-WindowsUpdateClient'
}

Get-WinEvent -FilterHashtable $filter -MaxEvents 5 | 
  Select-Object -Property TimeCreated, @{N='Values'; E={ $_.Properties.Value | Convert-ArrayToObject -PropertyName Software,GUID,Code  -DiscardRemaining }} |
  Select-Object -Property TimeCreated -ExpandProperty Values

Result:

Software                                                                                         GUID                                 Code TimeCre
                                                                                                                                           ated   
--------                                                                                         ----                                 ---- -------
9NMPJ99VJBWV-Microsoft.YourPhone                                                                 892a7e42-d65a-4114-bb83-516048385d2e    1 30.0...
9WZDNCRDTBJJ-MICROSOFT.GETSTARTED                                                                e70f94da-7a66-4183-92c3-913affb4be81    1 30.0...
9MSPC6MP8FM4-Microsoft.Whiteboard                                                                4618c3a8-72c7-4c6a-93fb-52a0a5bbd1b4    1 30.0...
Security Intelligence-Update für Microsoft Defender Antivirus - KB2267602 (Version 1.343.1905.0) c809e399-1a7a-427d-8744-75b8005c5778  200 30.0...
Security Intelligence-Update für Microsoft Defender Antivirus - KB2267602 (Version 1.343.1769.0) 6f55ed2a-65fc-4499-8439-9a38d86b9189  200 28.0...

Proposed technical implementation details (optional)

Convert-ArrayToObject is no rocket sience but it uses technology not commonly known by the admin audience:

function Convert-ArrayToObject
{
    param
    (
        [Object[]]
        [Parameter(Mandatory,ValueFromPipeline)]
        $InputObject,
  
        [String[]]
        [Parameter(Mandatory)]
        $PropertyName,
        
        [Switch]
        $DiscardRemaining,
        
        [Switch]
        $AsHashtable
    )
  
    begin 
    { 
        $i = 0 
        $overflow = $false
        $maxPropertyIndex = $PropertyName.Count 
        $hashtable = [Ordered]@{}
    }
    process
    {
        $InputObject | ForEach-Object {
            if ($i -lt $maxPropertyIndex -and -not $overflow)
            {
                $hashtable[$PropertyName[$i]] = $_
                $i++
            }
            else
            {
                if ($DiscardRemaining)
                {
                    $overflow = $true
                }
                else
                {
                    if (-not $overflow)
                    {
                        $i--
                        $hashtable[$PropertyName[$i]] = [System.Collections.ArrayList]@($hashtable[$PropertyName[$i]])
                        $overflow = $true
                    }
                    $null = $hashtable[$PropertyName[$i]].Add($_)
                }
            }
        }
    }
    end
    {
        if ($AsHashtable)
        {
            return $hashtable
        }
        else
        {
            [PSCustomObject]$hashtable
        }
    }
}

It takes an array and a list of property names and turns the array into an object:

PS> 1..3 | Convert-ArrayToObject -PropertyName A,B,C

A B C
- - -
1 2 3

When there are not enough property names, all remaining items go into the last property (similar to how variable assignments work):

PS> 1..10 | Convert-ArrayToObject -PropertyName A,B,C

A B C              
- - -              
1 2 {3, 4, 5, 6...}

The switch -DiscardRemaining in contrast omits overflow items:

PS> 1..10 | Convert-ArrayToObject -PropertyName A,B,C -DiscardRemaining

A B C
- - -
1 2 3

On request, the switch AsHashtable returns the composed hashtable without converting it to an object:

PS> 1..10 | Convert-ArrayToObject -PropertyName A,B,C -DiscardRemaining -AsHashtable

Name                           Value                                                                                                              
----                           -----                                                                                                              
A                              1                                                                                                                  
B                              2                                                                                                                  
C                              3 

Note

This cmdlet could obviously be created and maintained in the PowerShell Gallery.

However it is just a generic and useful item like i.e. the new ternary operator. It makes sense only when it is part of the shipping language set and does not add additional module dependencies.

The function does not add dependencies to PowerShell and won't enlarge the package. It simply shields the complexity of object generation and hashtables from an entry-level audience and provides experienced scripters with a convenient way of performing every-day tasks.

During code reviews, it turned out that even experienced users often use Add-Member to compose objects. Convert-ArrayToObject could provide a more appropriate alternative in most of these cases as well.

Further Thoughts

Ideally, Convert-ArrayToObject would be accompanied by a corresponding Convert-ObjectToHashtable in an effort to better support the conversion of simple data to objects and vice versa:

function Convert-ObjectToHashtable
{
    <#
            .SYNOPSIS
            Turns an object into a hashtable.

            .DESCRIPTION
            When an object is turned into a hashtable, it displays one property per line
            which can be better when you want to output the data in a gridview.

            .PARAMETER ParentObject
            The object you want to convert

            .PARAMETER ExcludeEmpty
            Exclude empty properties.

            .PARAMETER SortProperty
            Sort property names

            .PARAMETER FlattenArray
            Convert arrays to strings

            .PARAMETER ArrayDelimiter
            Delimiter used for flattening arrays. Default is comma.

            .PARAMETER ReturnAsNewObject
            Returns the hashtable as object again with all the requested manipulations in place

            .EXAMPLE
            Get-ComputerInfo | Convert-ObjectToHashtable -ExcludeEmpty -SortProperty | Out-GridView
            Get computer info and show each property in an individual line in gridview.
            Remove Convert-ObjectToHashTable to see the difference.
    #>


    param
    (
        [Parameter(Mandatory,ValueFromPipeline)]
        [Object]
        $ParentObject,
        
        [Switch]
        $ExcludeEmpty,
        
        [Switch]
        $SortProperty,
        
        [Switch]
        $FlattenArray,
        
        [string]
        $ArrayDelimiter = ',',
        
        [Switch]
        $ReturnAsNewObject
    )
    
    process
    {
        $properties = $ParentObject.PSObject.Properties | 
            Where-Object { !$ExcludeEmpty -or ![String]::IsNullOrWhiteSpace($_.Value) } 
            
        if ($SortProperty)
        {
            $properties = $properties | Sort-Object -Property Name
        }
        
           
        $hashtable = [Ordered]@{}
        foreach($property in $properties)
        {
            if ($property.Value -is [Array] -and $FlattenArray)
            {
                $hashtable.Add($property.Name, $property.Value -join $ArrayDelimiter)
            }
            else
            {
                $hashtable.Add($property.Name, $property.Value)
            }
        }
        
        if ($ReturnAsNewObject)
        {
            [PSCustomObject]$hashtable
        }
        else
        {
            $hashtable
        }
    }
}

The use case of the latter function would be a plethora of object manipulation, i.e. the exclusion of empty properties, sorting object properties, and displaying object properties line by line:

Get-Service -Name Spooler | Convert-ObjectToHashtable -ExcludeEmpty -SortProperty | Out-GridView

For reporting purposes, this could be used to sanitize objects: exclude empty properties, sort property names and work around the issue that CSV and excel do not support arrays by automatically flatten arrays, getting rid of those properties that always contain just type names.

-ReturnAsObject would then return the sanitized hashtable as object, ready to be sent to Export-Csv or Export-Excel:

#requires -Version 3.0 -Modules ImportExcel

Get-Service | Convert-ObjectToHashtable -SortProperty -FlattenArray -ArrayDelimiter ' ' -ReturnAsNewObject |
    Export-Excel

This example creates an excel report and correctly shows dependent services that usually would not show in a readable way. To run this example, you need to install the module ImportExcel or replace Export-Excel by a default cmdlet such as Export-Csv:

Install-Module -Name ImportExcel -Scope CurrentUser 

Caveat: removing empty properties via -ExcludeEmpty is safe only when processing a single object which is why the previous example isn't removing empty properties. In the context of a single object, though, automatically removing empty properties can be very useful, i.e. to focus on ActiveDirectory attributes for a particular user that are actually set:

Get-ADUser -Identity $env:USERNAME -Properties * | 
  Convert-ObjectToHashtable -ExcludeEmpty -SortProperty | 
  Out-GridView

If you process multiple input objects, removing empty properties could lead to objects with less hashtable keys than others. While this isn't bad a priori, when outputting these objects, PowerShell would show only the properties present in the first object it encounters (as always).

[iSazonov]

Looks to tricky being used broadly. It seems using PS classes is more simple and readable.

[dennisl68-castra]

Why not just store the data in a PSCustomObject in the first place?

[jdhitsolutions]

My initial thought was to simply use [pscustomobject]. But I think what I think Tobias is going after is the less scripting proficient PowerShell user who wants to do this on the fly in the console. Although I don't see what is that difficult with this syntax, once you learn it of course.

$date = Get-Date
$lottery = 1..49 | Get-Random -Count 7
[pscustomobject]@{Username = $env:username; Drawing= $date;Result = $lottery}

[TobiasPSP]

Looks to tricky being used broadly. It seems using PS classes is more simple and readable.

jdhitsolutions got the idea. Classes are natural to programmers, not ITPro. PowerShell was not designed to be yet another programming language. It was invented to be a practical scripting language with a low entry point for ITPros. They need a simple way to wrap information into objects. Classes are way out of scope to this audience.

That said, I am more than open and curious for other approaches as long as they do the job. Can you (iSazonov) elaborate how you think classes could help here, and how the suggested approach is trickier than using classes?

Please consider that we are not in a situation where we own software projects and can design classes and constructors. In the intended target group you have to deal with raw information from a number of sources that are beyond own control, and need a way to wrap them into objects without knowing more than the most basic programming principles.

[dennisl68-castra]

PowerShell already support using and creating classes out of the box.

about_Classes

[TobiasPSP]

Why not just store the data in a PSCustomObject in the first place?

That's a great comment. Most advanced scripters do. But as code review shows, even advanced programmers frequently resort to inappropriate ways like using Add-Member to create simple data objects. Entry-level and occasional scripters, which are the primary target group for this suggestion, almost never use [PSCustomObject] simply because they don't know or care much about types and hashtables.

Basically, anything can be done in PowerShell via direct .NET and types. The ease of use of PowerShell however comes from generic cmdlets which by design shield .NET from the user. So one could argue that we could get rid of all cmdlets since the same can always be done using underlying .NET ;-). So I guess it's a matter of evaluating whether a given cmdlet is simple and reusable enough to be of value, not whether it is unique in what it does.

[dennisl68-castra]

[PSCustomObjects] isn't very hard.
It just needs to be documented properly in various places (i.e. help Add-Member etc.).
Several years passed before I encountered
$result = New-Object psobject | select Property1, Property2, Property3

[TobiasPSP]

Although I don't see what is that difficult with this syntax, once you learn it of course.

I guess it is a question of how deep you want users to dive into programming. I like to view PowerShell as a video game with different levels. In level 1 you learn how to discover and run individual commands, and in level 2 you learn how to couple multiple commands using pipeline and variables. These two levels are where most ITPro live that I encounter in the IT departments I visit every week, so to help them, new tools need to be in these levels.

Level 3 would involve hashtables and types, so yes you can argue that users should be at least educated to level 3 to make use of structured output. However my experience is that level 3 is a barrier for many because it also marks the transition from batch scripters to programmers.

I am not hurt in any way if it turned out I am the only one thinking this way ;-). I do have the feeling though that PowerShell is moving away from its original user base, and I suspect this is because most people actively involved in its development seem to be developers or experienced scripters these days.

[SeeminglyScience]

I agree that there should be more options for transforming input into objects. I do worry that this particular design would lead to a lot of enumeration related confusion, especially for the type of folks this would be targeting.

Either way since this can be done at the command level it would be good to prove the concept with a gallery module like was done with PSThreadJob.

[iRon7]

This request has quiet some similarities with #13817 Enhance hash table syntax
and the suggested Join-Object in issue: #14994 Add a Join-Object cmdlet to the standard PowerShell equipment (including the linked issues within)

$env:username |Join $date |Join (,$Lottery) -Name UserName, Drawing, Result
UserName Drawing              Result
-------- -------              ------
user     7/30/2021 8:13:03 PM {2, 30, 4, 5…}

Note the unary comma for the $lottery value as you probably also want to be able to a side-by-side paste of arrays:

$env:username |FullJoin @(1..7) |Join $Lottery -Name UserName, Index, Drawing
UserName Index Drawing
-------- ----- -------
user         1       2
             2      30
             3       4
             4       5
             5      26
             6      12
             7      32

What they all (including the linked cases) have in common, is the search for a syntactic way to transpose data: move rows to columns (objects to properties) and/or vice versa.
Saying that, I think if you would just change the name of you purposed cmdlet to ConvertTo-Property it will have a more general perspective. Also knowing that you actually not converting an array but convert each (singular) $InputObject to a property.

[rw-hardin]

I use Powershell daily, both in complex scripts/automation and in one-line commands.....and everywhere in between. Personally, I think this is amazing. There have been a ton of times where I needed to put together some data that would only be used once or twice (making writing some advanced script take more time than just doing it all by hand), as well as converting objects into a decent output to be consumed by another cmdlet or even just dumped out to a file.

Both of the functions in this suggestion would definitely be time-saving for ITPros at all levels. Those of us that taught ourselves how to use classes and complexity in Powershell, as well as those who have no idea how to do more than a single one-liner.

[dennisl68-castra]

I really doesn't see the issue.
The example is quite doable as is in PowerShell

$date = Get-Date
$lottery = 1..49 | Get-Random -Count 7
$drawing = New-Object pscustomobject | select UserName, Drawing, Result

$drawing.UserName = $env:USERNAME
$drawing.Drawing = $date
$drawing.Result = $lottery

PS C:\> $drawing

UserName Drawing             Result
-------- -------             ------
dennis   2021-07-30 23:36:43 {40, 4, 46, 14…}

[rw-hardin]

Still more complex than what was suggested. What would be the downside to including it as an enhancement?

[rw-hardin]

I do have the feeling though that PowerShell is moving away from its original user base, and I suspect this is because most people actively involved in its development seem to be developers or experienced scripters these days.

@TobiasPSP I 100% agree, I love the complexity that has been added over the years, because Powershell can now do so much more than when I first touched it way back....but if something like this sort of suggestion is scoffed at...

I was actually relieved when I saw the simplicity of this suggestion. Yes, we can create custom objects, or use the code in the function itself if we wanted to. But I've found myself trying to combine different arrays or hashtables into a single object (usually for a report or some such) before, and it always involves spending time coding/testing/retesting/etc. I know many ops people that would love to have this so they can get those reports for the audit guys breathing down their necks.

Just my 2 cents. Not worth terribly much, of course.

[dennisl68-castra]

Ok, now I see the issue. But in a wider perspective.

The example provided isn't very complicated if hitting [pscustomobject] to begin with.
But if you consider you actually need to transpose data between objects, then I think what we are actually looking for is the Join-Object suggestion.

To wrap my head around the actual data conversion I tried out a oneliner.
(I'm still using a psobject on the receiving end, not a hashtable)
It got kind a' ugly ;)

# Create the objects
$name = $ENV:USERNAME
$date = Get-Date
$lottery = 1..49 | Get-Random -Count 7

$drawing = New-Object pscustomobject | Select Username, Drawing, Result

# Transpose in one line
$nul,@($name,$date,$lottery) | #need a multidimensional array to get the correct count to carry over 
    foreach {
        $pipecount = $_.count; 
        for ($i=0; $i -lt $pipecount; $i++){
            ($drawing.psobject.properties | select -index $i).value = $_[$i]
        }
    }

# Output the result
$drawing

Username Drawing             Result
-------- -------             ------
sven     2021-07-31 02:25:03 {10, 39, 25, 18…}

[Paxxi]

I like this idea and I think it would be great for all experience levels. Having a cmdlet for it means it's more discoverable and more natural to use, especially in oneliners.

The naming is good from a discoverability perspective as it will be easy to find on your favorite search engine. It does feel "wrong" when looking at the current cmdlets and their naming as it doesn't follow the verb-noun style.

How about naming it ConvertTo-Object to fit into the naming style? It would make it natural to also have a ConvertFrom-Object that does the opposite transformation. The downside is that it might be less discoverable on the name alone.

function ConvertTo-Object
{
    param
    (
        [Object[]]
        [Parameter(Mandatory,ValueFromPipeline, ParameterSetName="Array")]
        $InputArray,
  
        [String[]]
        [Parameter(Mandatory, ParameterSetName="Array")]
        $PropertyName,
        
        [Switch]
        [Parameter(ParameterSetName="Array")]
        $DiscardRemaining,
        
        [Switch]
        [Parameter(ParameterSetName="Array")]
        $AsHashtable,

        [HashTable]
        [Parameter(Mandatory,ValueFromPipeline, ParameterSetName="HashTable")]
        $InputHashTable,

        [System.Collections.Specialized.OrderedDictionary]
        [Parameter(Mandatory,ValueFromPipeline, ParameterSetName="OrderedHashTable")]
        $InputOrdered,

        [String[]]
        [Parameter(Mandatory=$false, ParameterSetName="OrderedHashTable")]
        $PropertyName,
    )
...
}

I don't know if that design is technically feasible but it should get the idea across at least. The PropertyName parameter for OrderedDictionary would be to allow renaming properties. I don't think it would be sensible to have that for a HashTable as it would be unpredictable and confusing.

A ConvertFrom-Object could look something like this

function ConvertFrom-Object
{
    param
    (
        [Object]
        [Parameter(Mandatory,ValueFromPipeline)]
        $InputObject,
        
        [Switch]
        [Parameter(ParameterSetName="HashTable")]
        $AsHashtable,

        [Switch]
        [Parameter(ParameterSetName="Array")]
        $AsArray
    )
...
}

[TobiasPSP]

Thank you all so much for sharing thoughts. I am trying to sum a number of things up. Let me know if I missed anything.

Proposal

Add one or more cmdlets to expose object manipulation on discoverable cmdlet level rather than having to use .NET, type conversion, or classes:

• Convert unstructured data (arrays) to objects/hashtables
• Convert hashtables to objects
• Convert objects to hashtables
• Merge objects

Why is that needed?

Occasional scripters work on cmdlet level and typically have no deep knowledge about types, hashtables, let alone classes. Even experienced scripters seem to often use Add-Member to compose simple data objects (rather than using [PSCustomObject] or classes) - likely because Add-Member is the only discoverable cmdlet in PowerShell that creates new objects (aside from Select-Object which isn't really appropriate in this context).

There is currently no intuitive self-learning path to go beyond Add-Member on cmdlet level.

All other techniques for object creation - however simple they may be - require extra knowledge, i.e. about hashtables or classes, which is an extra barrier for novice users still trying to cope with cmdlets and parameters.

Experienced users can benefit from these cmdlets, too, as their feature set spans a very powerful range and allows for advanced object manipulation (i.e. sorting properties, removing empty values, merging objects).

Is this functionality really needed?

Anything the proposed cmdlets do can be done via direct .NET access (type conversion, classes). So once you know how to create new objects via [PSCustomObject] or via classes, it is not much more complicated than using the proposed cmdlets.

Yet this proposal was not about teaching PowerShell new tricks in the first place. It is a proposal to lower the barrier for entry-level or occational scripters, and provide generic solutions for common object manipulation tasks.

This approach is in good tradition with existing cmdlet design. Cmdlets often exist just to be discoverable or shield the user from .NET. Compare Get-Alias with Get-ChildItem -Path alias:, same with Get-Variable. Look at Get-Date: [DateTime]::Now isn't hard, either. 'Get-Date -Format yyyyMMdd' could as well be replaced by [DateTime]::Now.ToString('yyyyMMdd'). Get-Process could be replaced by equally simple [System.Diagnostics.Process]::GetProcesses(), etc. pp.

And still there is consensus that it is a great thing that Get-Date, Get-Process and other cmdlets exist even though they don't really do anything new. Cmdlets exist to be discoverable and versatile.

How expensive is it?

Implenting the proposed cmdlets is trivial from an engineering point of view. There are no dependencies nor would the package grow. All proof-of-concept functions in this thread are short functions which essentially just wrap a discoverable user interface around the existing PowerShell capabilities.

Next steps

I'd suggest to collect the functionality commonly needed to convert unstructured data to structured data and vice versa. Once we know what the feature set should be, we can start composing a cmdlet family and think about naming.

Prerequisites

A prototype module can be developed and hosted on github and the gallery so everyone was invited to add ideas or participate in dev or testing.

Prereq would be that we get a general thumbs up from the PowerShell working group that ultimately this module can become part of the shipping modules. The proposed cmdlets make no sense if they stay put in a 3rd party module since the target group isn't familiar with the gallery and shouldn't be discriminated in a way that their scripts would only run with a 3rd party helper module being installed.

[SteveL-MSFT]

@PowerShell/wg-powershell-cmdlets discussed this and agreed with the scenario of making custom object creation easier for new users particularly from a discovery point of view. We propose a change to New-Object parameter set that takes a hashtable and emit a PSCustomObject. This is more discoverable than casting a hashtable to a [PSCustomObject] for new users. A -TypeName taking an array of strings should also be considered for custom typing (used with formatting). This should be a bucket 3 breaking change where currently a hashtable doesn't emit something useful used with New-Object

[IISResetMe]

Pardon my ignorance but... doesn't that already exist @SteveL-MSFT?

PS ~> New-Object -Property @{ A = 1; B = 2; C = 3 } -TypeName PSCustomObject

A B C
- - -
1 2 3

[SeeminglyScience]

It does also support PSTypeName, e.g.

New-Object psobject -Property @{
    PSTypeName = 'CountingInfo'
    A = 1
    B = 2
    C = 3
}

Though if you need property order to be retained you do need ([ordered]@{ ... }).

[TobiasPSP]

Pardon my ignorance but... doesn't that already exist @SteveL-MSFT?

PS ~> New-Object -Property @{ A = 1; B = 2; C = 3 } -TypeName PSCustomObject

A B C
- - -
1 2 3

You are absolutely right, the original issue title is a bit misleading, and initially this was about a new cmdlet family for a broad range of data-to-object manipulation (and vice versa). In the course of debate here, a number of low hanging fruit were identified that could easily be integrated into the existing New-Object cmdlet.

[iSazonov]

In the course of debate here, a number of low hanging fruit were identified that could easily be integrated into the existing New-Object cmdlet.

Maybe open new issue with clear proposal?

[IISResetMe]

Thanks for the clarification @TobiasPSP.

I think it makes perfect sense to

1. Add pipeline support for the -Property dictionary so you could do:

$data = @(
  [ordered]@{ A = 1; B = 2; C = 3 }
  [ordered]@{ A = 1; B = 3; C = 5 }
  [ordered]@{ A = 1; B = 4; C = 7 }
)

$objects = $data |New-Object

2. Make -TypeName default to PSCustomObject when absent

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.