Powershell durable functions: cannot pass complex output between activity functions

[jformacek]

Description

Consider the following code in orchestration function:

$Data = $Context.Input
$result = Invoke-DurableActivity -FunctionName 'Test1' -Input $Data
Invoke-DurableActivity -FunctionName 'Test2' -Input $result

Activity function Test1 takes list of Azure management groups and returns it to output (just a sample to demonstrate)
Activity function Test2 is expected to take the list and process it

#Test1
param($name)

$groups = Get-AzManagementGroup
return $groups

#Test2
param($name)

foreach($group in $name)
{
    write-host $group.DisplayName
}

Code in Test2 however does not get list of Management groups, but rather list of objects of System.Colections.Hashtable, that is not possible to work with (DisplayName or other props of Management group not propagated from the objects)

List of Management groups is serializable to JSON, so I would expect that it can be passed between activity functions as needed, as long as it can be serialized.

Expected behavior

Complex output of activity function shall be usable as input in chained activity function

Actual behavior

Complex output of activity function is not usable in chained activity function

Relevant source code snippets

see above

Known workarounds

so far only simple objects seem to be able to be passed between activity functions. Workaround is to store complex objects outside of function, e.g. in storage blob.

App Details

  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  },

[cgillum]

Adding @AnatoliB for FYI

[AnatoliB]

@jformacek When you say "list of objects of System.Colections.Hashtable", do you mean a list of strings containing "System.Colections.Hashtable"? In this case, one more workaround would be to take control of serialization, increase the serialization depth, and pass the JSON string to Invoke-DurableActivity, for example:

Invoke-DurableActivity -FunctionName 'Test2' -Input ($result | ConvertTo-Json -Depth 5)

However, if you are actually talking about objects of System.Colections.Hashtable type, and the required property values are not in the hashtable, please let us know.

[jformacek]

Hello,
this seems to help - one just has to realize that all input passed to to activity functions is actually automatically deserialized to hashtable/array of hashtables, and that default serialization depth provided by runtime (it's 4, right?) may not be enough.

[ConnorMcMahon]

This seems like something we should handle in our serialization docs for Durable. I believe we have advice there for Python + Node, so some PowerShell specific advice (such as default serialization depth) may be a good idea as well.

[snerte]

I am also experiencing this issue while calling an Invoke-ActivityFunction from an orchestration function.

I have tried taking control over serialization, but it hasn't succeeded so far. Even as I pass a string, the $InputObject is of type System.Colections.Hashtable.

Orchestrator:

    $jsonInput = [PSCustomObject]@{
        Result = $_
        (...)
    } | ConvertTo-Json -Depth 5

    Write-Host "Calling ActivityFunction with: $($jsonInput)"

    [PSCustomObject]@{
        Result = $_
        Task   = Invoke-ActivityFunction -FunctionName 'ActivityFunction' -Input $jsonInput -NoWait
    }

Output from Write-Host in Orchestrator:

INFORMATION: Calling ActivityFunction with: { (...) some long JSON string }

So I know my deserialization works.

ActivityFunction:

param(
    [Parameter(Mandatory)]
    [string]
    $InputData)

Write-Host "ActivityFunction: Got JSON input $InputData"

$InputData = $InputData | ConvertFrom-Json

My Write-Host in ActivityFunction outputs: INFORMATION: SaveCustomerEnvironmentCheckResult: Got JSON input System.Collections.Hashtable. It seems like something else is wrong.

From the serialization I get this exception, which makes sense if the value is not valid JSON:

EXCEPTION: Conversion from JSON failed with error: Unexpected character encountered while parsing value: S. Path '', line 0, position 0. Exception : Type : System.ArgumentException Message : Conversion from JSON failed with error: Unexpected character encountered while parsing value: S. Path '', line 0, position 0. TargetSite : Name : ConvertFromJson DeclaringType : Microsoft.PowerShell.Commands.JsonObject MemberType : Method Module : Microsoft.PowerShell.Commands.Utility.dll StackTrace : at Microsoft.PowerShell.Commands.JsonObject.ConvertFromJson(String input, Boolean returnHashtable, Nullable`1 maxDepth, ErrorRecord& error) at Microsoft.PowerShell.Commands.ConvertFromJsonCommand.ConvertFromJsonHelper(String input) at Microsoft.PowerShell.Commands.ConvertFromJsonCommand.EndProcessing() at System.Management.Automation.Cmdlet.DoEndProcessing() at System.Management.Automation.CommandProcessorBase.Complete() InnerException : Type : Newtonsoft.Json.JsonReaderException TargetSite : Name : ParseValue DeclaringType : Newtonsoft.Json.JsonTextReader MemberType : Method Module : Newtonsoft.Json.dll StackTrace : at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonTextReader.Read() at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Microsoft.PowerShell.Commands.JsonObject.ConvertFromJson(String input, Boolean returnHashtable, Nullable`1 maxDepth, ErrorRecord& error) Message : Unexpected character encountered while parsing value: S. Path '', line 0, position 0. Source : Newtonsoft.Json HResult : -2146233088 Source : Microsoft.PowerShell.Commands.Utility HResult : -2147024809 CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand InvocationInfo : MyCommand : ConvertFrom-Json ScriptLineNumber : 13 OffsetInLine : 27 HistoryId : 1 ScriptName : C:\home\site\wwwroot\SaveCustomerEnvironmentCheckResult\run.ps1 Line : $InputData = $InputData | ConvertFrom-Json PositionMessage : At C:\home\site\wwwroot\SaveCustomerEnvironmentCheckResult\run.ps1:13 char:27 + $InputData = $InputData | ConvertFrom-Json + ~~~~~~~~~~~~~~~~ PSScriptRoot : C:\home\site\wwwroot\SaveCustomerEnvironmentCheckResult PSCommandPath : C:\home\site\wwwroot\SaveCustomerEnvironmentCheckResult\run.ps1 InvocationName : ConvertFrom-Json CommandOrigin : Internal ScriptStackTrace : at <ScriptBlock>, C:\home\site\wwwroot\SaveCustomerEnvironmentCheckResult\run.ps1: line 13 PipelineIterationInfo :

[snerte]

Also I think an error message would be in handy when an object is deeper than supported, a user will not expect serialization issues in the framework. At least I didn't ;)

Or even better, do not set a serialization depth... But I'm guessing that is a PowerShell issue. Even if so, this is very unexpected for users.

[snerte]

So... this is a workaround for the issue. ActivityFunction:

param(
    [Parameter(Mandatory)]
    [object]
    $InputData)

Write-Host "ActivityFunction: Got JSON input $($InputData | ConvertTo-Json -Depth 5)"

The framework will try to serialize to an object. Even if the input from the orchestrator was of type string. I think this is a bug that should be fixed:

# sample user input
$object = Get-Host
$myStringInput = $object | ConvertTo-Json

# framework deserialization
$frameWorkJson = $myStringInput | ConvertTo-Json

# framework serialization
$string= $frameWorkJson | ConvertFrom-Json

$string.GetType() #should be of type string, containing deserialized object!

[davidmrdavid]

Thank you for reaching out @snerte. The current serialization behavior is problematic, I agree, but unfortunately it is quite hard to fix without introducing a breaking change. I'm currently working on releasing the next major version of the Durable Functions for PowerShell SDK, and I have included the fix for this serialization problem in there. I'll have more updates on this once we approach our release date, but I'll make a note to officially document this problem in the meantime. Thanks!

[anthonywhite]

Thank you for reaching out @snerte. The current serialization behavior is problematic, I agree, but unfortunately it is quite hard to fix without introducing a breaking change. I'm currently working on releasing the next major version of the Durable Functions for PowerShell SDK, and I have included the fix for this serialization problem in there. I'll have more updates on this once we approach our release date, but I'll make a note to officially document this problem in the meantime. Thanks!

hi @davidmrdavid do you have a release date yet for that next version?

[Manbearpiet]

Yeah we're interested in this too 👍

[davidmrdavid]

@anthonywhite , @Manbearpiet , @snerte, @jformacek:

This took a bit to release, but this new version of the SDK has finally been announced here. That version greatly increases the serialization depth (depth is now 100 and due to this now being a standalone package, we can pretty safely increase this depth as well based on user feedback as needed).

This SDK package is still in public preview, but the plan is to GA it fairly soon. In any case, I wanted to give y'all a heads up in case you want to try it out and provide us with feedback. All the details should be in the announcement above. Thank you!