Error when invoking Update-AzSynapseSparkPool with "-SparkConfiguration"

[lilysuwito]

Description

1. Call this command to add a spark configuration to Synapse workspace.

$config = New-AzSynapseSparkConfiguration -WorkspaceName 'xxxxxxx-synapse-xxxxxxx-dev' -DefinitionFile "C:\replace-your-path\spkconf.json"

The content of the spkconf.json is:
{ "configs": { "spark.synapse.key1": "hello world", "spark.synapse.key2": "true" }, "annotations": [ "Sample" ] }

2. That works and no issue. The Apache Spark configurations published to the Synapse workspace just fine.
3. Then execute this command to assign the config to the spark pool.

Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse-xxxxxxx-dev" -Name "existing-spark-pool" -Tag @{"key" = "value"} -NodeCount 5 -NodeSize Medium -SparkConfiguration $config
and gets error:

I've also tried to use the New-AzSynapseSparkPool cmdlet to create a brand new pool and attached the sparkconfiguration, but still getting the same error.

Issue script & Debug output

$config = New-AzSynapseSparkConfiguration -WorkspaceName 'xxxxxxx-synapse-xxxxxxx-dev' -DefinitionFile "C:\replace-your-path\spkconf.json"

Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse-xxxxxxx-dev" -Name "existing-spark-pool" -Tag @{"key" = "value"} -NodeCount 5 -NodeSize Medium -SparkConfiguration $config

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.1
PSEdition                      Core
GitCommitId                    7.3.1
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.11.1                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     2.2.0                 Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Cl…

Error output

PS C:\Repo\xxxxxxxxxx-xxxxxxxxxx-1> Resolve-AzError
DEBUG: 1:40:07 AM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
DEBUG: 1:40:07 AM - using account id 'xxxxxxxxxx@microsoft.com'...
DEBUG: 1:40:07 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].        
WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.       
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

   HistoryId: 28

Message        : Operation returned an invalid status code 'BadRequest' Spark pool request validation failed. SparkPoolConfigContent Deserialization  
                 Failed. Content should be deserialied to SparkConfigurationInfoObject.
StackTrace     :    at Microsoft.Azure.Commands.Synapse.Models.SynapseAnalyticsManagementClient.CreateOrUpdateSparkPool(String resourceGroupName,     
                 String workspaceName, String sparkPoolName, BigDataPoolResourceInfo createOrUpdateParams, Nullable`1 Force)
                    at Microsoft.Azure.Commands.Synapse.UpdateAzureSynapseSparkPool.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)  
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : Microsoft.Azure.Commands.Common.Exceptions.AzPSException
InvocationInfo : {Update-AzSynapseSparkPool}
Line           :             Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse-xxxxxxx-dev" -Name "newperfpool" -Tag @{"key" = "value"}       
                 -NodeCount 5 -NodeSize Medium -SparkConfiguration $config

Position       : At line:1 char:13
                 +             Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse …
                 +             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 28

Message        : Operation returned an invalid status code 'BadRequest'
StackTrace     :    at Microsoft.Azure.Management.Synapse.BigDataPoolsOperations.BeginCreateOrUpdateWithHttpMessagesAsync(String resourceGroupName,   
                 String workspaceName, String bigDataPoolName, BigDataPoolResourceInfo bigDataPoolInfo, Nullable`1 force, Dictionary`2
                 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.Synapse.BigDataPoolsOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName,        
                 String workspaceName, String bigDataPoolName, BigDataPoolResourceInfo bigDataPoolInfo, Nullable`1 force, Dictionary`2
                 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.Synapse.BigDataPoolsOperationsExtensions.CreateOrUpdateAsync(IBigDataPoolsOperations operations, 
                 String resourceGroupName, String workspaceName, String bigDataPoolName, BigDataPoolResourceInfo bigDataPoolInfo, Nullable`1 force,   
                 CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.Synapse.BigDataPoolsOperationsExtensions.CreateOrUpdate(IBigDataPoolsOperations operations, String  
                 resourceGroupName, String workspaceName, String bigDataPoolName, BigDataPoolResourceInfo bigDataPoolInfo, Nullable`1 force)
                    at Microsoft.Azure.Commands.Synapse.Models.SynapseAnalyticsManagementClient.CreateOrUpdateSparkPool(String resourceGroupName,     
                 String workspaceName, String sparkPoolName, BigDataPoolResourceInfo createOrUpdateParams, Nullable`1 Force)
Exception      : Microsoft.Azure.Management.Synapse.Models.ErrorResponseException
InvocationInfo : {Update-AzSynapseSparkPool}
Line           :             Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse-xxxxxxx-dev" -Name "newperfpool" -Tag @{"key" = "value"}       
                 -NodeCount 5 -NodeSize Medium -SparkConfiguration $config

Position       : At line:1 char:13
                 +             Update-AzSynapseSparkPool -WorkspaceName "xxxxxxx-synapse …
                 +             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 28


   HistoryId: -1

Message        : You cannot call a method on a null-valued expression.
StackTrace     :    at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)      
                    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
                    at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean
                 createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior errorHandlingBehavior, Object      
                 dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Object[] args)
                    at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior,    
                 Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Boolean
                 propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDefine, Object[] args)
                    at Microsoft.PowerShell.Commands.PSPropertyExpression.GetValue(PSObject target, Boolean eatExceptions)
Exception      : System.Management.Automation.RuntimeException
InvocationInfo : {}
Line           :                                         if (($myinv -and $myinv.ScriptName -or $myinv.ScriptLineNumber -gt 1 -or
                 $err.CategoryInfo.Category -eq 'ParserError') -and !($myinv.ScriptName.EndsWith('.psm1',
                 [System.StringComparison]::OrdinalIgnoreCase))) {

Position       : At line:34 char:45
                 + …         if (($myinv -and $myinv.ScriptName -or $myinv.ScriptLineNumbe …
                 +               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : -1

[RakeshMohanMSFT]

@lilysuwito Thank you for reaching out, we are looking into it.

Thank you for your feedback. This has been routed to the support team for assistance.

[RakeshMohanMSFT]

@lilysuwito Please use this template instead. it will work.

{
  "configs": {
    "spark.synapse.key1": "hello world",
    "spark.synapse.key2": "true"
  },
  "created": "2023-01-31T10:29:22.2520000+05:30",
  "createdBy": "ramaruth@microsoft.com",
  "annotations": [
    "Sample"
  ],
  "configMergeRule": {
    "artifact.currentOperation.spark.synapse.key1": "replace",
    "artifact.currentOperation.spark.synapse.key2": "replace"
  }
}

Hi @lilysuwito. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.