Parsing error when using [] rather than . to parse nested json object

[belindaboehm]

When attempting to evaluate an expression containing a json object using [attributeName] rather than .attributeName notation, I receive the following error

E           lark.exceptions.UnexpectedCharacters: No terminal matches '[' in the current parser context, at line 1 col 81
E
E           pelineReturnValue.returnValue.attribute1[pipeline().parameters.parameter]
E                                                   ^
E           Expected one of: 
E               * EXPRESSION_ARRAY_INDEX
E               * EXPRESSION_NULL_CONDITIONAL_OPERATOR
E               * EXPRESSION_WS
E               * DOT

In this case, the expression to be evaluated is
"@activity('get nested object').output.pipelineReturnValue.returnValue.attribute1[pipeline().parameters.parameter]"

which is used to retrieve a specific component of the following json object:

{
    "attribute1": {
        "param1": "value of attribute1, param1",
        "param2": "value of attribute1, param2"
    },
    "attribute2": "value of attribute 2"
}

A pipeline parameter is used to specify which component within attribute1 should be returned so this cannot be specified directly.

For the above json object and providing pipeline parameter of parameter = "param1", the expression should evaluate to "value of attribute1, param1".

The expression is evaluated correctly within ADF.

The following expression is also evaluated without error
"@activity('get nested object').output.pipelineReturnValue.returnValue.attribute1.param1"

failing test

from data_factory_testing_framework import TestFramework, TestFrameworkType
from data_factory_testing_framework.state import (
    RunParameter,
    RunParameterType,
    DependencyCondition
)
def test():
    testFramework = TestFramework(framework_type=TestFrameworkType.DataFactory, root_folder_path='', should_evaluate_child_pipelines=False,)
    pipeline = testFramework.get_pipeline_by_name(name="test_parse_nested_objects") 
    activities = testFramework.evaluate_pipeline(pipeline, parameters=
            [
                RunParameter(RunParameterType.Pipeline, "parameter", "param1"),
                RunParameter(RunParameterType.System, "pipelineReturnValue", "")
            ]
        )
    
    # activity 1: get the nested object to be consumed by a subsequent activity
    activity = next(activities)
    activity.set_result(DependencyCondition.SUCCEEDED, output = {
        "pipelineReturnValue": {
            "returnValue" :{
                "attribute1": {
                    "param1": "value of attribute1, param1",
                    "param2": "value of attribute1, param2"
                },
                "attribute2": "value of attribute 2"
		    }
        }
    })
    
    # activity 2: read one of the attributes out of the nested response - fails when attempting to evaluate next activity
    activity = next(activities)

Pipelines to reproduce error

Parent

{
	"name": "test_parse_nested_objects",
	"properties": {
		"activities": [
			{
				"name": "get nested object",
				"type": "ExecutePipeline",
				"dependsOn": [],
				"policy": {
					"secureInput": false
				},
				"userProperties": [],
				"typeProperties": {
					"pipeline": {
						"referenceName": "test_parse_nested_objects_child",
						"type": "PipelineReference"
					},
					"waitOnCompletion": true
				}
			},
			{
				"name": "read value for parameter",
				"type": "SetVariable",
				"dependsOn": [
					{
						"activity": "get nested object",
						"dependencyConditions": [
							"Succeeded"
						]
					}
				],
				"policy": {
					"secureOutput": false,
					"secureInput": false
				},
				"userProperties": [],
				"typeProperties": {
					"variableName": "set_value",
					"value": {
						"value": "@activity('get nested object').output.pipelineReturnValue.returnValue.attribute1[pipeline().parameters.parameter]",
						"type": "Expression"
					}
				}
			}
		],
		"parameters": {
			"parameter": {
				"type": "string",
				"defaultValue": "param1"
			}
		},
		"variables": {
			"nested_config": {
				"type": "String"
			},
			"set_value": {
				"type": "String"
			}
		},
		"annotations": []
	}
}

Child

{
	"name": "test_parse_nested_objects_child",
	"properties": {
		"activities": [
			{
				"name": "lookup",
				"type": "SetVariable",
				"dependsOn": [],
				"policy": {
					"secureOutput": false,
					"secureInput": false
				},
				"userProperties": [],
				"typeProperties": {
					"variableName": "pipelineReturnValue",
					"value": [
						{
							"key": "returnValue",
							"value": {
								"type": "Object",
								"content": [
									{
										"key": "attribute1",
										"value": {
											"type": "Object",
											"content": [
												{
													"key": "param1",
													"value": {
														"type": "String",
														"content": "value of attribute1, param1"
													}
												},
												{
													"key": "param2",
													"value": {
														"type": "String",
														"content": "value of attribute1, param2"
													}
												}
											]
										}
									},
									{
										"key": "attribute2",
										"value": {
											"type": "String",
											"content": "value of attribute 2"
										}
									}
								]
							}
						}
					],
					"setSystemVariable": true
				}
			}
		],
		"variables": {
			"nested_config": {
				"type": "String"
			},
			"set_value": {
				"type": "String"
			}
		},
		"annotations": []
	}
}

[LeonardHd]

Thank you @belindaboehm for the excellent bug report. Using your sample and the pipelines you provided I can confirm that Azure Data Factory indeed evaluates, and our evaluator fails.

Will need to investigate further. I will keep you posted!

[LeonardHd]

@belindaboehm thanks for reporting the issue, it should be fixed with version 1.2.2.

[belindaboehm]

thanks @LeonardHd - working perfectly. Much appreciated!