Fix: JSON Reference Path Error in AWS Step Functions Distributed Map

[nidhinnru]

Since April 30th, 2024, I've encountered a recurring error when creating/updating a state machine for Metaflow orchestration, particularly under the --use-distributed-map functionality. Recent changes to AWS API validation for state machine definitions have introduced more stringent checks before a state machine is created, as detailed in the AWS changelog.

The issue stems from the item_reader component improperly parsing JSON parameters due to a syntax error in the specified JSON Key path. This problem was not previously detected because there were no ValidateStateMachineDefinition checks in place. The original line causing the error in JSON was:

.parameter("Key.$", "$.Body.ResultFiles.SUCCEEDED.[0].Key")

To work around this issue, I modified the JSON path by removing the extra period before the array index, updating it to: .parameter("Key.$", "$.Body.ResultFiles.SUCCEEDED[0].Key")

This correction aligns with AWS's expected JSON path syntax and has resolved the error previously encountered during state machine creation. This change was tested by generating JSON definition using the --with retry step-functions create --use-distributed-map --only-json command, manually adjusted the JSON as needed, and subsequently using the AWS CLI for state machine creation.

Original Error:
Bootstrapping virtual environment(s) ... Virtual environment(s) bootstrapped! AWS Step Functions error: InvalidDefinition("An error occurred (InvalidDefinition) when calling the CreateStateMachine operation: Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value must be a valid Reference Path or a valid intrinsic function call at /States/review_videos_Map/ItemReader/Parameters/Key.$'")

Impact:
This fix ensures compatibility with AWS's updated validation requirements and prevents the InvalidDefinition error, which was thrown due to incorrect JSON Key path syntax which were previously allowed by AWS.

Fix:
Updated reference path as per https://states-language.net/#ref-paths.