External payload storage is not reducing the workflow & task size

[devinderpal]

Describe the bug
I recently migrated from 2.x version and bumped into this bug on 3.6 (and even in the main branch). I had to implement a workaround to make this functionality work for our product.

When the payload size crosses the threshold value, the affected task and workflow input/output data are uploaded successfully to the S3 bucket. But when the object is converted to JSON to store in the DB, that big JSON is still there. It results in data crossing the MySQL threshold to store. Even though we are using MySQL, but this issue is valid for all data stores.

I believe the code issue is below in both TaskModel and WorkflowModel classes. As the inputPayload is populated with the large data, inputData field returned after Jackson's Object to JSON conversion will still have the data.

public Map<String, Object> getInputData() {
        return externalInputPayloadStoragePath != null ? inputPayload : inputData;
    }

    public Map<String, Object> getOutputData() {
        return externalOutputPayloadStoragePath != null ? outputPayload : outputData;
    }

Details
Conductor version: 3.6
Persistence implementation: MySQL

To Reproduce
Steps to reproduce the behavior:

1. Have a workflow with large data set. Even a single node with large input can be used to reproduce
2. Run the flow
3. The data is uploaded to S3 bucket. externalInputPayloadStoragePath, externalOutputPayloadStoragePath for TaskModel and WorkflowModel populates correctly.
4. The database save will either fail (depending on the size of the large data) or it will succeed. But the JSON will still be big. That is the large fields will still be in the object.

Expected behavior
The data should store in the DB after setting the large data set the field to an empty map.

Additional context
Add any other context about the problem here.

[aravindanr]

@devinderpal Thanks for reporting this. We are working on a fix. It should be addressed in the next release.

[devinderpal]

Great!

…

Sent from my iPhone

On Apr 11, 2022, at 6:54 PM, Aravindan Ramkumar ***@***.***> wrote:  @devinderpal Thanks for reporting this. We are working on a fix. It should be out in the next version. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[aravindanr]

v3.7.2 contains the fix.