Batch transform fails for large files ( IP Insights Model )

[ksachdeva11]

Describe the bug
I am using AWS inbuilt IP Insights model to run batch transform. This batch transform is failing with the below error -

2020-10-14T04:53:51.549:[sagemaker logs]: bucket-name/v1_2020-10-13.csv: Unable to get response from algorithm

To reproduce
Run the batch transform on this model and you will see the error.

Expected behavior
Batch transform should work without any issues.

Screenshots or logs
2020-10-14T04:53:51.549:[sagemaker logs]: bucket-name/v1_2020-10-13.csv: Unable to get response from algorithm

System information
A description of your system. Please provide:

• SageMaker Python SDK version:
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): IP Insights
• Framework version:
• Python version:
• CPU or GPU: CPU
• Custom Docker image (Y/N): N

Additional context
The CSV file I am using is 600MB and instance type I'm using for batch transform is ml.m5.4xlarge

[ksachdeva11]

@andremoeller @laurenyu Would you be able to help, please?

[laurenyu]

cc @ChuyangDeng

[chuyang-deng]

@andremoeller Could you help to confirm that 600MB is a safe size for batch transform?

[andremoeller]

Hi @ksachdeva11

That message Unable to get response from algorithm indicates that the IPInsights server couldn't respond to individual inference requests, but it's not quite definitive why.

Would you be able to run a job with the following values? These are best guesses based on the instance type, the fact that your data is CSV -- together, these mean "split up the input files into 1MB requests by newline characters, and send 16 concurrent requests to the IPInsights server"

MaxConcurrentTransforms: 16
MaxPayloadInMB: 1
SplitType: "Line"
BatchStrategy: "MultiRecord"
ContentType: "text/csv"

Also, would you be able to share which values for these four fields you used in your request?

I've also notified the IPInsights owners as well, in case there's another root cause.

Thanks!