Config parsing can be improved

[ericangelokim]

https://github.com/aws/sagemaker-inference-toolkit/blob/master/src/sagemaker_inference/model_server.py#L42

This code here can be improved to add a config file, and not just use the mms default or env vars that are parsed.

The utils can include support to merge/override config files, instead of appending string values to write to file.

[laurenyu]

thanks for the suggestion!

[arvarik]

+1

Currently it seems like only these 4 variables are able to be configured.

Having the ability to define your own config.properties would be nice (job_queue_size is particularly useful to me)

[sanjams2]

For users less keen on updating config files, it would also be nice if these custom config values could be passed to start_model_server. If there is worry these values would be lost on restart, then Inference Toolkit could take care of writing these custom values out to a file on disk that can be re-used.

[ziweijiang]

+1
When I'm trying to do inference for a file larger than 9M, the MMS will get stuck. And I found a solution in MMS github issue, configuring max_request_size. But I'm not able to do it. It would be really helpful it custom config file is supported.

[neo]

I come from here also looking to configure the job_queue_size: https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-troubleshooting.html