how to use gpu in sagemaker instance

[haiderasad]

hi i am going with a custom docker image with all the cuda cudnn installed and also tested locally gpu is being utilized. but when upload to ecr and create endpoint it does not create endpoint and says kindly make sure docker serve command is valid , from debugging i came to found out that inference toolkit is needed inside image for the image to see if sagemaker gpu is avail or not, but there is no sample dockerfile from which i can understand , kindly tell
1)how to enable cuda support in custom built docker images for sagemaker
2)will using prebuilt images e.g accountnum.aws.amazon.com/pytorch:1.10-cuda113-py3 directly use cuda/gpu of sagemaker instance?

[holopekochan]

For inference endpoint, probably, you can use GPU instance in SingleModel mode,
As I tried MultiModel mode with
763104351884.dkr.ecr.$REGION.amazonaws.com/pytorch-inference:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker

It shows error ClientError: An error occurred (ValidationException) when calling the CreateEndpointConfig operation: MultiModel mode is not supported for instance type ml.g4dn.xlarge.
from here, it said GPU instance is not supported aws/sagemaker-python-sdk#1323