SM Elastic Inference Accelerators are not available during inference

[vinayak-shanawad]

Hi Team,

Greetings!!

We are able to deploy on real-time endpoint with elastic inference accelerators. But SM Elastic Inference Accelerators are not available during inference, could you please have a look?

Note:

1. We are bringing our own trained model using Pytorch = 1.10.2 on SM.
2. We don't find any errors in cloud watch logs.
3. Converted our trained model to TorchScript and able to load that model as well during inference.
4. Tried both Torchscript's trace and script modes but no luck.

SageMaker version: 2.76.0

Code"
from sagemaker.pytorch import PyTorchModel
from sagemaker import get_execution_role

endpoint_name = 'ner-bert'

pytorch = PyTorchModel(entry_point='deploy_ei.py',
source_dir='code',
model_data=model_data,
role=get_execution_role(),
framework_version='1.3.1',
py_version='py3',
sagemaker_session=sagemaker_session)

predictor = pytorch.deploy(initial_instance_count=1,
instance_type='ml.m5.large',
accelerator_type='ml.eia2.xlarge',
endpoint_name=endpoint_name)

Thanks,
Vinayak

[mohanasudhan]

@vinayak-shanawad Thanks for reporting this issue and apologies for delay in the response. Please refer to Elastic Inference document for recommendation https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html

Q: We currently use Amazon Elastic Inference (EI) accelerators. Will we be able to continue using them after April 15, 2023?

Yes, you will be able use Amazon EI accelerators. We recommend that you migrate your current ML Inference workloads running on Amazon EI to other hardware accelerator options at your earliest convenience.

Recommend going over the migration steps https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html#ei-migration

Please reopen the issue if you see any issue with migration.