Can't load model estimater after training

[gwc4github]

I was trying to follow the Sagemaker instructions here to load the model I just trained and test an estimation. I get the error message:
NotImplementedError: Creating model with HuggingFace training job is not supported.
Can someone share some sample code to run to do this? Here is the basic thing I am trying to do:

from sagemaker.estimator import Estimator

# job which is going to be attached to the estimator
old_training_job_name='huggingface-sdk-extension-2021-04-02-19-10-00-242'

# attach old training job
huggingface_estimator_loaded = Estimator.attach(old_training_job_name)

# get model output s3 from training job
testModel = huggingface_estimator_loaded.model_data

ner_classifier = huggingface_estimator.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

I also tried some things with .deploy() and endpoints but didn't have any luck there either.

[gwc4github]

@philschmid I know you are an expert on the Sagemaker side of the product. Can you point me in the right direction for this?

[philschmid]

Hey @gwc4github,

I have to tell you that using Estimator.deploy() is currently not possible yet. Currently, the easiest way to test your model is to upload it to the Hub and then use the inference-API.

If you want to use SageMaker for inference you can write some custom code to load and run inference for your model. Documenetation on to bring your own model

​Q: How can I run inference on my trained models?

A: You have multiple options to run inference on your trained models. One option is to use Hugging Face Accelerated Inference-API hosted service: start by uploading the trained models to your Hugging Face account to deploy them publicly, or privately. Another great option is to use SageMaker Inference to run your own inference code in Amazon SageMaker. We are working on offering an integrated solution for Amazon SageMaker with Hugging Face Inference DLCs in the future - stay tuned

I hope that we can provide a prototype for Estimator.deploy as soon as possible.

[C24IO]

Hey @gwc4github you would have to implement a model_loading and inference handler for this to get setup within a SageMaker Endpoint. Would you please mind sharing the Framework (TF/PyTorch), version, CPU/GPU for your usecase. I can send you the recipe for writing a model_handler post that.

Here is how it will look like, from within SageMaker endpoint:

    self.manifest = ctx.manifest
    properties = ctx.system_properties
    self.device = 'cpu'
    model_dir = properties.get('model_dir')
    
    #print('model_dir ' + model_dir)
    self.model = RobertaModel.from_pretrained(model_dir)
    self.tokenizer = RobertaTokenizerFast.from_pretrained(model_dir)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[gwc4github]

@C24IO Sorry I missed your generous offer. I must have missed the notification.
We are using PyTorch (mainly because that is what most of the sample code is written in) and for CPU/GPU we are using 1 GPU for now (during development.)

[C24IO]

Ok, please check this workshop for setting up PyTorch models, https://github.com/tescal2/TorchServeOnAWS/tree/master/2_serving_natively_with_amazon_sagemaker Please let me know if you run into any issues.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[philschmid]

@gwc4github we released last week the inference solution for Hugging Face on SageMaker. You can learn more about it in our new blog post or in the documentation
You just need to make sure that the model.tar.gz you create in your TrainingJob contains all model artifacts, like Tokenizers and model etc..

[gwc4github]

Thanks for the update Philipp! I'll take a look!