Create Custom Handler Example for BERT Model

[mycpuorg]

BERT Models are not part of TorchServe's default handlers Given their popularity, it is worth while to add custom handlers showing the usage under examples.

[MFreidank]

Hi, I just created a medium post showcasing a simple custom handler for huggingface/transformers BERT:
https://medium.com/@freidankm_39840/deploy-huggingface-s-bert-to-production-with-pytorch-serve-27b068026d18

Thought I'd share it for reference, happy to help formalize it out for your documentation if you think that would be useful.

[mycpuorg]

Thank you! I think pushing this will be very helpful. We can bring this in under examples section.

[MFreidank]

@mycpuorg Great, if I can help put it together - happy to. Just was not sure what dataset to use for the training script (same dataset as in the case of the standard text classifier?) and whether you would want one example to show general usage or multiple ones per use-case (huggingface/transformers has examples for NER, sentence classification, Q&A etc).

[dhaniram-kshirsagar]

@maaquib @dhanainme Can you check if this can be closed.

[choshiho]

Hi, I just created a medium post showcasing a simple custom handler for huggingface/transformers BERT: https://medium.com/@freidankm_39840/deploy-huggingface-s-bert-to-production-with-pytorch-serve-27b068026d18

Thought I'd share it for reference, happy to help formalize it out for your documentation if you think that would be useful.

Hi, MFreidank, thank you for your sharing blog post https://medium.com/@freidankm_39840/deploy-huggingface-s-bert-to-production-with-pytorch-serve-27b068026d18. I follow your instructions and download bert-base-chinese.tar.gz in my directory transformer_model which contains bert_config.json, pytorch_model.bin, vocab.txt.

Then I use the following steps to serve my model, but get "Backend worker monitoring thread interrupted or backend worker process died. Load model failed: my_tc, error: Worker died." error message. My custom_handler.py is almost the same as your transformers_classifier_torchserve_handler.py in your blog post, except that

self.model = BertModel.from_pretrained(model_dir)
self.tokenizer = BertTokenizer.from_pretrained(model_dir)

in the initialize function. I can't load the pre-trained model successfully.
torch-model-archiver --model-name my_text_classifier --version 1.0 --serialized-file ./transformer_model/pytorch_model.bin --extra-files "./transformer_model/bert_config.json,./transformer_model/vocab.txt" --handler "./custom_handler.py"

mkdir model_store && mv my_text_classifier.mar model_store/

torchserve --start --model-store model_store --models my_tc=my_text_classifier.mar --ts-config config.properties

curl -X POST http://127.0.0.1:8083/predictions/my_tc -T demo_text.txt

Is there some points to notice to save checkpoint model? Because as per your blog post, your model can serve successfully.