Scriptable Tokenizer for Text Classification Example

[mreso]

…ptable tokenizer into a single artifact

Description

This example shows how to combine a model with a scriptable tokenizer and script the two together with TorchScript to achieve a single artifact.
The combination of model and tokenizer makes sure that the same tokenizer is used in training and inference.
Additionally, it decreases handler complexity and makes it easy to deploy these in the future C++ based backend.

Training the model depends on torchdata which is mentioned in the README. Inference does not require the package.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

Test plan:

• Execute commands in examples/text_classification_with_scriptable_tokenzier/README.md
• As well as examples/text_classification/README.md (text_handler.py was slightly modified)

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[msaroufim]

Thanks Matthias, super glad to see this work in. Left a bunch of feedback. The important ones is we need some test, updates to the README to clarify benefits to users and a few more comments in the train.py

[mreso]

Just adding the logs for running the unit test manually:

[msaroufim]

LGTM - although I think we should revert the changes to test/pytest and create_mar.sh let's fix those properly in another PR pending review in the CONTRIBUTING-dev.md spec I shared

[msaroufim]

is this change needed?

[mreso]

I think we should revert the changes to test/pytest and create_mar.sh let's fix those properly in another PR

Sounds good, reverted the changes

[lxning]

@mreso could you run sanity and regression test to make sure the pytest you added is verified?

[mreso]

Thank you @lxning! Here are the logs for the regression tests: https://gist.github.com/mreso/0bcaed895517f74a09cbd4e80621969c
Let me know if you need anything else!