Add libtorchtext cpp example

[Nayef211]

Reference Issue #1644

Description

• Adding an example that demonstrates how to use libtorchtext and libtorch libraries in a C++ application
• The example demonstrates the use of a scripted GPT2BPETokenizer and shows that the tokenization results are consistent across Python and C++

The example is inspired by the libtorchaudio examples from the torchaudio repo and the tokenizer example from @mreso

[Nayef211]

@parmeet one thing I wanted to double check with you is whether to reuse the gpt2_bpe_encoder.json and gpt2_bpe_vocab.bpe in the test/asset folder instead of checking it in the example folder. I thought it would be cleaner if all artifacts for the example were self-contained in one folder, but I could go either way here.

[mthrok]

gpt2_bpe_vocab.bpe and gpt2_bpe_encoder.json seem to be the second set of the same assets added in https://github.com/pytorch/text/tree/main/test/asset

This kind of trend is one of the reasons why I was suggesting not to check-in such a huge asset. #1462 (comment)

[parmeet]

I think instead we could just ask users to download the artifacts in readme? I agree with @mthrok to avoid check-in artifacts.

wget https://download.pytorch.org/models/text/gpt2_bpe_vocab.bpe

wget https://download.pytorch.org/models/text/gpt2_bpe_encoder.json

[Nayef211]

I think instead we could just ask users to download the artifacts in readme? I agree with @mthrok to avoid check-in artifacts.

wget https://download.pytorch.org/models/text/gpt2_bpe_vocab.bpe

wget https://download.pytorch.org/models/text/gpt2_bpe_encoder.json

Thanks for the feedback @mthrok and @parmeet. Just removed the assets and added instructions on how to download it.

[parmeet]

Thanks @Nayef211 for adding this example application, LGTM!!

[JiedaokouWangguan]

Hi @Nayef211 , I was wondering if we need a CMakeLists.txt in http://text/examples/libtorchtext/tokenizer/? I was not able to run the script
I'm sorry if the question is too dumb, I'm very new to cmake

[Nayef211]

Hi @Nayef211 , I was wondering if we need a CMakeLists.txt in http://text/examples/libtorchtext/tokenizer/? I was not able to run the script I'm sorry if the question is too dumb, I'm very new to cmake

@JiedaokouWangguan thanks for calling out this issue. I think it should be resolved by #1908. Would you be able to check out the PR locally and see if you are able to get the example working for you?

[JiedaokouWangguan]

Thanks for the quick fix! Let me have a try