[REVIEW] Port of clx subword tokenizer to cudf

[davidwendt]

Reference #4981

The tokenizer is added to nvtext.

C++ API interface

std::unique_ptr<TokenizerResult> subword_tokenize(
  cudf::strings_column_view const& sentences,
  std::string const& filename_hashed_vocabulary,
  uint32_t max_sequence_length,
  uint32_t stride,
  bool do_lower,
  bool do_truncate,
  uint32_t max_num_sentences,
  uint32_t max_num_chars,
  uint32_t max_rows_tensor,
  rmm::mr::device_memory_resource* mr = rmm::mr::get_default_resource());

A 2nd API allows the caller to load the hashed vocabulary in a separate call in case it can be reused to tokenize multiple batches using the same vocabulary token ids.

The tokenize process occurs in 2 steps after the vocabulary file is loaded. The first step normalizes the data to optionally lower-case the character, remove accents, map spaces, add padding, and resolve some multi-byte characters. The 2nd step identifies the tokens (words between whitespace) and maps them to token ids in the vocabulary file.

The result is 3 columns of integers. The 1st is the set of token ids assigned to each word as found in the vocabulary file. The 2nd is an attention mask identifying which token values are valid. The 3rd is a vector or metadata per output row.

This PR includes gtest and gbenchmark code that uses a minimal vocab hash that it generates.
Also included is a starting set of Python/Cython interfaces to match the CLX version.

[GPUtester]

Please update the changelog in order to start CI tests.

View the gpuCI docs here.

[codecov]

Codecov Report

Merging #5511 into branch-0.15 will increase coverage by 0.27%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           branch-0.15    #5511      +/-   ##
===============================================
+ Coverage        85.87%   86.15%   +0.27%     
===============================================
  Files               72       72              
  Lines            12181    12926     +745     
===============================================
+ Hits             10461    11136     +675     
- Misses            1720     1790      +70     

Impacted Files	Coverage Δ
python/cudf/cudf/core/column/string.py	85.91% <100.00%> (-0.29%)	⬇️
python/cudf/cudf/core/column/column.py	89.57% <0.00%> (+1.60%)	⬆️
python/cudf/cudf/core/indexing.py	98.02% <0.00%> (+2.12%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 754f292...08a33ed. Read the comment docs.

[davidwendt]

rerun tests

[harrism]

Great work. Thanks for answering all my comments.