Tokenizer's APIs Polishing

[tarekgh]

• Eliminate the perplexing specialTokens flag parameter from all APIs.
• Standardize the casing of the Tiktoken name throughout the entire codebase and public interfaces.
• Modify Model.Encode to operate with spans instead of strings. This change maintains consistency across all Model APIs and enables the possibility of incorporating Tokenizer's APIs that also operate with spans in the future.
• Introduce support for precise Last/IndexOf functionality. This enhancement enables the retrieval of indices even within pre-tokenized unit strings passed to the Model. For instance, the Llama model, which does not pre-tokenize, requires this capability to be supported effectively.

[tarekgh]

CC @ericstj @michaelgsharp

[codecov]

Codecov Report

Attention: Patch coverage is 58.63053% with 290 lines in your changes are missing coverage. Please review.

Project coverage is 68.45%. Comparing base (19fb805) to head (d2dfefd).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7108      +/-   ##
==========================================
- Coverage   68.49%   68.45%   -0.04%     
==========================================
  Files        1262     1262              
  Lines      254295   254775     +480     
  Branches    26242    26320      +78     
==========================================
+ Hits       174172   174410     +238     
- Misses      73432    73663     +231     
- Partials     6691     6702      +11     

Flag	Coverage Δ
Debug	68.45% <58.63%> (-0.04%)	⬇️
production	62.85% <55.81%> (-0.04%)	⬇️
test	88.57% <97.87%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...crosoft.ML.Tokenizers/PreTokenizer/PreTokenizer.cs	96.87% <ø> (-0.35%)	⬇️
...rc/Microsoft.ML.Tokenizers/PreTokenizer/Roberta.cs	57.14% <ø> (ø)
...kenizers/PreTokenizer/SentencePiecePreTokenizer.cs	100.00% <ø> (ø)
...ML.Tokenizers/PreTokenizer/TiktokenPreTokenizer.cs	90.24% <100.00%> (ø)
...Microsoft.ML.Tokenizers/PreTokenizer/Whitespace.cs	100.00% <ø> (ø)
...crosoft.ML.Tokenizers.Tests/EnglishRobertaTests.cs	95.71% <100.00%> (ø)
test/Microsoft.ML.Tokenizers.Tests/LlamaTests.cs	100.00% <100.00%> (ø)
...Microsoft.ML.Tokenizers.Tests/PreTokenizerTests.cs	95.23% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/TitokenTests.cs	100.00% <100.00%> (ø)
...st/Microsoft.ML.Tokenizers.Tests/TokenizerTests.cs	98.57% <96.77%> (-1.43%)	⬇️
... and 7 more

... and 7 files with indirect coverage changes

[tarekgh]

@stephentoub please let me know if you have more feedback on this PR or we are good to go. Thanks!