Address the feedback regarding Bert tokenizer

[tarekgh]

No description provided.

[tarekgh]

@stephentoub the change here is addressing the feedback you had in the other PR #7275. Thanks!

[codecov]

Codecov Report

Attention: Patch coverage is 44.73684% with 63 lines in your changes missing coverage. Please review.

Project coverage is 68.87%. Comparing base (a7a6d88) to head (b7492da).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs	43.68%	40 Missing and 18 partials ⚠️
...crosoft.ML.Tokenizers/Normalizer/BertNormalizer.cs	66.66%	2 Missing and 1 partial ⚠️
...icrosoft.ML.Tokenizers/Model/WordPieceTokenizer.cs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7280      +/-   ##
==========================================
- Coverage   68.89%   68.87%   -0.02%     
==========================================
  Files        1467     1467              
  Lines      273875   273955      +80     
  Branches    28363    28380      +17     
==========================================
+ Hits       188686   188697      +11     
- Misses      77901    77947      +46     
- Partials     7288     7311      +23     

Flag	Coverage Δ
Debug	68.87% <44.73%> (-0.02%)	⬇️
production	63.33% <44.73%> (-0.02%)	⬇️
test	89.18% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...icrosoft.ML.Tokenizers/Model/WordPieceTokenizer.cs	75.05% <0.00%> (ø)
...crosoft.ML.Tokenizers/Normalizer/BertNormalizer.cs	62.85% <66.66%> (-3.81%)	⬇️
src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs	63.23% <43.68%> (-10.97%)	⬇️

... and 7 files with indirect coverage changes