remove tf dependency from tokenizer.py

[aireenmei]

Description

This is part of the bigger plan of removing tensorflow dependency in MaxText

• combine SentencePieceTokenizerGrain and SentencePieceTokenizer under the name SentencePieceTokenizer
• Uses the sentencepiece library to load sentencepiece tokenizer instead of the tensorflow_text library
• replace tf.io.gfile with Google Storage client for gs:// path
• move TokenizeOp to input_pipeline_utils.py, in the future will put functions using tf to a separate input_pipeline_utils_tf.py to make installing tf optional

Tests

CI test

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

❌ Patch coverage is 35.13514% with 24 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/maxtext/utils/gcs_utils.py	15.38%	11 Missing ⚠️
src/maxtext/input_pipeline/input_pipeline_utils.py	36.36%	7 Missing ⚠️
src/maxtext/input_pipeline/tokenizer.py	45.45%	5 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

🤖 Hi @aireenmei, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

## 📋 Review Summary

This PR successfully removes the tensorflow_text dependency by consolidating SentencePieceTokenizerGrain and SentencePieceTokenizer using the native sentencepiece library. The implementation correctly handles GCS loading and transitions the tokenization operation to a unified tf.py_function approach within input_pipeline_utils.py.

🔍 General Feedback

• Simplification: Merging the two classes and moving the file logic to gcs_utils.py significantly cleans up the tokenization architecture.
• Testing: The unit tests have been appropriately updated to accommodate the return type change (from tf.Tensor to list).
• Completeness: The removal of tokenizer imports in tfds_data_processing_c4_mlperf.py looks clean and accurate.

[hengtaoguo]

LGTM!