Llama3-8b 🦙

[khatwanimohit]

No description provided.

[qihqi]

qq; why not import and use this: https://github.com/google/JetStream/blob/main/jetstream/engine/token_utils.py#L353

[khatwanimohit]

The reason I created a class for it and not used the jetstream is because of the error below. Our input pipeline is tfds based so we are tokenizing Symbolic Tensors instead of np array's

File "/__w/maxtext/maxtext/MaxText/tokenizer.py", line 79, in __call__  *
        features[k], _ = self.sp_tokenizer.encode(str(features[k]), is_bos = self.add_bos, is_eos = self.add_eos)
    File "/usr/local/lib/python3.10/dist-packages/jetstream/engine/token_utils.py", line 271, in encode  *
        tokens = np.array(self.vocab.encode_tf(s))

    NotImplementedError: Cannot convert a symbolic tf.Tensor (SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

Do you suggest another way to get around this problem ?

[qihqi]

I don't think the Tiktoken class you have created can handle tf.Tensor as well.

Looks like the SentencePieceTokenizer you used from tensorflow_text made it into a tf op. And I don't think tensorflow_text has tiktoken.

Modifying maxtext to pass in numpy arrays (for both tiktoken and sentencepiece) should be the way to go.

[khatwanimohit]

Tiktoken class here works with tf.Tensors. See the tests passing here: http://shortn/_YvmkdOTxIQ.