Converted Gemma 2b tflite generate same token

[dengzheng-cloud]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Android 11

MediaPipe Tasks SDK version

0.10.11

Task name (e.g. Image classification, Gesture recognition etc.)

llminference

Programming Language and version (e.g. C++, Python, Java)

python

Describe the actual behavior

convert model.tflite and push that to /data/local/tmp/llm, while llminference output same token

Describe the expected behaviour

after converted, llminference should output dialog

Standalone code/steps you may have used to try to get what you need

To convert gemma 2b huggingface model into mediapipe tflite, i use this website https://developers.google.com/mediapipe/solutions/genai/llm_inference/android#convert-model

Other info / Complete Logs

python code reference site: https://colab.research.google.com/github/googlesamples/mediapipe/blob/main/examples/llm_inference/conversion/llm_conversion.ipynb#scrollTo=LSSxrLyQPofw
init config with feedforward,attention,embedding quant bits=4

converter.ConversionConfig(input_ckpt=input_ckpt, ckpt_format='safetensors', model_type='GEMMA_2B', backend=backend, output_dir=output_dir, combine_file_only=False, vocab_model_file=vocab_model_file, output_tflite_file=output_tflite_file, feedforward_quant_bits=4,attention_quant_bits=4,embedding_quant_bits=4)


[LOG output]
WARNING:jax._src.xla_bridge:An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1711354375.520733   36623 model_ckpt_util.cc:350] Inserting empty buffer to TfLite.

then push model.tflite into /data/local/tmp/llm. chat with Android llminference, input "hello", get "As an disreg disreg disreg disreg disreg ..."

[kuaashish]

Hi @dengzheng-cloud,

Gemma 2B model does not need to be converted into LLInference format if downloaded from Kaggle. This is documented here.
If you are not using Gemma, please let us know which model you are using so that we shall reproduce the issue with that model at our end.

Thank you!!

[dengzheng-cloud]

Hi @dengzheng-cloud,

Gemma 2B model does not need to be converted into LLInference format if downloaded from Kaggle. This is documented here. If you are not using Gemma, please let us know which model you are using so that we shall reproduce the issue with that model at our end.

Thank you!!

already done, choose to use gemma from kaggle, will the details of how the gemma model performs int4 quantization be released later?

[kuaashish]

Hi @dengzheng-cloud,

Based on our internal discussion, Our team is actively working on this and it's definitely on our roadmap. While we can not provide an time, but it will be available soon.

Thank you!!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[adityak6798]

I'm facing the same issue when running gemma 2b on android (in a kotlin codebase). The call returns the same output token repeatedly, whereas trying the same prompt on the vertex ai portal with gemma 2b gives a good output.