Hello, thanks for the timely response, but I have several problems during running

[zsg0827]

1. When running fp16, the output will be garbled and will not be output normally.
2. When running int4 and int8, the output result cannot be stopped normally.

[iangitonga]

What system are you using? Operating system, CPU, etc.
I ran the colab notebook and it seems fine.

[zsg0827]

I use the MACBOOK-intel CPU

[iangitonga]

Ok. I found a bug in the tokenizer and I'm currently fixing it.

[iangitonga]

The issue with the output text not stopping is now fixed. Also, I hope issue of text being garbled is resolved. Ensure you do not use -ffast-math option. Check the new flags in README file.