unkown token issue

[nimblestart]

I have set this up on a windows laptop and when running the model below where I'm stuck, I do not have an option to enter my query, any help will be appreciated

Enter a query:
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'

[KeulzZz]

same issue here...

[mnkjiu]

I have the same issue. I tried it a few times asking similar questions (related to the same topic) and I got that error. But when I change my question (an entirely different topic) I am not seeing it.

Not sure whether does the prompting has anything to do with it or not, doing more testing at the moment.

[johnbrisbin]

When a query is processed, the first stage is looking up similar texts in the database you ingested. These documents can contain most anything. However, these are then used in the prompt submitted to the LLM you chose. Each LLM has to tokenize the information so it can be used with the training of the LLM. It looks like it is failing because one of the returned documents from the initial search contained characters of an unexpected type, perhaps in Cyrllic script. Or the entries are corrupt.
That would explain some questions resulting in this error and some not if that document was implicated in the first question and not the next.
That doesn't explain it failing before the query is entered, however. I had a problem during ingestion where a couple of documents could not be ingested because of the content and an assumption that it used a windows character set (which it did not).
As to what you might try:
Rename your db folder to something else an try ingestion again. If there was some corruption in the db, this might fix it, This should not be an upgrade issue, unless you change to another embedding LLM. If it happens again it may be a specific document and you might have to try adding a few documents via ingest.py then testing, then ingesting a few more, testing until you can figure out which document causes the issue.

[amitli1]

I got same error when using the example from readme.md.
Is there any one it works on his environment?

[johnbrisbin]

That should not be happening on the piece of fluff example that is pre-installed in the source_documents folder. It has no Cyrillic text or anything outside ASCII so far as I know.
Are each of you on Windows machines? If so are you using the Windows python installs? They may try to interpret utf8 as if it were a windows character set in some cases. There are 'official' installations for Windows from https://www.python.org/ that I personally prefer.
What version python are you using? One of the more common issues is using a python 9 version (not supported, 10 minimum). That used to have an easily identifiable error but other errors have shown up of late.

I also note that the README.md is rather poorly worded:

And wait for the script to require your input.

>Enter a query:
Hit enter.

More correctly, it should be:

And wait for the script to require your input.

>Enter a query:
Enter your question, then hit enter.

Sorry if that was obvious.

[amitli1]

I'm using windows 10 , python 3.10.9.
Is it matter if we are running on windows or linux ?

[johnbrisbin]

Python is pretty good about bridging over the differences, except in a few areas. One is launching subprocesses. The other is Microsoft. I found a module loaded under a MS installation of python that assumed a windows character set instead of unicode. It was easy to override, but it caused a vaguely similar error (memory going...).
So, is your python 10 one installed by Microsoft or one from python.org?
If you do install a standard python, you will have to go through the requirements.txt and any manual pip installs again. And there are no guarantees that it will fix it.