-
Notifications
You must be signed in to change notification settings - Fork 13.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to tokenize: langchain with gpt4all model #2659
Comments
are you using the Llama Cpp? I am encountering the same issue |
@adriacabeza yes im using gpt4all
|
I'm getting same error, but running:
I got this error too: |
I encountered the same error, with a similar setup, where I used Chroma and RetrievalQAWithSourcesChain. When using RetrievalQA.from_chain_type and the .run(query) method instead, I can successfully get results. This works:
This throws the error:
|
Try this walk-around, passing the
|
same error occured |
Review the content that you have indexed. Ensure you dont have any special characters or escaping characters in your dataset IE: " or / |
I am experiencing the same issue with the following commands:
myDoc.txt is long, gets broken into parts (by langchain), the 2 first parts get summarize well, then the error occurred with the 3rd part. The 3rd part is longer than the 1st and 2nd part. myDoc.txt does not have any special characters. I suspect the problem comes from configuration of Size of context, vs Size of parts of text |
How would this apply in the case of load_summarize_chain ? |
I am encountering the same issue when using I have tried the same way with a pdf only includes simple sentence, and works totally fine. I think the content might cause the issue. |
I am getting the same error by following the official tutorial from the docs: https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/chroma_self_query_retriever.html The only difference is that I use |
Hi, @Mohamedballouch! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale. From what I understand, the issue you reported is about a runtime error when trying to tokenize a text. It seems that the error message indicates that the text being tokenized is "b" and the number of tokens is -908. Other users have encountered the same issue and have provided potential workarounds, such as passing However, it seems that the issue has been resolved by using the workaround suggested by other users. By passing Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you for your contribution to the LangChain repository! |
RuntimeError: Failed to tokenize: text="b" Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\ndu Home Wireless.\nUnlimited internet with a free router\n\ndu home wireless is a limited mobility service and subscription.\n\nOnce the device is activated, your home location will be assigned as the first usage location and will only work there. If you decide to move or shift locations, please request us to unlock it and it will be re-assigned to the new location.\n\nAs a new du Home Wireless customer, you\xe2\x80\x99ll receive an LTE router for internet connection at the time of the service application. It will include a Data SIM with unlimited data.\n\ndu Home wireless advantages.\nUnlimited data: Enjoy wireless connectivity with unlimited data for 12 months.\n\nHigh-quality streaming: Stream your favorite entertainment, chat and game at the same time.\n\n5G-enabled router: Connect all your devices with the latest WiFi 5 technology.\nWhat is du Home Wireless?\n\nWhat type of internet activities does the Home Wireless Plan support?\n\nIt supports the following internet activities depending on the plan you get:\n\nStandard internet activities\nVideo and music streaming\nGaming and VR\nSocial media\nWeb surfing\nEmail and office work\n\nCan I connect with more than one device at the same time?\n\nYes, you can. Ideally, the average number of connected devices shouldn\xe2\x80\x99t exceed 4 large screens on our Home Wireless Plus and Home Wireless Entertainment Plans.\n\nWill I always get a consistent speed?\n\nInternet speed is not guaranteed. Individual results will vary as it might be affected by many factors such as the weather, interference from buildings and network capacity. The areas wherein you\xe2\x80\x99ll get the best coverage experience are the following:\n\nNear a window\nIn an open space away from walls, obstructions, heavy-duty appliances, or electronics such as microwave ovens and baby monitors\nNear a power outlet\n\nWill I be able to bring my own router?\n\nYes, you have the option to use your own router.\n\nTo connect, check the below steps:\n\nInsert your du SIM card in the back of the router\nConnect to power and turn on the device\nConnect to your router using the Wi-Fi SSID and WiFi password information at the sticker on the bottom\nFor connection steps, check the video: Watch now\n\nHow can I subscribe to the Internet Calling Pack on the Home Wireless Entertainment Plan?\n\nThe free Internet Calling Pack subscription will be added to your plan for a period of three months by default.\n\nWho is eligible to get the free Internet Calling Pack?\n\nNew Home Wireless Entertainment subscribers will enjoy this added benefit.\n\nHome Wireless plans are our new range of Home Plans, that offer instant connectivity, unlimited data and premium entertainment so you can enjoy instant, plug-and-play high-quality internet.\n\nWhere does this service work?\n\nThis service has limited mobility. Once the device is activated, your home location will be assigned as the first usage location and will only work there. If you decide to move or shift locations, you will have to ask us to unlock it so your Home Wireless can be re-assigned to your new location.\n\nWhat kind of router do I get with this plan?\n\nYou will receive a 5G-enabled router.\n\nWhat happens if I don\xe2\x80\x99t have 5G?\n\nIt will automatically connect to 4G.\n\nIs a Landline required for a home wireless connection?\n\nNo, it\xe2\x80\x99s not.\n\nHow does this service work?\n\nAs a new Home Wireless customer, you\xe2\x80\x99ll receive a router for internet connection at the time of your service application. It will include a Data SIM with unlimited data.\n\nQuestion: What is du Home Wireless?\nHelpful Answer:"" n_tokens=-908
The text was updated successfully, but these errors were encountered: