Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempted Fix: Sometimes tool_calls is undefined and crashes server #21

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Ademsk1
Copy link

@Ademsk1 Ademsk1 commented May 1, 2024

Hi

This might help with issue #20 . I'm hoping to test it for openai later on today, but I don't have access to a local language model right now, are there any models you recommend I use?

Thanks

@Ademsk1 Ademsk1 changed the title Attempted Fix: Somethings tool_calls is undefined and crashes server Attempted Fix: Sometimes tool_calls is undefined and crashes server May 1, 2024
@mishushakov
Copy link
Owner

Nice. Have you tested it?

@Ademsk1
Copy link
Author

Ademsk1 commented May 2, 2024

@mishushakov tested for the openai section, doesn't crash the server anymore :D
Screenshot 2024-05-02 at 11 18 39

I haven't tested the Llama one, but I think it should work the same. I started working on one last night, so will let you know of the results later on this evening.

@mishushakov
Copy link
Owner

Very good! I will run some tests too. Looking forward to get it merged 😄

@Ademsk1
Copy link
Author

Ademsk1 commented May 2, 2024

@mishushakov Been trying to test using this model as my model, but my machine cannot handle it. What model are you using?

@mishushakov
Copy link
Owner

TinyLlama

@mishushakov
Copy link
Owner

I can test for you, no worries

@Ademsk1
Copy link
Author

Ademsk1 commented May 3, 2024

Tested this with TinyLlama and no crashes :D
Screenshot 2024-05-03 at 13 37 33

that being said, there weren't any crashes without the try/catch block for the local model, because we don't reference any tool_calls or the like. Still I think it's practical to have it there just in case.

@Ademsk1
Copy link
Author

Ademsk1 commented May 3, 2024

While I have you @mishushakov , maybe you have more experience, with these models, I seem to get this error when trying to scrape data from a fairly large website.
GGML_ASSERT: ... n_tokens_all <= cparams.n_batch
Did you have this error at all? Any advice?

@mishushakov
Copy link
Owner

I believe you're hitting the input token limit. We need to think about a way of splitting websites into multiple chunks, maybe. Can you open an issue for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants