Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix delta role error when using custom LLM #3223

Merged
merged 5 commits into from
Nov 11, 2023
Merged

Fix delta role error when using custom LLM #3223

merged 5 commits into from
Nov 11, 2023

Commits on Nov 10, 2023

  1. Fix delta role error when using custom LLM 

    This addresses a delta chunk issue that happens when you use a custom baseURL on the openai chat model. Some models like llama 2 on openrouter may have a empty delta and result in a undefined error.

Example error:
```
Cannot read properties of undefined (reading 'role')",
  "error.stack": "TypeError: Cannot read properties of undefined (reading 'role')\n    at _convertDeltaToMessageChunk (/home/ubuntu/node_modules/langchain/dist/chat_models/openai.cjs:72:24)\n    at ChatOpenAI._streamResponseChunks (/home/ubuntu/node_modules/langchain/dist/chat_models/openai.cjs:409:27)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ChatOpenAI._streamIterator (/home/ubuntu/node_modules/langchain/dist/chat_models/base.cjs:77:34)\n    at async RunnableSequence._streamIterator (/home/ubuntu/node_modules/langchain/dist/schema/runnable/base.cjs:780:30)\n    at async Object.pull (/home/ubuntu/node_modules/langchain/dist/util/stream.cjs:73:41)
```

on _convertDeltaToMessageChunk, openai works fine but when you start using models from openrouter or other baseURL llms, it doesnt take in account for empty deltas and assumes theres always a value.

I was able to test with a custom baseURL and normal openai that this works with no errors on streaming after this tweak.
    @ninjamaster1337
    ninjamaster1337 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    9b67764 View commit details
    Browse the repository at this point in the history

  2. add default role

    @ninjamaster1337
    ninjamaster1337 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    13af5e5 View commit details
    Browse the repository at this point in the history

  3. add ? on chunk text for streaming delta

    @ninjamaster1337
    ninjamaster1337 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    45a2c0b View commit details
    Browse the repository at this point in the history

  4. Simpler fix

    @jacoblee93
    jacoblee93 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    5fd8f19 View commit details
    Browse the repository at this point in the history

  5. Revert

    @jacoblee93
    jacoblee93 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    f3608a9 View commit details
    Browse the repository at this point in the history