Add support for chatting with models hosted on Databricks#152
Merged
Conversation
atheriel
force-pushed
the
databricks-provider
branch
2 times, most recently
from
November 12, 2024 18:54
7032406 to
4c5e61c
Compare
atheriel
changed the title
atheriel
marked this pull request as ready for review
November 12, 2024 18:55
hadley
reviewed
Nov 15, 2024
atheriel
force-pushed
the
databricks-provider
branch
from
November 16, 2024 00:50
4c5e61c to
6e1967f
Compare
Member
|
Shall we add this to the automated tests too? |
This commit adds support for chatbots that work with models hosted on
Databricks [0].
According to their documentation, Databricks models can be queried by
the OpenAI Python SDK, so in theory they would be compatible with our
OpenAI provider. But in practice there are enough differences that a new
provider is necessary.
There are other limitations, too: structured outputs are in public
preview, and there is no image support.
Finally, I've found that tool calling support (also in public preview)
is very limited, too limited for `elmer` to work properly at all:
> chat <- chat_databricks()
> chat$register_tool(tool(function() "2024-01-01", "Gets the current date"))
Using model = "databricks-dbrx-instruct".
> chat$chat("What's the current date in YMD format?")
Error in `req_perform_connection()` at elmer/R/httr2.R:36:3:
! HTTP 400 Bad Request.
• Bad request: Model does not support continuing the chat past the first tool call
Show Traceback
> chat
<Chat turns=2 tokens=726/11>
── user ────────────────────────────────────────────────────────────────────────────────────────
What's the current date in YMD format?
── assistant ───────────────────────────────────────────────────────────────────────────────────
[tool request (call_93d2e2fe-1a2e-4ef4-80df-4feaed245569)]: tool_001()
Databricks has announced that this limition will be lifted once tool
calling is out of public preview [1], so I suppose we'll have to cross
our fingers that this will start working at that point.
Unit tests and docs are included.
[0]: https://docs.databricks.com/en/machine-learning/model-serving/score-foundation-models.html
[1]: https://docs.databricks.com/en/machine-learning/model-serving/function-calling.html#limitations
Signed-off-by: Aaron Jacobs <aaron.jacobs@posit.co>
atheriel
force-pushed
the
databricks-provider
branch
from
November 18, 2024 20:17
6e1967f to
04340af
Compare
Collaborator
Author
|
This PR only supports PAT authentication, which we could use for CI, but I'd rather land support for Databricks service principals first. I have a SP set up for CI use already. Can we hold off until then? |
Member
|
Yeah, sure thing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds support for chatbots that work with models hosted on Databricks.
According to their documentation, Databricks models can be queried by the OpenAI Python SDK, so in theory they would be compatible with our OpenAI provider. But in practice there are enough differences that a new provider is necessary.
There are other limitations, too: structured outputs are in public preview, and there is no image support.
Finally, I've found that tool calling support (also in public preview) is very limited, too limited for
elmerto work properly at all:Databricks has announced that this limition will be lifted once tool calling is out of public preview, so I suppose we'll have to cross our fingers that this will start working at that point.
Unit tests and docs are included.
Closes #55.