LLM Tools support for Google Generative AI integration #117644

Shulyaka · 2024-05-17T15:08:02Z

Proposed change

Implement LLM Tools feature (#115464) for google_generative_ai_conversation. Added as a model picker and API picker in options flow:

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Deprecation (breaking change to happen in the future)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request: Update Google AI to allow HASS access home-assistant.io#32839

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
I have followed the perfect PR recommendations
The code has been formatted using Ruff (ruff format homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

home-assistant · 2024-05-17T15:08:09Z

Hey there @tronikos, mind taking a look at this pull request as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of google_generative_ai_conversation can trigger bot actions by commenting:

@home-assistant close Closes the pull request.
@home-assistant rename Awesome new title Renames the pull request.
@home-assistant reopen Reopen the pull request.
@home-assistant unassign google_generative_ai_conversation Removes the current integration label and assignees on the pull request, add the integration domain after the command.
@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

balloob · 2024-05-18T04:18:50Z

homeassistant/components/google_generative_ai_conversation/__init__.py

-                genai.get_model, entry.options.get(CONF_CHAT_MODEL, DEFAULT_CHAT_MODEL)
-            )
-        )
+        await hass.async_add_executor_job(partial(genai.get_model, DEFAULT_CHAT_MODEL))


I changed this because when I had configured a bad model, the config entry refused to startup

Why is that bad? Isn't it better to fail as soon as possible?

The config entry wouldn't set up and the user couldn't change the model anymore 🙈

(happened to me)

homeassistant/components/google_generative_ai_conversation/conversation.py

home-assistant · 2024-05-18T04:40:19Z

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

allenporter · 2024-05-18T04:49:34Z

homeassistant/components/google_generative_ai_conversation/conversation.py

            )
-            return conversation.ConversationResult(
-                response=intent_response, conversation_id=conversation_id
+            LOGGER.debug("Tool call: %s(%s)", tool_input.name, tool_input.tool_args)


tool_input.tool_name

tronikos · 2024-05-18T05:14:05Z

homeassistant/components/google_generative_ai_conversation/conversation.py

@@ -80,6 +135,11 @@ async def async_process(
        self, user_input: conversation.ConversationInput
    ) -> conversation.ConversationResult:
        """Process a sentence."""
+        tools: list[dict[str, Any]] | None = None


Do the tools need to be constructed on every interaction with the conversation entity? How about constructing them in init once and maybe even share them among the multiple conversation entities?

It's up to the LLM API objects to decide if they want to cache them. Here we just fetch them.

homeassistant/components/google_generative_ai_conversation/conversation.py

homeassistant/components/google_generative_ai_conversation/config_flow.py

tronikos · 2024-05-18T05:59:07Z

Don't forget to update the documentation and link to it here. It says "This conversation agent is unable to control your house"

balloob · 2024-05-18T16:06:06Z

Marked as draft as we want to change the LLM helper slightly.

tronikos · 2024-05-19T05:05:04Z

homeassistant/components/google_generative_ai_conversation/__init__.py

-                genai.get_model, entry.options.get(CONF_CHAT_MODEL, DEFAULT_CHAT_MODEL)
-            )
-        )
+        await hass.async_add_executor_job(partial(genai.get_model, DEFAULT_CHAT_MODEL))


Why is that bad? Isn't it better to fail as soon as possible?

tronikos · 2024-05-19T05:08:27Z

homeassistant/components/google_generative_ai_conversation/config_flow.py

+                "models/gemini-1.0-pro",  # duplicate of gemini-pro
+                "models/gemini-1.5-flash-latest",
+            )
+            and "Vision" not in api_model.display_name


Not sure if the API is translating the display name. Seems safer to check api_model.name instead.

…t#117644) * initial commit * Undo prompt chenges * Move format_tool out of the class * Only catch HomeAssistantError and vol.Invalid * Add config flow option * Fix type * Add translation * Allow changing API access from options flow * Allow model picking * Remove allowing HASS Access in main flow * Move model to the top in options flow * Make prompt conditional based on API access * convert only once to dict * Reduce debug logging * Update title * re-order models * Address comments * Move things * Update labels * Add tool call tests * coverage * Use LLM APIs * Fixes * Address comments * Reinstate the title to not break entity name --------- Co-authored-by: Paulus Schoutsen <balloob@gmail.com>

homeassistant/helpers/llm.py

tests/components/google_generative_ai_conversation/test_conversation.py

Shulyaka added 3 commits May 16, 2024 12:20

initial commit

1e4e12d

Undo prompt chenges

37a4faf

Move format_tool out of the class

d9bd295

home-assistant bot added cla-signed integration: google_generative_ai_conversation new-feature Quality Scale: No score labels May 17, 2024

Shulyaka and others added 10 commits May 18, 2024 00:44

Merge branch 'dev' into google_ai_tools

07f1fdc

Only catch HomeAssistantError and vol.Invalid

2e62688

Add config flow option

a8f0445

Fix type

3c2cd07

Add translation

68d7c8f

Allow changing API access from options flow

c9f9385

Allow model picking

e3fd6de

Remove allowing HASS Access in main flow

2aaed5b

Move model to the top in options flow

d71fca7

Make prompt conditional based on API access

a98e801

balloob reviewed May 18, 2024

View reviewed changes

balloob added 2 commits May 18, 2024 04:21

convert only once to dict

cf2b230

Reduce debug logging

56c5e99

balloob marked this pull request as ready for review May 18, 2024 04:23

balloob requested a review from a team as a code owner May 18, 2024 04:23

Update title

cb4f0e1

allenporter requested changes May 18, 2024

View reviewed changes

home-assistant bot marked this pull request as draft May 18, 2024 04:40

balloob added 3 commits May 18, 2024 04:41

re-order models

6f39352

Address comments

fa81db5

Move things

01e57ee

balloob marked this pull request as ready for review May 18, 2024 04:45

home-assistant bot requested a review from allenporter May 18, 2024 04:45

allenporter approved these changes May 18, 2024

View reviewed changes

allenporter reviewed May 18, 2024

View reviewed changes

Update labels

0ae2f2b

allenporter approved these changes May 18, 2024

View reviewed changes

tronikos suggested changes May 18, 2024

View reviewed changes

balloob marked this pull request as draft May 18, 2024 16:01

Shulyaka and others added 5 commits May 18, 2024 19:06

Add tool call tests

b076a2d

Merge branch 'dev' into google_ai_tools

d582426

coverage

b8f2db9

Merge remote-tracking branch 'origin/dev' into google_ai_tools

5358aaa

Use LLM APIs

581d6d0

balloob marked this pull request as ready for review May 19, 2024 02:59

home-assistant bot requested review from allenporter and tronikos May 19, 2024 02:59

balloob mentioned this pull request May 19, 2024

Update Google AI to allow HASS access home-assistant/home-assistant.io#32839

Merged

8 tasks

Fixes

f59fa45

tronikos approved these changes May 19, 2024

View reviewed changes

balloob added 2 commits May 20, 2024 01:42

Address comments

40d927f

Reinstate the title to not break entity name

7b49680

balloob merged commit c3196a5 into home-assistant:dev May 20, 2024
33 of 34 checks passed

github-actions bot locked and limited conversation to collaborators May 21, 2024

MartinHjelmare reviewed May 21, 2024

View reviewed changes

homeassistant/helpers/llm.py Show resolved Hide resolved

tests/components/google_generative_ai_conversation/test_conversation.py Show resolved Hide resolved

home-assistant unlocked this conversation May 21, 2024

balloob mentioned this pull request May 21, 2024

Address late feedback Google LLM #117873

Merged

20 tasks

github-actions bot locked and limited conversation to collaborators May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LLM Tools support for Google Generative AI integration #117644

LLM Tools support for Google Generative AI integration #117644

Shulyaka commented May 17, 2024 •

edited by balloob

home-assistant bot commented May 17, 2024

balloob May 18, 2024

tronikos May 19, 2024

balloob May 19, 2024

balloob May 19, 2024

home-assistant bot commented May 18, 2024

allenporter May 18, 2024

tronikos May 18, 2024

balloob May 19, 2024

tronikos commented May 18, 2024

balloob commented May 18, 2024

tronikos May 19, 2024

tronikos May 19, 2024

LLM Tools support for Google Generative AI integration #117644

LLM Tools support for Google Generative AI integration #117644

Conversation

Shulyaka commented May 17, 2024 • edited by balloob

Proposed change

Type of change

Additional information

Checklist

home-assistant bot commented May 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

home-assistant bot commented May 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tronikos commented May 18, 2024

balloob commented May 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shulyaka commented May 17, 2024 •

edited by balloob