Integrate plugins into chat by diegoasua · Pull Request #276 · BasedHardware/omi

diegoasua · 2024-06-13T20:53:24Z

First take at integrating plugins into chat.

josancamon19 · 2024-06-14T06:08:13Z

-String qaStreamedBody(String context, List<Message> chatHistory) {
+String qaStreamedBody(String personality, String context, List<Message> chatHistory) {
  var prompt = '''
+    This is your personality, you should try to answer in this style: $personality


Can we do this part of the prompt optional?

josancamon19 · 2024-06-14T06:09:43Z

    );
  }

+  // FIXME: only supports 1 active plugin. When multiple are active uses the first one


nice, is there a way we could have a selector on the chat page so people can select which plugin to chat, maybe like a floating dropdown top right or smth

I don't get why more than one plugin can be active at a time. It makes more sense to only allow 1 plugin active. Also for the memories. Is there a reason several plugins can be active simultaneously?

Ideally plugins are used only whenever they are useful, so if you are having a medical conversation, it uses x plugin, entrepreneurship, but that requires better plugin prompt, maybe you could take a look so we filter out useless plugins per memory. It's complex to expect the user will remind before every conversation what plugin they might need, so if you activate many you are set and forget

btw, how can I find you on discord? would love to chat

for the multiple plugin case. We can pass all personalities to the system prompt and ask the LLM to choose which one is most relevant for the current conversation. This will fulfill the promise to the user they don't have to remember turning plugins on and off. They choose what they want, and chat layer is flexible enough to understand and adapt. Reduces strain on what is asked by us from the user improving overall UX.

I still think we should limit how many plugins are active at the same time. The more active plugins the more chances of this failing/confusing the LLM. As a measure for now we can limit active plugins to 5.

josancamon19 · 2024-06-14T06:10:06Z

Looking great so far!

diegoasua · 2024-06-16T15:55:03Z

I am removing BLE scan by name even though it's out out of scope of this PR as we identified here this change was undone #225 (comment)

diegoasua · 2024-06-17T12:23:41Z

Alright this is ready to review @josancamon19 I tested with a single and multiple plugins. I tested the Paul Graham plugin together with other two and asked questions from my memories about neural networks and seems that it selected the right plugin in chat, and the right memory.

Now that being said chat will only be as good with plugins if the plugin prompts are well written. The example here is not great for the purpose of steering behavior in a conversation.

The prompt is good for generating memories but not so much for driving the personality of the conversation. I see two options.

We ask that new plugins have a new field personality that drives the conversation. An example of a good personality plugin would be:
You are Shakespeare. You will talk in Shakesperian ways, poetic and using old English words here and there. You will not deviate from this character
We would have to add this field to the existing ones, but there are only like 10 at the moment.
We dynamically generate the personality field by asking an LLM to do it when the plugin is submitted to the database. We pass the plugin and ask it to add the new field with a few shot examples. For me this would be the preferred method.

The prompt itself gives instructions on how to process memories not in how to behave the conversation. LLMs can infer what you mean, but it will drive their behavior stronger if they are actually given a personality based on those instructions.

josancamon19 · 2024-06-17T18:40:53Z

Will review today @diegoasua

josancamon19 · 2024-06-19T05:56:22Z

 Future<List<BTDeviceStruct?>> scanDevices() async {
  List<BTDeviceStruct> foundDevices = await bleFindDevices();
-  var filteredDevices = foundDevices.where((device) => device.name == "Friend").toList();
+  // var filteredDevices = foundDevices.where((device) => device.name == "Friend").toList();


What is the purpose of this change?

#276 (comment)

see here for more discussion #225 (comment)

This would show all ble available devices right? as far as I understand, if both options work, why to do it like this is what I don't understand

No it will not show all BLE devices. If the device is named different from "Friend" it will not show up. When that line is active it is filtering BLE devices by name. It will go through the onboarding screen with a real device as long as its name is "Friend" but not with a simulator. Recently we stopped filtering devices by name and use UUID instead. So this line is not only unnecessary but also prevents us from pairing a simulator.

See #225 (comment)

josancamon19 · 2024-06-20T01:31:34Z

-    await streamApiResponse(ragContext, _callbackFunctionChatStreaming(memoryIds), widget.messages, () {
-      widget.messages.last.memoryIds = memoryIds;
-      prefs.chatMessages = widget.messages;
+    //     await streamApiResponse(ragContext, _callbackFunctionChatStreaming(memoryIds), widget.messages, () {


let's remove this

To be clear, you mean remove the commented out part?

josancamon19 · 2024-06-20T01:32:08Z

 }

-String qaStreamedBody(String context, List<Message> chatHistory) {
+String qaStreamedBody(dynamic personality, String context, List<Message> chatHistory) {


For now let's use the description as their personality and reformat the plugins so the description has this personality (ish).

josancamon19 · 2024-06-20T01:34:14Z

Also, let's add on chat a plugin selector.

Basically, there'd be one of the plugins to select, in a dropdown in the chat in some way, but not automatically, initially let's do it like this, and see how people interact with it

diegoasua · 2024-06-20T19:47:24Z

Alright now there is a dropdown in chat that lets you choose which personality to chat with from the list of downloaded plugins. you can only choose one. Give it a spin and let me know. I have only tested it in iOS so it would be good if someone can give it a go in Android too.

@josancamon19 ready for review

josancamon19 · 2024-06-21T23:00:55Z

 Future<List<BTDeviceStruct?>> scanDevices() async {
  List<BTDeviceStruct> foundDevices = await bleFindDevices();
-  var filteredDevices = foundDevices.where((device) => device.name == "Friend").toList();
+  // var filteredDevices = foundDevices.where((device) => device.name == "Friend").toList();


This would show all ble available devices right? as far as I understand, if both options work, why to do it like this is what I don't understand

josancamon19 · 2024-06-21T23:25:49Z

 }

-String qaStreamedBody(String context, List<Message> chatHistory) {
+String qaStreamedBody(


The only linting rule should be 120 characters line and default android studio rules

Please relint

This would show all ble available devices right? as far as I understand, if both options work, why to do it like this is what I don't understand

I answered this above but I want to make sure this is clear, so copy pasting the response here again:

No it will not show all BLE devices. If the device is named different from "Friend" it will not show up. When that line is active it is filtering BLE devices by name. It will go through the onboarding screen with a real device as long as its name is "Friend" but not with a simulator. Recently we stopped filtering devices by name and use UUID instead. So this line is not only unnecessary but also prevents us from pairing a simulator.
See #225 (comment)

re: linting

ack that will apply that rule

I wanted to add, Dart docs suggests we should not use more than 80 characters per line as it affects readability.

josancamon19 · 2024-06-21T23:29:30Z

@@ -14,6 +14,8 @@ import 'package:friend_private/backend/preferences.dart';
 import 'package:friend_private/pages/chat/widgets/ai_message.dart';


This should be a dropdown that goes on the Appbar only when chat page is selected.

No bottom sheet, a dropdown.

It should contain the plugin name + plugin picture.

It should look good, and match the current design.

When there are no plugins enabled it should have that default dropdown item, and a 2nd one with enable plugins click, that opens the plugins page.

sounds good. I appreciate these more clear deliverables. They guide development better

diegoasua · 2024-06-22T16:53:02Z

ready for another round of reviews @josancamon19 . What's new:

Moved selection to dropdown in AppBar
Added two default fields for when plugins are deactivated: default and "enable plugins". The later takes you to the plugin page
Followed app design system
Show default and enable plugins when no plugins downloaded
clicking on enable plugins takes you to the plugin page
plugins show image in selection

josancamon19 · 2024-06-22T20:22:43Z

                ),
              ),
              // Text(['Memories', 'Device', 'Chat'][_selectedIndex]),
+              if (_controller!.index == 2)


This looks bad, I don't see how this fits with current designs besides de border gradient.

And let's please remove the padding GPT comments.

Put this in a separate widget.

Finally, please put this component in a separate function, could be inside same file.

Remove the "Default" and put only "View Plugins"

Also, messages answered with plugin personality should have the plugin id as optional field in db, and instead of the message sender icon we have on the left of ai message it should use the plugin image as the sender.

Please explain how you want the widget to look like. If you don't set specific deliverables I don't know what you want. You say it looks ugly and doesn't fit in current app designs. Is it the font? Spacing? Size? Palette? I can iterate on it but I need to know what you want. Otherwise this is not going to work. Please advise

josancamon19 · 2024-06-26T23:08:46Z

Closing PR, as I don't see significant progress, and reassigning ticket.

feat: first steps

75905b3

diegoasua mentioned this pull request Jun 13, 2024

Allow plugins to integrate with chat ($500) #223

Closed

josancamon19 reviewed Jun 14, 2024

View reviewed changes

diegoasua added 4 commits June 14, 2024 21:54

fix: revert system prompt, empty prompt for no plugin

b3ac1f8

feat: support multiple active plugins

878945a

Merge branch 'main' into diego/plugin-chat-integration

f1fd9bc

fix: remove BLE scan by name

18685da

diegoasua added 2 commits June 17, 2024 19:06

fix: improve multi-plugin personality selection

02d2f35

fix: make Dart static analyzer happy

52c423a

Merge branch 'main' into diego/plugin-chat-integration

01bc635

josancamon19 reviewed Jun 20, 2024

View reviewed changes

diegoasua added 8 commits June 20, 2024 20:58

Merge branch 'main' into diego/plugin-chat-integration

3205e42

UI: plugin select dropdown

11278cf

fix: dropdown location, display

796c780

rounded modal UI and better font

06f8398

selectable entries in modal

3694602

pass down only 1 personality to LLM call

7b9bc45

fix: adding removed

cc871d5

prompt improvement

1e04aa1

diegoasua added 2 commits June 21, 2024 14:18

Merge branch 'main' into diego/plugin-chat-integration

c8e662d

fix: main breaking changes merge

a4da1de

josancamon19 suggested changes Jun 21, 2024

View reviewed changes

diegoasua added 2 commits June 22, 2024 18:59

dropdown in AppBar with images

888b542

fix: positioning, sync with settings

bc27f26

diegoasua added 6 commits June 22, 2024 19:50

feat: default and enable plugins as options in dropdown

e0c57e5

fix: dropdown width default

b2358d8

fix: width when plugins not selected

bdd034d

go to plugin page when clicked in enable plugin

b14835a

dropdown UI makeover, design system guidelines

c2713f8

fix: bugs with plugin dropdown update

ad85454

diegoasua added 3 commits June 22, 2024 23:53

fix UI error

aea9ebc

Merge branch 'main' into diego/plugin-chat-integration

63d0aca

small fix

38dc7b2

josancamon19 suggested changes Jun 22, 2024

View reviewed changes

Merge branch 'main' into diego/plugin-chat-integration

5d83683

josancamon19 closed this Jun 26, 2024

		@@ -14,6 +14,8 @@ import 'package:friend_private/backend/preferences.dart';
		import 'package:friend_private/pages/chat/widgets/ai_message.dart';

Conversation

diegoasua commented Jun 13, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josancamon19 commented Jun 14, 2024

Uh oh!

diegoasua commented Jun 16, 2024

Uh oh!

diegoasua commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josancamon19 commented Jun 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josancamon19 commented Jun 20, 2024

Uh oh!

diegoasua commented Jun 20, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diegoasua Jun 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diegoasua Jun 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diegoasua commented Jun 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josancamon19 commented Jun 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

diegoasua commented Jun 17, 2024 •

edited

Loading

diegoasua Jun 22, 2024 •

edited

Loading

diegoasua Jun 22, 2024 •

edited

Loading