Chat module issues & enhancements list

[rmackay9]

This is a consolidated list of issues and enhancements for the MAVProxy chat module added with PR #1278

Bugs:

• error when reloading modules (issue)
• typing "chat show" into the MAVProxy produces an innocuous error. We should either remove that completely or make it re-show the chat GUI in case the user has closed it

Enhancements:

• get around OpenAI unreliability in accessing files.
• Store thread-id and allow users to re-use the thread (perhaps this should be the default)
• Voice-to-text recording is currently hard-coded to 5sec. We should change this to a "Listen". So when the user pushes the button it will stay depressed and record when the volume is over a given threshold. Once it detects the end of a sentence (e.g. volume drops below the threshold for maybe 2 or 3 seconds) it sends to openai.
• Add version support so that chat module can find compatible assistants (e.g. append "VerX.Y" to the assistant name and use during init)
• Adding a Cancel button to cancel a run that's been sent to the assistant.
• Add function to allow AI to send arbitrary mavlink messages to the vehicle
• Support "streaming" which is now available from OpenAI (see docs here)
• Support multiple vehicles

References:

• MAVProxy wiki is here
• ArduPilot "ai" discord channel is a good place for real-time chatting
• OpenAI's API reference: https://platform.openai.com/docs/api-reference/assistants

Resolved items:

• pass parameter units, range, human-readable-text with each parameter request? PR: Chat: relative move improvements and other enhancements #1306
• try to reduce lag by lowering the "max_tokens" value sent with requests (see reference here) -- seems out-of-date
• Handle OpenAI API Errors and display a better warning to the user (see docs here). Common errors include AuthenticationError and RateLimitError
• Allow AI to retrieve and set parameters (may require uploading vehicle specific parameter description file)
• Add function to allow AI to retrieve the latest mavlink message(s) sent by the vehicle (probably using something like, mpstate.master().messages.get('HEARTBEAT', None)) -- completed Chat: improved feedback, parameter get/set and get latest mavlink support #1286
• Improve reliability of takeoff command and "please fly here". -- see PR Chat: improve takeoff and fly-to commands #1283
• investigate occasional freezing where the Input text box remains greyed out probably waiting endlessly for a reply from the Assistant that never comes. We should probably add a status bar to the bottom of the screen and display the "latest_run.status". -- not seeing anymore, perhaps was a server side issue
• Under Windows the user input text is grey instead of red meaning it is slightly more difficult to see what is your text and what is the chatbot's reply. -- fixed by Chat: minor GUI and assistant setup enhanecments #1288

[YousefYaghoobi]

Hi Randy.
Thanks for such an interesting project.

I have tried to follow along with this project.
After installation and using the ''CHAT" module:

• when I ask "What vehicle am I using?" or "What is the vehicle location? the answers are correct;
• when I try to "ARM" the vehicle or change the vehicle "MODE" I get the below errors:
  • "chat: handle_function_call: unrecognised function call: get_mode_mapping"
  • "Got COMMAND_ACK: COMPONENT_ARM_DISARM: UNSUPPORTED" (when I do the force arming)

I have tried for both SITL (ARDUCOPTER & ROVER) and real ROVER which is running on PIXHAWK 6C.

I was not sure whether or not to open a new issue.
Your help is highly appreciated.

[rmackay9]

Hi @YousefYaghoobi,

The best place to discuss this is actually in the ArduPilot Discord server's "ai" channel. In any case I think the issue is just that I've made changes that require updating the Assistant. I think if you run "setup_assistant.py --upgrade" it should work.

[YousefYaghoobi]

Thanks for your quick reply Randy.

Noted:

The best place to discuss this is actually in the ArduPilot Discord server's "ai" channel.

I did run "setup_assistant.py --upgrade" but still getting the same results.

[rmackay9]

Hi @YousefYaghoobi,

This really isn't the best place to have these conversations. Can you post in the ai chat going forward please?

Thinking about this more, the issue must be that you're not running the latest MAVProxy. Can you upgrade?

[YousefYaghoobi]

Sure.