Status Update: 2023-01-03

[haixuanTao]

🎆

[haixuanTao]

Dora-Mineflyer

-> Fixed one issue about FOV of minecraft bot, to unlock Computer vision potential and multi-modal LLM potential by using Selenium on the web.

Voyager: Due to the Mineflayer's limitation, we currently can not directly get the bot's view in the game. Although there's a plugin called prismarine-viewer, the video quality is not good enough, so we opt not to use it. Our demo video is generated by replay-mod. We start the recording and let the bot play for hours, then come back to the recording and render the view from the bot.

This in the future can be used for any web-automation but also can be the premise for any-game automation using multi-modal llm.

See: dora-rs/dora-mineflayer@359414d

[haixuanTao]

Kind of like the Google Demo

[haixuanTao]

Dora-Robomaster

-> Fixed an issue with the webcam having weird freezes by directly using opencv-python ffmpeg h264 stream reader within Python. This was due to lost packet and Robomaster python based EPCamera modules.

• In the future we can also think about how to use rust: opencv-rust to add more performance. We can for example integrate hardware acceleration as well as better async IO listening.
  -> https://github.com/dora-rs/dora-robomaster/blob/main/operators/opencv_stream.py

Note that we don't have any dependencies on Robomaster
h264 video stream also support fast audio + video stream which could be useful if we want to apply whisper on a remote track

[haixuanTao]

dora-rs

-> Fixed alias installation issue: #399
-> and in the way to fix couples of issues with dora on Windows: #398

[haixuanTao]

Upcoming Demos

• CentraleSupelec LLM Demos ( 17-01-2024 )
• Dora-Mineflyer ( 31-01-2024 )
• Dora FOSDEM ( 03-02-2024 )

[haixuanTao]

See: https://fosdem.org/2024/schedule/event/fosdem-2024-3225-dora-rs-simplifying-robotics-stack-for-next-gen-robots/