proposal: better structure, stateless Client, and Plugin system

[nekomeowww]

We are facing difficulties to support various of inputs, processors, middlewares, and message bridge of different need, and customization, we need better structure for abstraction with scopes and division of files in this project. And also, the verification capabilities.

Introduction

Currently, we mixed all code of entire app within three directories:

• apps/stage-web: for Web specific impl
• apps/stage-tamagotchi: for Desktop specific impl
• packages/stage-ui: for shared components, stores, utils, types of both Web and Desktop.

and for Desktop (which is the so called tamagotchi) version, we have apps/stage-tamagotchi/src-tauri written in Rust with Tauri framework for cross platform support.

Renderers of models here:

• Stage.vue: the core is implemented here for stage of models
• VRM
• Live2D

But essentially, thanks for @sumimakito @luoling8192 and @kwaa, by referencing other impl of AI VTuber, and AI companion apps, this kind of structure may survive in long term...

Proposal

All thoughts behind the scene: https://www.tldraw.com/p/CbJDTPT0Jh17cbwas0yUJ?d=v-8525.-3980.24463.12690.brxzFGJvDgFYbKF3sP2Fw

I would love to propose a better structure similar to this with the following abstractions by introducing Primitives and data flows:

Warning

We should not rely on any Browser specific capabilities, except Scene, and UI. Or otherwise it's not possible to run AIRI in headless mode in long running servers.

Warning

When implementing the building blocks of the following Primitives, especially Input, Output, ModelProvider... except Scene, and UI, we should try to avoid to use any Vue specific library, or rely on Pinia to manage global states.

• Client
  • Scene
    • Renderer
      • Live2D
      • VRM
      • MMD
    • SceneControl
      • for Pixi
      • for three.js
    • BodyControl
      • in 2D
      • in 3D
  • Config
    • Schema
    • get, set, and update config
  • MessageChannel
    • process & exchange messages between modules over transports
    • in this way, we can even have both Tauri and Electron as runtime since they all in abstractions of MessageChannel when deals with IPC
      • IPC
      • WebSocket (implemented already)
      • WebRTC (someone might need but not for now)
      • HTTP RPC (someone might need but not for now)
      • gRPC (someone might need but not for now)
  • Input
    • types
      • AudioInput: wav/pcm
      • VideoInput: mp4
      • ImageInput: png/jpg
      • TextInput
  • InputProcessor
    • any Memory impl should live here to update memory from chat
  • Output
    • output can be selective to be displayed in UI, or instructions to MessageChannel
    • types
      • AudioOutput: wav/pcm
      • VideoOutput: mp4
      • ImageOutput: png/jpg
      • TextOutput
  • OutputProcessor
    • any Memory impl should live here to update memory from chat
    • Output processor is a bit hard and sync...
      • @sumimakito and me discussed internally and we may add more proposal for others to reference.
  • ModelProvider
    • VAD
    • TTS
    • ASR/STT
    • Chat
  • ModelProviderProcessor
    • If some plugin would love to modify or manipulate requests of ModelProviders, then choose this way
  • UI
    • Settings
      • declaration of UI of settings
    • Widgets
      • declaration of UI in chat, stage, and other places
    • Component Calling Components
• Server
  • InferenceBackend
    • works for ModelProvider, basically any form of ModelProvider, this release us from binding strictly to Tauri and not be able to migrate back to Electron due to several issues introduced from bug: cannot render WebGL canvas on Linux #263.

For what we described in the #255 proposal, plugin system is needed for many things to be able to work as integration from remote or external scripts. Such as translation, extra provider, more UI, MCP, OS operation, etc.

Input & output

With memory

Game playing

For game playing, the most complex thing, my concept of this is...

All games divide into two categories:

• API controllable (wrapped Client from API, such as Minecraft, Factorio we have implemented so far)
• Humanlike controllable (CV/OCR with Keyboard/Mouse/Gamepad input events, such as Helldivers 2, osu!, and Genshin Impact)

Perhaps we should use MessageChannel to exchange data and messages between services like Minecraft as events or result of tasks, voice chat from Discord, and internal instructions (destination parameter set to internal for Output) from AIRI.

Let's deep dive into the examples...

API controllable

Discord <-> [discord MessageChannel (text & voice)] <-> AIRI
Minecraft <-> [minecraft MessageChannel (text & API calls)] <-> AIRI

data flows:

Client UI -> [chat] -> AIRI
Minecraft console -> [chat] -> AIRI
Discord voice chat -> [chat (transcribed] -> AIRI

and AIRI will produce instructions from gathered data from plugins, memory, and many other plugins, modules.

graph TD
    subgraph "InputSource"
        UI[Client UI]
        MC[Minecraft Console]
        DC[Discord Voice Chat]
    end
    
    subgraph "MessageChannel Bridges"
        DCChannel[Discord MessageChannel<br/>text & voice]
        MCChannel[Minecraft MessageChannel<br/>text & API calls]
    end
    
    subgraph "AIRI "
        AIRI[AIRI Core]
        LLM[LLM ModelProvider]
        Memory[Memory & Plugins]
    end
    
    subgraph "Gameplay agent"
        MCPlugin[Minecraft Gameplay Plugin]
        MCInvoke[Minecraft API]
    end
    
    UI -->|chat| AIRI
    MC -->|console messages| MCChannel
    DC -->|voice transcribed| DCChannel
    MCChannel -->|chat| AIRI
    DCChannel -->|chat| AIRI
    
    AIRI --> LLM
    LLM --> Memory
    Memory -->|Internal instruction| MCPlugin
    MCPlugin --> MCInvoke
    
    AIRI <--> DCChannel
    AIRI <--> MCChannel

Loading

Humanlike controllable

data flows:

[CV/OCR agent] -> [llm ModelProvider (optional)] -> [throttled & aggregated discovered metrics & features] -> [TextInput] -> AIRI

and AIRI will produce instructions from gathered data from plugins, memory, and many other plugins, modules.

But since not every community supports to have a AI to play game, we may have two different ways to build it:

No, we will never support any form of cheating, game playing is fun, and enjoy, AI companion should support human players in the first place... in any time.

So.. most of the time, I mean the competitive games like League of Legends, DOTA 2, COD, and perhaps, War thunder, World of Warships, etc., we will leave this for community to judge as this somehow violates the moral in game playing in the described competitive games.

• If not competitive games, then it's Keyboard/Mouse/Gamepad simulate-able
• If competitive games, then chat will be used, we could have simple chat, tactical support, guidance, and encouragement for those games.

If AI playing is acceptable

AIRI -> [llm ModelProvider] -> Internal instruction -> [CV/OCR with Keyboard/Mouse/Gamepad agent] -> Actions / Hardware Inputs

or in mermaid:

graph TD
    subgraph "Vision Input"
        Screen[Screen Capture]
        CVAgent[CV/OCR Agent]
    end
    
    subgraph "AIRI "
        LLM1[LLM ModelProvider<br/>optional]
        Metrics[throttled & aggregated discovered metrics & features]
        TextInput[TextInput]
        AIRI1[AIRI]
        LLM2[LLM ModelProvider]
    end
    
    subgraph "Gameplay agent"
        InternalInst[Internal Instruction]
        InputAgent[CV/OCR with<br/>Keyboard/Mouse/Gamepad Agent]
        Actions[Actions / Hardware Inputs]
    end
    
    Screen --> CVAgent
    CVAgent --> LLM1
    LLM1 --> Metrics
    Metrics --> TextInput
    TextInput --> AIRI1
    AIRI1 --> LLM2
    LLM2 --> InternalInst
    InternalInst --> InputAgent
    InputAgent --> Actions

Loading

Otherwise

AIRI -> [llm ModelProvider] -> Internal instruction -> [tool calls to search game wiki, community, forums, walkthrough helpers, etc.] -> [Output]

or in mermaid

graph TD
    subgraph "Vision input"
        Screen2[Screen Capture]
        CVAgent2[CV/OCR Agent]
    end
    
    subgraph "AIRI "
        LLM3[LLM ModelProvider<br/>optional]
        Metrics2[throttled & aggregated discovered metrics & features]
        TextInput2[TextInput]
        AIRI2[AIRI]
        LLM4[LLM ModelProvider]
    end
    
    subgraph "Tool call"
        InternalInst2[Internal Instruction]
        Tools[Tool Calls:<br/>search game wiki, community, forums, walkthrough helpers, etc.]
        Output[TextOutput or to AudioOutput]
    end
    
    Screen2 --> CVAgent2
    CVAgent2 --> LLM3
    LLM3 --> Metrics2
    Metrics2 --> TextInput2
    TextInput2 --> AIRI2
    AIRI2 --> LLM4
    LLM4 --> InternalInst2
    InternalInst2 --> Tools
    Tools --> Output

Loading

[nekomeowww]

I tried my best to summarize everything I could imagine and support in the current time, please challenge me for any missing parts @typed-sigterm @sumimakito @Lilia-Chen.

Perhaps, it'd better to be reviewed by you @luoling8192 @LemonNekoGH, see if this structure works for Factorio, and Minecraft.
Perhaps, since @skirkru has been working on providers for a while, and tested through many parts, for ModelProviders, please give me some review of this proposal so we could better improve this.

The very first thing I could imagine, is to slowly split out

• Renderers (make them stateless to not to depend on pinia stores, but I believe Renderers and AvatarProvider may have their own config panels besides Config for Settings specifically, it's ok to have dedicated UI config for Renderers)
• Pipelines needed for chaining MessageChannels, ModelProviders, and Input, Output, InputProcessor, as well as OutputProcessor...
• I will finish the already WIP hearing and voiceover for AudioInput and AudioOutput based on the current structure but better API and syncing capabilities of subtitles, leave me at least some APIs to control how Output renders to UI chat when outputting.

[ilya-id]

@nekomeowww Thank you for putting this comprehensive proposal together.

My assessment is that foundational stability, particularly with the core providers and chat interface, must be the immediate priority. An ambitious architectural shift like this one depends entirely on that stable base.

So, I think our main focus should be securing the fundamentals first before anything else, our selling point first.

[Misaka-0x447f]

Translations does not needs to be plugin, at least in very low priority. Reason is less built-in translation may not significantly reduce package size, and code review time cost for such pr is low.
For memory, why not make it an MCP?

[nekomeowww]

No. I mean a Translator (through API) plugin where it will be used to help Chinese player to use 中文 to do role playing, messages will be translated before other pipelines before chat, once chat llm responded, extracted, then, translate it again back to Chinese. Then the data can be carried on for other pipelines.

It's not essential. But it is simple, and demonstrated the way easy, and deterministic when verifying from human perspective.

[nekomeowww]

@nekomeowww Thank you for putting this comprehensive proposal together.

My assessment is that foundational stability, particularly with the core providers and chat interface, must be the immediate priority. An ambitious architectural shift like this one depends entirely on that stable base.

So, I think our main focus should be securing the fundamentals first before anything else, our selling point first.

I agree, so I will focus on the input audio & speech transcription part. Someone could take a look at TTS for now.

[Lilia-Chen]

Just wondering, what's the current TODO/issues for TTS? It's... workable... but what's the next step?

[sumimakito]

Nice write-up.

I wrote and deleted much in the past hours, simulating the real-world use case of the architecture we planned here, with some modifications, but I discarded them because I'm unsure if they actually work. (Maybe because I'm feeling under the weather 🤒)

We will eventually need plugins for extensibility, but before we actually start working on the plugin system itself, we can introduce the globally managed pipelines first–for example, we currently have queues for input and output coupled tightly in the Stage components instead of a globally accessible one.

We can have many input sources, but the types of data always fall into a few definite ones that the models support. Therefore, we can separate pipelines by the data type they input/output. This is also useful when we want to interrupt a specific type of input/output, like the issue we had recently with the audio output.

After we have build the pipelines, the real use case becomes clearer, and that would also be helpful for deciding what are needed if we now want to create inputs from plugins.

For example, we can implement the most basic input and output pipeline for chat apps nowadays–mic-in, text-in, TTS-out, and text-out. Then, we can start working on splitting them into plugins that defines inputs/outputs, and supporting more input sources/output processors.

BTW, I will refactor some code for TTS and the queue soon.