feat: add spatialreal avatar plugin

[3DRX]

This PR adds SpatialReal avatar plugin, we've been developing it for a while at https://github.com/spatialwalk/livekit-plugins-spatialreal, and publish packages to https://pypi.org/project/livekit-plugins-spatialreal.

Our example project can be found at https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-agents. For a deployed demo, please checkout SpatialReal playground, which is built using livekit agents and SpatialReal avatar using this plugin.

This plugin have https://pypi.org/project/avatarkit/ as dependency, which is our python SDK, it's a thin layer that handles websocket connection and protobuf messages.

During this PR I also make changes according to the ai review, I personally tested these changes from the example project with no regressions found.

Another thing I'd like to discuss is pypi package transfer, I think it is good for developers (less confusion) if we can still publish release to the same pypi package afther this plugin gets merged, and we are willing to transfer the ownership of the pypi package to livekit organization. Please let me know what should I do :)

We have early adopters shipping SpatialReal avatar in production using other integration methods for months, and recently we have developers using this plugin in their product for about a week, during the process we've fixed all known issues.

[3DRX]

Update, just rebased to resolve uv.lock conflict. No code change.

[3DRX]

Hi @tinalenguyen , are there anything I can help / change to make it easier for this patch to be merged?

(The conflict is minor, just because of new package and dependency introduction, and can be easily resolved if we are ready to merge this).

[tinalenguyen]

Hi @3DRX, thanks for following up and creating the PR! I tested it out and encountered this error:

 agents\examples\avatar_agents\spatialreal\agent_worker.py", line 34, in entrypoint
    await spatialreal_avatar.start(session, room=ctx.room)
 agents\livekit-plugins\livekit-plugins-spatialreal\livekit\plugins\spatialreal\avatar.py", line 191, in start
    agent_participant_identity = room.local_participant.identity
                                 ^^^^^^^^^^^^^^^^^^^^^^
 venv\Lib\site-packages\livekit\rtc\room.py", line 205, in local_participant
    raise Exception("cannot access local participant before connecting")
Exception: cannot access local participant before connecting

I see that you use ingress/egress for the audio flow, is there a reason why the avatar doesn't publish data directly to the room? It should be faster that way

[3DRX]

Hi @3DRX, thanks for following up and creating the PR! I tested it out and encountered this error:

 agents\examples\avatar_agents\spatialreal\agent_worker.py", line 34, in entrypoint
    await spatialreal_avatar.start(session, room=ctx.room)
 agents\livekit-plugins\livekit-plugins-spatialreal\livekit\plugins\spatialreal\avatar.py", line 191, in start
    agent_participant_identity = room.local_participant.identity
                                 ^^^^^^^^^^^^^^^^^^^^^^
 venv\Lib\site-packages\livekit\rtc\room.py", line 205, in local_participant
    raise Exception("cannot access local participant before connecting")
Exception: cannot access local participant before connecting

I see that you use ingress/egress for the audio flow, is there a reason why the avatar doesn't publish data directly to the room? It should be faster that way

Hi @tinalenguyen thanks for the review. The problem you encountered is because await ctx.connect() needs to be called before calling spatialreal_avatar.start(). I just updated the example. Also since SpatialReal renders avatar at client side, standard video player won't work, a client SDK is needed. You may want to checkout https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-agents.

Regarding ingress/egress, it's an unfortunate naming collision. The ingress endpoint of spatialreal and egress config field does not mean ingress/egress in livekit. In livekit's context they mean avatar worker (ingesting agent's audio output, and publish audio & avatar animation to the same livekit room). Currently SpatialReal avatar worker will publish data directly to the room, we are using the golang SDK :)

[3DRX]

Regarding ingress/egress, it's an unfortunate naming collision. The ingress endpoint of spatialreal and egress config field does not mean ingress/egress in livekit.

I think I need to figure out a better way to name/communicate this, to avoid misunderstanding.

[tinalenguyen]

@3DRX Thank you for the context, I see now! Typically that logic exists server side so users wouldn't see those ingress/egress endpoints

I tested it out and I wasn't able to see the avatar join the room at all, but audio was published. Could you pull from main and bump the versions to see if that helps?

Also, is there a reason why livekit_egress doesn't accept a JWT token instead of LiveKit API secret and key? It would be good to scope credentials to each room that way. I also don't think you need to set audio_output to False in the RoomOptions with that approach