add bey avatar example by niqodea · Pull Request #1669 · livekit/agents

niqodea · 2025-03-17T16:42:59Z

Add resources for the Beyond Presence API.

livekit-agents-bey: plugin to handle API calls and local setup for avatar generation
examples/avatar/bey: a basic script demonstrating how to use the API via the plugin

Marking this as a draft since the API is not live yet. Feedback on integration or improvements is welcome!

Note: we reserved the livekit-plugins-bey PyPI package name, let me know if I should add someone from the LiveKit team as owner. 🙏

changeset-bot · 2025-03-17T16:43:05Z

⚠️ No Changeset found

Latest commit: 7b8acb3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

CLAassistant · 2025-03-17T16:43:08Z

All committers have signed the CLA.

niqodea · 2025-03-17T16:45:29Z

livekit-plugins/livekit-plugins-bey/livekit/plugins/bey/core.py

+        # allow your local agent to publish transcripts on behalf of the avatar agent
+        .with_attributes({ATTRIBUTE_PUBLISH_ON_BEHALF: ctx.room.local_participant.identity})


Not sure this comment is correct

I think it's to mark that the avatar agent publishes video and audio on behalf of the agent.

@longcw maybe related to our prior discussion on #1424: Is there a canonical way for hiding one of the call participants now such that the user has the experience of a 1:1 call?

How we implemented the plugin here is that only the avatar worker publishes to the room such that the agent can be hidden client side later, and we thought this line above allows the agent to publish transcripts in the name of the avatar.

However, from your comment it sounds like the intended way now is to somehow have the avatar publish audio/video in the name of the agent and then have the avatar worker hidden?

Maybe you can clarify this and give us some suggestions for best practices around this once you review this file

Ideally it works in the latter way you described, the avatar publish audio/video in the name of the agent and then have the avatar worker hidden.

The client can detect this by reading the ATTRIBUTE_PUBLISH_ON_BEHALF attribute and then handle the avatar participant as designed.

The benefit is that the other operations related to the agent can keep unchanged, e.g. perform a RPC call, send a text or file through data stream to the agent. These operations need a dest participant identity, using the agent identity is straightforward.

I see the advantage of keeping the other operations unchanged but I guess the main thing we want to achieve is that we can speak to an avatar as a user right? Do you have an example of how the user could see the video through the "agent worker" in the frontend? We're using ATTRIBUTE_PUBLISH_ON_BEHALF like in your latest example on the dev branch https://github.com/livekit/agents/blob/dev-1.0/examples/avatar/agent_worker.py#L48 but in the end we get our "Avatar Worker" who outputs video and audio (see screenshot). The other "agent worker" neither outputs audio or video and I guess is just there to forward the user audio. So we're currently hiding that one (agent worker) on the frontend. So does the ATTRIBUTE_PUBLISH_ON_BEHALF config not work properly? I.e. should the "agent worker" receive the audio and video we send through the "avatar worker"?

It needs some modification on the frontend client. For example, if you using livekit playground it will only show the avatar video without another participant (just to show it's feasible)

I think we will support this in client SDK or you can customize the frontend to hide the participant with ATTRIBUTE_PUBLISH_ON_BEHALF.

I guess the misunderstanding part is that ATTRIBUTE_PUBLISH_ON_BEHALF is not supported automatically by the client sdk at this point.

@longcw could you share a code snippet for how this is implemented in the LK playground?

EDIT: I'd also suggest to add this somewhere in your docs / avatar examples / ... so people know what's the proper way to handle it

sure, I'll share an example on how to handle the ATTRIBUTE_PUBLISH_ON_BEHALF in the client.

niqodea · 2025-03-17T16:46:54Z

examples/avatar/bey/requirements.txt

Is this requirements.txt correct? Not sure, other plugins seem to not specify livekit but maybe I am missing something

niqodea · 2025-03-17T16:49:09Z

examples/avatar/bey/README.md

+# LiveKit Beyond Presence Avatar Example
+
+This example demonstrates how to create an animated avatar using Beyond Presence that responds to audio input using LiveKit's agent system.
+The avatar worker generates synchronized video and audio based on received audio input using the Beyond Presence API.
+
+## How it Works
+
+1. The LiveKit agent and the Beyond Presence avatar worker both join into the same LiveKit room as the user.
+2. The LiveKit agent listens to the user and generates a conversational response, as usual.
+3. However, instead of sending audio directly into the room, the agent sends the audio via WebRTC data channel to the Beyond Presence avatar worker.
+4. The avatar worker only listens to the audio from the data channel, generates the corresponding avatar video, synchronizes audio and video, and publishes both back into the room for the user to experience.


I would be inclined to change "avatar worker" to "avatar agent", since that seems more in line with what it actually is (an agent joining the call). A worker, from what I understood, is a process that takes care of a job and can spawn zero or more agents for the room. WDYT?

niqodea · 2025-03-17T16:50:00Z

examples/avatar/bey/agent_worker.py

+    @local_agent_session.output.audio.on("playback_finished")
+    def on_playback_finished(ev: PlaybackFinishedEvent) -> None:
+        logger.info(
+            "playback_finished",
+            extra={"playback_position": ev.playback_position, "interrupted": ev.interrupted},
+        )


Copied this instruction from other examples, what is its purpose exactly?

here it's just for logging. You can ignore it.

@longcw Any reason why this is included in all avatar examples then? Is there some common use case related to avatars that you would use this for?

If not, I'd probably suggest to keep the examples minimal and omit these

Suggested change

@local_agent_session.output.audio.on("playback_finished")

def on_playback_finished(ev: PlaybackFinishedEvent) -> None:

logger.info(

"playback_finished",

extra={"playback_position": ev.playback_position, "interrupted": ev.interrupted},

)

niqodea · 2025-03-17T16:51:16Z

examples/avatar/bey/README.md

+## How it Works
+
+1. The LiveKit agent and the Beyond Presence avatar worker both join into the same LiveKit room as the user.
+2. The LiveKit agent listens to the user and generates a conversational response, as usual.
+3. However, instead of sending audio directly into the room, the agent sends the audio via WebRTC data channel to the Beyond Presence avatar worker.
+4. The avatar worker only listens to the audio from the data channel, generates the corresponding avatar video, synchronizes audio and video, and publishes both back into the room for the user to experience.


Do you see any problem with this explanation? Please let me know if you think something is wrong or unclear! 🙏

niqodea · 2025-03-19T17:30:46Z

@longcw Since you're leading the integration of avatar examples for LK agents, I had a few questions:

Do you plan to merge [draft] Avatar integration example for agent 1.0 #1614 before or soon after the release of LK Agents 1.0?
Would it make sense to merge this PR into yours before you finalize it? If so, would you be open to taking over the example or plugin files?
Looking ahead, how do you see ownership of these components? For instance, would it make sense for the LiveKit team to take over the plugin project and design it as they see fit using our API docs as a reference? Or would it be better for us to maintain ownership and handle the plugin abstractions and API interactions on our end?

Let me know your thoughts. Happy to collaborate to make this as smooth as possible!

longcw · 2025-03-20T13:16:56Z

@longcw Since you're leading the integration of avatar examples for LK agents, I had a few questions:

Do you plan to merge [draft] Avatar integration example for agent 1.0 #1614 before or soon after the release of LK Agents 1.0?

Would it make sense to merge this PR into yours before you finalize it? If so, would you be open to taking over the example or plugin files?

Looking ahead, how do you see ownership of these components? For instance, would it make sense for the LiveKit team to take over the plugin project and design it as they see fit using our API docs as a reference? Or would it be better for us to maintain ownership and handle the plugin abstractions and API interactions on our end?

Let me know your thoughts. Happy to collaborate to make this as smooth as possible!

Probably not, so you can create the PR to the dev branch for your plugin.
I think it would be good if you can help to maintain the plugin if there is an API change on your sever side. I can help to review and clean it. Also, I'll help to update the plugin if there is any change on the agent side.

niqodea · 2025-03-21T09:15:22Z

@longcw Thank you, makes sense! I'll rebase this on top of #1364 then.

niqodea · 2025-03-21T15:02:45Z

I updated the PR to:

Specify the correct setup of hiding the avatar agent and have it post audio and video on behalf of the local agent
Remove avatar_agent_name as a configurable parameter in the plugin (since the avatar agent will be hidden anyway)
Other minor refactoring

Co-authored-by: Felix Altenberger <felix@beyondpresence.ai> Co-authored-by: Lucas Jacobson <lucas@beyondpresence.ai> Co-authored-by: Nicola De Angeli <nicola@beyondpresence.ai>

niqodea · 2025-04-14T10:11:09Z

Hi @longcw, any actionable for me to help merging this into the main examples PR? 🙏

longcw · 2025-04-14T11:52:51Z

Hi @longcw, any actionable for me to help merging this into the main examples PR? 🙏

can you rebase this pr to main and change the target branch to main.

niqodea · 2025-04-15T06:10:39Z

@longcw I merged the latest main since the previous history already had a lot of merges which made rebasing a bit difficult, hope that's also ok! The diff should now be meaningful again.

longcw · 2025-04-15T07:53:34Z

Thanks @niqodea! I have tested your avatar api with the token and it works well. If you don't mind I can take this one, I may create a new pr with some clean up.

niqodea · 2025-04-15T08:43:25Z

Sure, go ahead! Thank you!

Co-authored-by: Felix Altenberger <felix@beyondpresence.ai> Co-authored-by: Lucas Jacobson <lucas@beyondpresence.ai> Co-authored-by: Nicola De Angeli <nicola@beyondpresence.ai>

github-actions · 2025-04-15T09:12:30Z

⚠️ Changeset Required

We detected changes in the following package(s) but no changeset file was found. Please add one for proper versioning:

livekit-agents

👉 Create a changeset file by clicking here.

niqodea · 2025-04-15T09:13:24Z

@longcw I just merged the latest #1700 which contains a small fix and specifies the correct versions for livekit-plugins-bey.

longcw and others added 10 commits March 7, 2025 10:45

add avatar example in a single file

bbe8bc7

clean up

188a737

Update simili demo to latest agents 1.0 version

9dce8be

Adds some updated documentation for Simli avatar

24c46e9

update simli avatar example

3a38ca4

update avatar example readme

48e29ae

Merge remote-tracking branch 'origin/dev-1.0' into longc/avatar-example

9a41bcf

add integrated simli example

b2e78b1

Merge remote-tracking branch 'origin/dev-1.0' into longc/avatar-example

c9acc62

update avatar examples

940b6ff

niqodea commented Mar 17, 2025

View reviewed changes

niqodea force-pushed the add-bey-example branch from 20fcb83 to 3125dbe Compare March 17, 2025 16:52

niqodea changed the title ~~add bey example~~ add bey avatar example Mar 18, 2025

fa9r mentioned this pull request Mar 18, 2025

Extending VoicePipelineAgent with a generative video avatar (we have the model, we need to figure out how to plug it in) #1424

Closed

niqodea force-pushed the add-bey-example branch from 3125dbe to 1cd8969 Compare March 21, 2025 14:40

niqodea changed the base branch from longc/avatar-example to dev-1.0 March 21, 2025 14:41

niqodea force-pushed the add-bey-example branch from 1cd8969 to 219d5fc Compare March 21, 2025 15:02

niqodea changed the base branch from dev-1.0 to longc/avatar-example March 21, 2025 15:02

niqodea force-pushed the add-bey-example branch from 219d5fc to e10f534 Compare March 21, 2025 15:16

niqodea mentioned this pull request Mar 24, 2025

add bey avatar plugin #1700

Closed

longcw added 4 commits March 27, 2025 22:53

Merge branch 'dev-1.0' into longc/avatar-example

d470a4e

fix bithuman agent

a1fc062

Merge branch 'dev-1.0' into longc/avatar-example

977df1b

update avatar example for dev-1.0

c8105eb

niqodea and others added 5 commits March 31, 2025 17:25

add better exception handling

cfef12a

add bey api url to environment

783f03a

Merge remote-tracking branch 'origin/longc/avatar-example' into HEAD

8803ff8

add bey example

262927d

Co-authored-by: Felix Altenberger <felix@beyondpresence.ai> Co-authored-by: Lucas Jacobson <lucas@beyondpresence.ai> Co-authored-by: Nicola De Angeli <nicola@beyondpresence.ai>

use default avatar argument

d8732d2

niqodea force-pushed the add-bey-example branch from 7b8acb3 to d8732d2 Compare March 31, 2025 16:30

niqodea added 2 commits April 1, 2025 18:19

move v1 out of base url

bf455ca

Merge branch 'add-bey-plugin' into add-bey-example

999dfab

niqodea marked this pull request as ready for review April 14, 2025 10:09

niqodea requested a review from longcw April 14, 2025 10:09

Merge remote-tracking branch 'origin/main' into add-bey-example

7757699

niqodea changed the base branch from longc/avatar-example to main April 15, 2025 06:09

niqodea and others added 9 commits April 15, 2025 11:10

add bey plugin

de6626e

Co-authored-by: Felix Altenberger <felix@beyondpresence.ai> Co-authored-by: Lucas Jacobson <lucas@beyondpresence.ai> Co-authored-by: Nicola De Angeli <nicola@beyondpresence.ai>

rename api key variable

6d97489

introduce ege stock avatar id constant

4e481a9

format

92851fe

add better exception handling

bb18f06

add bey api url to environment

89e63ec

move v1 out of base url

09d9ee5

fix typing

83c5dc9

Merge branch 'add-bey-plugin' into add-bey-example

afe08ad

longcw mentioned this pull request Apr 17, 2025

add bey avatar plugin #2031

Merged

longcw closed this Apr 17, 2025

		# allow your local agent to publish transcripts on behalf of the avatar agent
		.with_attributes({ATTRIBUTE_PUBLISH_ON_BEHALF: ctx.room.local_participant.identity})

Comments

Conversation

niqodea commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

CLAassistant commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fa9r Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fa9r Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niqodea commented Mar 19, 2025

Uh oh!

longcw commented Mar 20, 2025

Uh oh!

niqodea commented Mar 21, 2025

Uh oh!

niqodea commented Mar 21, 2025

Uh oh!

niqodea commented Apr 14, 2025

Uh oh!

longcw commented Apr 14, 2025

Uh oh!

niqodea commented Apr 15, 2025

Uh oh!

longcw commented Apr 15, 2025

Uh oh!

niqodea commented Apr 15, 2025

Uh oh!

github-actions bot commented Apr 15, 2025

⚠️ Changeset Required

Uh oh!

niqodea commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

niqodea commented Mar 17, 2025 •

edited

Loading

changeset-bot bot commented Mar 17, 2025 •

edited

Loading

CLAassistant commented Mar 17, 2025 •

edited

Loading

fa9r Mar 18, 2025 •

edited

Loading

fa9r Mar 18, 2025 •

edited

Loading

niqodea commented Apr 15, 2025 •

edited

Loading