Inconsistent Webhooks

[alan-willard-simplisafe]

Describe the bug
To start, I am not quite sure this is a bug per se, but there is appears to be at a minimum some inconsistencies between webhook events, and events received through the Livekit SDK

In short, I am looking at migrating some parts of my system from relying on events received through the Go Livekit SDK (participant joined, participant left, namely) over to using webhooks. This seems to be reliable in majority of cases, but with some outliers.

1. I've noticed that in the case of participant_joined events, the webhook is notably faster than the same event being received by the application using the Go SDK. In some cases up to 2 seconds faster. This in and of itself is not necessarily a problem, but indicates to me that the conditions that lead to sending the the event through the webhook and sending the same event over the room connection / SDK differ. This could play into the second issue.

2. There's a discrepancy between events. Notably, I will receive participant_joined events through the SDK but not from the webhook. Investigation of the logs seems to indicate that participant_connection_aborted hook does fire. Livekit logs show something like removing participant without connection. This seems to tell me that

• The SDK indicates a participant has joined the room.

• The webhook tells me a participant FAILED to join the room.

  Which is correct? My system has been operating to this point purely on the SDK, which makes me feel that it is more reliable. But in either case the semantics between the two events are clearly different. This could be a bug, or just a documentation issue or SDK API problem, not sure.

Server

• Version: 1.9.4
• Environment: EKS

Client

• SDK: Go
• Version: v2.12.6

To Reproduce

1. Go app joins a room
2. Other client joins the room
3. Go app sees participant_connected over the SDK callback onParticipantConnected
4. Webhook received for participant_connected (often before the SDK callback) OR not received at all

Expected behavior
Webhooks and SDK events match 1:1 in a timely manner

[alan-willard-simplisafe]

#2399 might be related

[boks1971]

Yeah, the semantics are different.

Webhook participant_joined happens when the participant has established peer connection, i. e. media connectivity.
SDK emits ParticipantConnected when the signal connection is established. Signal connection establishment happens before media connectivity.

Describe the bug To start, I am not quite sure this is a bug per se, but there is appears to be at a minimum some inconsistencies between webhook events, and events received through the Livekit SDK

In short, I am looking at migrating some parts of my system from relying on events received through the Go Livekit SDK (participant joined, participant left, namely) over to using webhooks. This seems to be reliable in majority of cases, but with some outliers.

1. I've noticed that in the case of participant_joined events, the webhook is notably faster than the same event being received by the application using the Go SDK. In some cases up to 2 seconds faster. This in and of itself is not necessarily a problem, but indicates to me that the conditions that lead to sending the the event through the webhook and sending the same event over the room connection / SDK differ. This could play into the second issue.

This could happen due to signal batching for subscribe-only participants. participant_joined webhook is sent when the media connection is established. To reduce signalling traffic storms, subscribe-only participant updates are batched in 3 second intervals. This could lead to ParticipantConnected event being behind webhook.

2. There's a discrepancy between events. Notably, I will receive participant_joined events through the SDK but not from the webhook. Investigation of the logs seems to indicate that participant_connection_aborted hook does fire. Livekit logs show something like removing participant without connection. This seems to tell me that>

• The SDK indicates a participant has joined the room.
• The webhook tells me a participant FAILED to join the room.
  Which is correct? My system has been operating to this point purely on the SDK, which makes me feel that it is more reliable. But in either case the semantics between the two events are clearly different. This could be a bug, or just a documentation issue or SDK API problem, not sure.

This is due to the difference mentioned above. When the media connection cannot be established, participant_connection_aborted webhook is sent like you observed. So, the sequence would be

1. SDK gets ParticipantConnected because the remote participant was able to establish signal connection.
2. Webhook sees participant_connection_aborted because that remote participant could not establish media connection.

Server

• Version: 1.9.4
• Environment: EKS

Client

• SDK: Go
• Version: v2.12.6

To Reproduce

1. Go app joins a room
2. Other client joins the room
3. Go app sees participant_connected over the SDK callback onParticipantConnected
4. Webhook received for participant_connected (often before the SDK callback) OR not received at all

Expected behavior Webhooks and SDK events match 1:1 in a timely manner

[alan-willard-simplisafe]

Thank you for the clarification! The only thing I'm not sure about is if my clients are subscribe-only, I'll have to look into that. Everything else makes sense though. I'll leave any potential action here up to you folks. Thanks!

[boks1971]

from @boks1971:

It need not necessarily be subscribe-only. Even if a participant has permissions to publish, a participant is considered a publisher only if they publish a track.