Manually completing ICE gathering

[clone1018]

Is there a way to manually "complete" ice gathering? I have a local <=> local webrtc project I'm testing right now that does not need ice support (and does not support candidate negotiation at the moment).

In pion, this is called webrtc.GatheringCompletePromise

[mickel8]

Hi @clone1018, I don't think we have something similar to what pion has. Once the gathering process is completed, you will be notified with :gathering_state_change message. See here.

If you take a look at gather_candidates source code, you will notice that it fetches host candidates and immediately send them to you so if you don't pass any stun servers, the gathering process will be very quick.

Does it help?

Also, what do you mean by saying that your project does not need candidate negotiation. You have to exchange at least one candidate, even when running locally 🤔

[clone1018]

Gotcha, I'll look at some of the code. Probably should have had some coffee before opening this :)

Basically my use case right now is a client sends me an offer, I create a peer connection, accept the offer, and send back an answer. The client's offer includes some candidates in the offer:

a=candidate:3 1 UDP 2130705919 local-ipv6-address 55415 typ host
a=candidate:1 1 UDP 2122317823 local-address 55415 typ host
a=candidate:2 1 UDP 2122317567 some-other-address 55415 typ host
a=end-of-candidates

There is no additional ice candidate negotiation (eg websocket, http post, etc). Are the candidates included in the offer SDP consumed automatically? The issue I'm experiencing is the negotiation seems to be completed but the client is not sending any rtp packets. After a little bit the client disconnects because the peer connection failed. I see some unhandled events in my peer connection handler but it's not super clear where it's failing.

<-- got offer -->
<-- set_remote_description(ex_pc, offer) -->
<-- create_answer(ex_pc) -->
<-- set_local_description(ex_pc, answer) -->
07:04:44.809 [warning] Unhandled: :negotiation_needed
<-- answer is sent to client -->
<-- then {:track, %MediaStreamTrack{}} is triggered for both audio & video as expected -->
<-- add_track(ex_pc, track) -->
07:04:44.815 [warning] Unhandled: {:signaling_state_change, :have_remote_offer}
07:04:44.815 [warning] Unhandled: {:track_muted, 35661075333720937873835115413}
07:04:44.815 [warning] Unhandled: {:track_ended, 35661075333720937873835115413}
07:04:44.815 [warning] Unhandled: {:signaling_state_change, :stable}
07:04:44.815 [warning] Unhandled: :negotiation_needed

Not sure if anything is obvious, mainly I'm not sure about what might be missing in ex_webrtc that would prevent this flow, but I'll keep testing locally and let you know!

[clone1018]

Ooo, I'm making some progress locally! Will post an update soon.

[mickel8]

It's also important that we don't include ICE candidates in our offer right now so you always have to send them separately

[clone1018]

I suspect this is the problem, I compared a Go example I had and ex_webrtc and the offer back is indeed missing ice candidates. Which makes sense why the client never actually completes the peer connection.

[LVala]

Yeah, that's most likely the problem, judging by the fact that you only exchange the offer/answer, not the candidates themselves. In a RTCPeerConnection the flow would be as follows (assuming I have a pc PeerConnection object with some tracks added:

pc.onicecandidate = event => console.log(event.candidate)
offer = await pc.createOffer()
await pc.setLocalDescription(offer)
send_to_remote_peer(offer)  // offer does not include any ICE candidates
// => onicecandidate fired with cand1
// => onicecandidate fired with cand1
// => onicecandidate fired with cand1
const ld = pc.currentLocalDescription
// ld contains cand1, cand2 and cand3

You can choose to send the cand1 etc. separately, if your PeerConnection supports Trickle ICE (that's what we always do, as it's usually makes the negotiation faster).

I our case, the currentLocalDescription will never include the candidates, so you can either use Trickle ICE (locally it doesn't make much sense as you've probably noticed) or, as a workaround, manually, append the values form messages {:ice_candidate, %{candidate: cand}} to every m-line in the SDP form offer (offer.sdp, offer as returned by PeerConnection.create_offer()) (I'm not sure if the format of ICE candidates form the message is the same as in the SDP, compare it to a proper SDP from a browser).

I'll probably make a fix for this tomorrow or on Wednesday.

[clone1018]

Thanks for the info! Since the candidates are not in the default answer, that makes total sense. For anyone else following along, this is the code I ended up with to gather the candidates and inject them into the SDP answer:

  defp gather_candidates(candidates) do
    receive do
      {:ex_webrtc, _from, {:ice_candidate, candidate}} ->
        gather_candidates([candidate | candidates])
    after
      400 -> candidates
    end
  end

  defp append_candidates(sdp, candidates) do
    sdp_candidates =
      Enum.map(candidates, fn %ExWebRTC.ICECandidate{candidate: x} -> "a=#{x}" end) ++
        ["a=end-of-candidates"]

    lines = String.split(sdp, "\r\n")

    recv_index = Enum.find_index(lines, fn x -> x == "a=recvonly" end)

    lines
    |> List.insert_at(recv_index + 1, sdp_candidates)
    |> List.flatten()
    |> Enum.reject(fn x -> x == "a=ice-options:trickle" end)
    |> Enum.join("\r\n")
  end

The gathering happens in the receive block to block the response until the candidates have finished gathering. It does appear the %ICECandidate{candidate: x} response is "valid" for what a browser generates too. Also was experimenting with removing the ice-options:trickle just in case that was giving some issue. Overall the client is still not sending media but I'm going to continue debugging!

[LVala]

Cool! Instead of using a timeout, you can wait for the {:gathering_state_change, :completed} (or something similar, check in docs). When using only host ICE candidates (so no STUN servers passed to PeerConnection), the gathering should be nearly instantaneous.

Also, feel free to share whatever issues you might have, it might be something obvious that differs from the standard and we just didn't put it in the docs.

[mickel8]

Looks like this feature, i.e. including ice candidates in the offer might also be needed for WHIP/WHEP implementation

[clone1018]

That's actually what I'm playing around with :D @Sean-Der put together https://github.com/Glimesh/broadcast-box a bit ago in Go as a proof of concept for WHIP integration with OBS.

I'm rebuilding that using ex_webrtc to better understand how the library works & can be integrated.

[mickel8]

Awesome! Did you start with WHIP or WHEP?

[clone1018]

Started with WHIP, both trying to ingest from OBS and a browser source. So far the browser source works with audio (haven't tested video), but the OBS ingest is where I'm experiencing the issues. It's pretty new to OBS, and could have issues on top of any potential ex_webrtc issues.

WHEP is pretty simple in my experience, and since it's the browser we can do real ICE trickle too.

[Sean-Der]

@clone1018 Can you open up OBS via CLI and share the logs? Would love to help debug. Does ex_webrtc have a chat/dev community we can talk on? We can do the Broadcast Box discord and debug in that one!

[mickel8]

We still don't know where we are going to talk about Elixir WebRTC (Elixir/Membrane's discord or slack) but for now we can jump into Membrane's discord. Here is the link: https://discord.com/invite/u3DHhRjNmU