Skip to content

Add option to enable/disable heuristic early-return logic#2133

Open
bundasmanu wants to merge 1 commit into
sipwise:masterfrom
bundasmanu:bundasmanu/add_param_that_allows_disable_of_heuristic_early_return
Open

Add option to enable/disable heuristic early-return logic#2133
bundasmanu wants to merge 1 commit into
sipwise:masterfrom
bundasmanu:bundasmanu/add_param_that_allows_disable_of_heuristic_early_return

Conversation

@bundasmanu

@bundasmanu bundasmanu commented Jul 6, 2026

Copy link
Copy Markdown

This PR is related to the following RTPEngine Forum question: (https://groups.google.com/g/rtpengine/c/YsTU99HGd2o).

After 13.5 (including) the following commit was introduced: (afce798).

The idea of this commit is quiet good, but could affect some scenarios, like: when clients are behind some VPN's.

So, the proposal here, is to include a new parameter: endpoint-learning-heuristic-disable-early-return. This parameter by default assumes false as value (so, the current behavior is preserved). But, passing true as value, reverts the current behavior and preserves the old logic (<= 13.4).

The following PR was already tested locally, and allows the correct turn on/off.

If possible, a backport should be made for versions >= 13.5 (14.x also).

…rning when there is a match with address advertised in the SDP
@rfuchs

rfuchs commented Jul 6, 2026

Copy link
Copy Markdown
Member

I don't think making this an option is a good solution. You said this is affecting scenarios with ICE - I'm thinking endpoint learning in general should be defused or even disabled altogether when ICE is in use, as ICE should be in charge of determining endpoint address. I was under the impression that this is already done. Let me check.

@rfuchs

rfuchs commented Jul 6, 2026

Copy link
Copy Markdown
Member

Have you tried what happens if you just disable endpoint learning?

@bundasmanu

Copy link
Copy Markdown
Author

Have you tried what happens if you just disable endpoint learning?

Yes.
Inside the topic in the forum, i describe that. If, i switch to delayed it solves the problem. But, switching that value, could lead to problems, that i'm not aware. As, this is totally dependent also on the clients environment side. Switching to off does not solve the problem.

And, using this eventual parameter introduces some help, for people that wants to keep using heuristic, but without that early-return. By default, assumes false, so will not affect current deployments.

@rfuchs

rfuchs commented Jul 6, 2026

Copy link
Copy Markdown
Member

Have you tried what happens if you just disable endpoint learning?

Yes. Inside the topic in the forum, i describe that. If, i switch to delayed it solves the problem. But, switching that value, could lead to problems, that i'm not aware. As, this is totally dependent also on the clients environment side. Switching to off does not solve the problem.

That's odd. ICE should still adjust the endpoint addresses based on the outcome of the negotiations, basically obsoleting the other learning mechanism. Perhaps there's some other underlying issue lurking. Sadly I have no way to reproduce this myself.

And, using this eventual parameter introduces some help, for people that wants to keep using heuristic, but without that early-return. By default, assumes false, so will not affect current deployments.

I can see it as a feasible solution for a backport, but for future versions going forward I'd rather avoid having it. Determining the endpoint should be delegated to ICE - the regular learning mechanism should be just a fallback.

@bundasmanu

Copy link
Copy Markdown
Author

Have you tried what happens if you just disable endpoint learning?

Yes. Inside the topic in the forum, i describe that. If, i switch to delayed it solves the problem. But, switching that value, could lead to problems, that i'm not aware. As, this is totally dependent also on the clients environment side. Switching to off does not solve the problem.

That's odd. ICE should still adjust the endpoint addresses based on the outcome of the negotiations, basically obsoleting the other learning mechanism. Perhaps there's some other underlying issue lurking. Sadly I have no way to reproduce this myself.

And, using this eventual parameter introduces some help, for people that wants to keep using heuristic, but without that early-return. By default, assumes false, so will not affect current deployments.

I can see it as a feasible solution for a backport, but for future versions going forward I'd rather avoid having it. Determining the endpoint should be delegated to ICE - the regular learning mechanism should be just a fallback.

@rfuchs, i completely see your point.

And its indeed an interesting case. But, that commit is the root cause that switch the behavior between the 13.4 and >= 13.5. The only difference between success and failure, is on the rtpengine version. I dunno, if the person that introduce that commit, evaluates the full impact of that change. Maybe more people in the future, when they start using >= 13.5, could start complain about it.

@rfuchs

rfuchs commented Jul 6, 2026

Copy link
Copy Markdown
Member

I think I was able to replicate this scenario with a test case (f006d72) and I think 9431125 should fix it. Are you able to test/confirm this against your specific scenario?

@bundasmanu

Copy link
Copy Markdown
Author

I think I was able to replicate this scenario with a test case (f006d72) and I think 9431125 should fix it. Are you able to test/confirm this against your specific scenario?

I think so. I will test your branch and i will give you feedback after it.

@bundasmanu

bundasmanu commented Jul 6, 2026

Copy link
Copy Markdown
Author

Hi @rfuchs,

Oh yes, it solves the problem.

EDIT: But, introduces a side effect. Now, i have ERR logs: Failed to push relay stream to kernel: Invalid argument and Failed to delete relay stream from kernel: No such file or directory.
Beforehand, these logs didn't appear. And on my patch, they also didn't appear. Only on your branch.

@rfuchs

rfuchs commented Jul 6, 2026

Copy link
Copy Markdown
Member

That's good information, thanks.

@rfuchs

rfuchs commented Jul 7, 2026

Copy link
Copy Markdown
Member

Hi @rfuchs,

Oh yes, it solves the problem.

EDIT: But, introduces a side effect. Now, i have ERR logs: Failed to push relay stream to kernel: Invalid argument and Failed to delete relay stream from kernel: No such file or directory. Beforehand, these logs didn't appear. And on my patch, they also didn't appear. Only on your branch.

Ok, this is actually unrelated and happens due to f2d57de

Will have a fix-up for this as well.

sipwise-jenkins pushed a commit that referenced this pull request Jul 7, 2026
With ICE in use, we don't want to use the regular endpoint mechanism,
and instead let ICE determine the appropriate peer address. This is more
secure and also avoids incorrect peer addresses learned from the SDP
shortcut logic.

Closes #2133

Change-Id: I67e11c06cdda5db26413529c948fcdb8deadc11b
@rfuchs

rfuchs commented Jul 7, 2026

Copy link
Copy Markdown
Member

Alright, took the opportunity to rework a few other related things that popped up, to hopefully improve the overall situation for good. Everything is pushed to https://github.com/sipwise/rtpengine/tree/rfuchs/2133 for now. Let me know if you can test this and how it goes.

@bundasmanu

Copy link
Copy Markdown
Author

Alright, took the opportunity to rework a few other related things that popped up, to hopefully improve the overall situation for good. Everything is pushed to https://github.com/sipwise/rtpengine/tree/rfuchs/2133 for now. Let me know if you can test this and how it goes.

Thank you @rfuchs, for all the effort here.

Tomorrow, i will re-test, and i will give feedback.

@bundasmanu

Copy link
Copy Markdown
Author

Hi @rfuchs,

Tested today again, with your most recent commits.

The kernel ERR logs have disappeared, but now i got the same behavior as before.

@rfuchs

rfuchs commented Jul 8, 2026

Copy link
Copy Markdown
Member

Hi @rfuchs,

Tested today again, with your most recent commits.

The kernel ERR logs have disappeared, but now i got the same behavior as before.

Sure you're running the code from that branch? Because there is a test for that scenario...

If confirmed then post a full debug log, because that means something else is going on.

@bundasmanu

Copy link
Copy Markdown
Author

Hi @rfuchs,
Tested today again, with your most recent commits.
The kernel ERR logs have disappeared, but now i got the same behavior as before.

Sure you're running the code from that branch? Because there is a test for that scenario...

If confirmed then post a full debug log, because that means something else is going on.

Yes, i'm running from your branch.

Debug log --> Debug Log

104.x (VPN IP) and 2.x (real IP)

@rfuchs

rfuchs commented Jul 8, 2026

Copy link
Copy Markdown
Member

Hi @rfuchs,
Tested today again, with your most recent commits.
The kernel ERR logs have disappeared, but now i got the same behavior as before.

Sure you're running the code from that branch? Because there is a test for that scenario...
If confirmed then post a full debug log, because that means something else is going on.

Yes, i'm running from your branch.

Debug log --> Debug Log

104.x (VPN IP) and 2.x (real IP)

This seems to be on purpose though?

ICE eventually converged on:

Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] Received ICE/STUN response code 0 for candidate pair R8jsGguQw8Ey9b12:6d7033193bac4abb0:1 from 104.28.181.215:59473 to 10.40.148.9
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] Setting ICE candidate pair R8jsGguQw8Ey9b12:6d7033193bac4abb0:1 as succeeded
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] ICE completed, using pair R8jsGguQw8Ey9b12:6d7033193bac4abb0:1
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: INFO: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] ICE negotiated: new peer for component 1 is 104.28.181.215:59473
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: INFO: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] ICE negotiated: local interface 10.40.148.9

Candidate 6d7033193bac4abb0 wasn't in the offer, but was learned as a peer-reflexive candidate and was immediately nominated (probably aggressive nominiation):

Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] Received ICE/STUN request from 104.28.181.215:59473 on 10.40.148.9:19819
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] Created candidate pair R8jsGguQw8Ey9b12:6d7033193bac4abb0:1 between 10.40.148.9 and 104.28.181.215:59473, type prflx
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] Triggering check for R8jsGguQw8Ey9b12:6d7033193bac4abb0:1
Jul 08 14:45:25 ip-10-40-148-9 rtpengine[10257]: DEBUG: [0p2i184n0lk82el5gn75/0s94lgbhnj/1 port 19819]: [ice] ICE pair R8jsGguQw8Ey9b12:6d7033193bac4abb0:1 has been nominated by peer

So if the priority of that candidate was higher than the two candidates from the offer, this one would be the correct one to use.

Sadly the priority isn't being logged. You could see it in the STUN attributes in Wireshark. I'll add the priority to the logs.

@rfuchs

rfuchs commented Jul 8, 2026

Copy link
Copy Markdown
Member

Pushed 3c6811f to the branch to log the priority

@bundasmanu

Copy link
Copy Markdown
Author

Pushed 3c6811f to the branch to log the priority

The priority logged matches the one in the SDP, and the candidate with Real IP, is the most priority one.
log

@rfuchs

rfuchs commented Jul 8, 2026

Copy link
Copy Markdown
Member

Where do you see the priority matching the one from the SDP?

SDP has priorities 1685987071 and 1685855999. Learned candidate has priority 1853628159, which is higher.

This is in line with what the RFC says: the type preference for peer-reflexive (learned) candidates must be higher than the type preference for server-reflexive candidates. Both candidates from the SDP are server-reflexive, so the learned peer-reflexive candidate wins.

So based on that I would say that using the candidate 104.28.181.215:59670 is correct. What makes you think that it's the wrong one?

@bundasmanu

Copy link
Copy Markdown
Author

Where do you see the priority matching the one from the SDP?

SDP has priorities 1685987071 and 1685855999. Learned candidate has priority 1853628159, which is higher.

This is in line with what the RFC says: the type preference for peer-reflexive (learned) candidates must be higher than the type preference for server-reflexive candidates. Both candidates from the SDP are server-reflexive, so the learned peer-reflexive candidate wins.

So based on that I would say that using the candidate 104.28.181.215:59670 is correct. What makes you think that it's the wrong one?

I see your point.

But, this behavior starts only after 13.4 version (using heuristic without early-return, works).
We agree that based on the priority makes sense to choose that one, but its the wrong one (from client use perspective).

I completely see your point, but this could impact in a lot of environments. Maybe my suggestion, could make sense to be backported, for people that relays in the same issue.

@rfuchs

rfuchs commented Jul 9, 2026

Copy link
Copy Markdown
Member

But, this behavior starts only after 13.4 version (using heuristic without early-return, works). We agree that based on the priority makes sense to choose that one, but its the wrong one (from client use perspective).

I completely see your point, but this could impact in a lot of environments. Maybe my suggestion, could make sense to be backported, for people that relays in the same issue.

Well, couple points.

ICE is very explicit with priorities. Your client offers only two server-reflexive candidates. What your client probably didn't expect is that the effective port from the lower-priority VPN candidate came out different from the one in the SDP, and so that results in a new, higher-priority candidate. But your client will be aware of this, from the STUN response, and will know that there is now a new higher-priority peer-reflexive candidate. Your client could just not nominate this new candidate if it doesn't want to use it, but it does so anyway. Your client is the controlling agent here, and rtpengine isn't even in a position to make any other choice.

ICE under RFC 5245 mandates that the nominated candidate pair with the highest priority must be used. Which interface does your client eventually end up using for outgoing RTP? How does it end up sending RTP?

ICE under RFC 8445 relaxes this a little bit and does away with aggressive nomination. Under that RFC, only one candidate pair must ever be nominated, but any working candidate pair may be used for media until then. This isn't what your client is doing though.

The only thing I can imagine is to allow for slightly misbehaving clients which nominate one pair but end up using another one, but I'm not sure if that's a good solution.

@rfuchs

rfuchs commented Jul 9, 2026

Copy link
Copy Markdown
Member

What do you get when, with the current unpatched code base, you set endpoint-learning=immediate together with media-handover (at least for the leg towards the WebRTC client)? Post a debug log if this seems to work.

@bundasmanu

Copy link
Copy Markdown
Author

Hi @rfuchs,

Sorry for the delay.

Regarding the client part, sure its the controller. But, the Client is the same one between the tests from 13.4 and >=13.5, the only changing piece is RTPEngine, that produces a different behavior using the same environment and flags. Doing captures on the client side, DTLS and STUN packets are the same in both cases.

Interface used was en0.

I don't want to switch to media-handover. But using endpoint-learning=immediate does not solve it, delayed yes.

I'm ok, using the old approach (<= 13.4 heuristic behavior, without (afce798)). The first commits on your branch actually solved it also (but that ERR logs, impact).

This is a hard topic, and yes hard to get answers. But, my concern is, massive tests and feedback from people were done, when that commit (afce798) was introduced?

@rfuchs

rfuchs commented Jul 9, 2026

Copy link
Copy Markdown
Member

Regarding the client part, sure its the controller. But, the Client is the same one between the tests from 13.4 and >=13.5, the only changing piece is RTPEngine, that produces a different behavior using the same environment and flags. Doing captures on the client side, DTLS and STUN packets are the same in both cases.

It doesn't matter, because both versions are not behaving correctly. Endpoint learning is largely incompatible with ICE. One version just coincidentally produces a result which you think is what is expected, and the other does not. Neither is actually correct.

Interface used was en0.

That doesn't really tell me anything. You mean it was not sending it via the VPN?

I don't want to switch to media-handover.

That is not helpful. From the code it looks like it might just do what you need (and tests are confirming this), but now I can't be sure.

This is a hard topic, and yes hard to get answers. But, my concern is, massive tests and feedback from people were done, when that commit (afce798) was introduced?

You're the first one who has ever reported a problem, and your problem is really unrelated to this commit. I've been trying to get to an actual and proper solution, and your insistence on this particular commit being the problem is also not helpful.

@bundasmanu

Copy link
Copy Markdown
Author

Ok @rfuchs.

Yes, it does not send it via utun<x> interface.

Ok, i will try to do the test with media-handover.

Ok, understood. I really appreciate your effort. But, i also can't do major changes on my side, as we have a massive number of clients, and this is a scenario hard to predict the full picture.

@rfuchs

rfuchs commented Jul 10, 2026

Copy link
Copy Markdown
Member

Ok, i will try to do the test with media-handover.

Ok, understood. I really appreciate your effort. But, i also can't do major changes on my side, as we have a massive number of clients, and this is a scenario hard to predict the full picture.

My idea is that for ICE, the media-handover behaviour could implicitly be enabled (with some modifications), as we already guarantee that only verified ICE pairs are able to see media, and so this would allow the remote client to switch to a different pair simply by redirecting its RTP, even if ICE nominated some other pair.

@rfuchs

rfuchs commented Jul 10, 2026

Copy link
Copy Markdown
Member

Branch rfuchs/2133 has been updated (top commit now 82f068a) to the latest proposed fix. No extra config or flags needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants