Add support for specifying a STUN server for Janus

[mtlynch]

Blocked on #1496

There are a few common scenarios where the browser and the Janus server on the TinyPilot device can't establish a WebRTC connection. See @cghague's investigations for details (especially the Conclusions note at the end):

https://github.com/tiny-pilot/tinypilotkvm.com/issues/939#issuecomment-1600620310

We should add an "Advanced" section to the Video Settings page that allows the user to configure a STUN server or 1:1 mapping for Janus to use in the WebRTC connection. For privacy, I'd like the STUN server to be off by default, but if they enable it, we should default to a Google STUN server but allow the user to enter any STUN server they choose.

UPDATE (2023-08-21, @jotaen4tinypilot): Initially, we wanted to implement both STUN and 1:1 mapping. However, we decided to restrict the work to STUN in the first step, and push 1:1 mapping support down the agenda.

[jotaen4tinypilot]

I started to look into this, and read up on the (super helpful) research, and also STUN and NAT issues in general. I’d like to bounce and discuss a few thoughts around the implementation, to get a better understanding, and to ensure alignment in regards to our approach.

Once I’m back from my break after next week, I could start to dive into this more, and come up with a more concrete plan of action.

Backend

Backend-wise, I’m wondering what the best approach would be to tackle this. In order to toggle and configure STUN mode in Janus, we have to write the STUN server address and port into the Janus config file, and then restart the uStreamer and Janus systemd services to apply the changes.

“Historically”, we would have created two variables ustreamer_h264_stun_server and ustreamer_h264_stun_port in ~/settings.yml, and then have the uStreamer Ansible role write the parameters into the config file template and apply everything.

However, as we are moving away from Ansible, that’s probably not the way we want to do it. So the questions are: how do we maintain state, and who is in charge for applying it? Some options:

• (a) Where do we maintain state and configuration?
  • (a.1) ~/settings.yml
  • (a.2) sqlite DB (like the video streaming mode setting)
  • (a.3) In the Janus config file directly (by grepping/parsing it)
• (b) How do we write and apply state and configuration changes?
  • (b.1) Extend existing uStreamer launch script to also write the Janus config file and restart the Janus systemd service
  • (b.2) Create new privileged script that’s run as one-off on changes directly from the Python app
  • (b.3) Create our own Janus systemd service, that has its own launcher similar to what we do with uStreamer

In regards to (a), I think (a.1) would be the easiest and most consistent place – we also still store all uStreamer parameters there. (a.2) would make it difficult to process the data outside of the Python app (e.g. via bash scripts, or other utilities), and (a.3) would feel a bit hacky, since we’d have to deal with config syntax idiosyncracies, and we’d still need a privileged script for accessing the file.

In regards to (b), I think it might depend a bit how tight we want to couple uStreamer and Janus on our end. (b.1) would couple tighter but would also better ensure “atomicity”, whereas (b.2) keeps things more separate, and is therefore “cleaner” in terms of code structure. I’m not sure (b.3) is an option, since as far as I’m aware the Janus systemd config is currently not in our control. I’d probably tend towards an independent privileged script for simplicity – like (b.2) –, but I also need to look more into it, and get more familiar with the whole topic. If you have thoughts or preferences here, I’d be curious to hear.

Frontend

I’m wondering whether we’d even need a separate “advanced” view of the video settings, or if we’d get away with an inlined solution. Below a rough mock-up of what would be possible – just as a starting point for further exploration. Would something like this be an option for us worth to consider, or would you prefer to stash the STUN-related things away in a separate place? (As they might be a bit more “special” compared to the other settings.)

Screen.Recording.2023-07-07.at.17.15.01.mov

The suggested dropdown list is natively supported by browsers, it might also have multiple entries. I’m wondering, however, whether it’s a good idea to hard-code external addresses in our UI. They ultimately aren’t in our control and might change over time. In this case, our UI could get stale, or worse, appears to be broken. If we’d be worried about that, we could consider to e.g. either link to a support page on tinypilotkvm.com with a list of public STUN servers, or we maintain curated addresses via a “static JSON file” hosted at tinypilotkvm.com/stun-server-list.json that we then fetch in the UI.

[mtlynch]

Where do we maintain state and configuration?

(a.1) ~/settings.yml
(a.2) sqlite DB (like the video streaming mode setting)
(a.3) In the Janus config file directly (by grepping/parsing it)

Sidenote: We should come up with more general rules about what goes in settings.yml vs. what goes in SQLite.

In regards to (a), I think (a.1) would be the easiest and most consistent place – we also still store all uStreamer parameters there. (a.2) would make it difficult to process the data outside of the Python app (e.g. via bash scripts, or other utilities), and (a.3) would feel a bit hacky, since we’d have to deal with config syntax idiosyncracies, and we’d still need a privileged script for accessing the file.

Yeah agreed. I lean towards settings.yml because maybe we eventually want to include that file in debug logs.

I definitely don't want to keep state in the Janus config file because that feels too messy.

(b) How do we write and apply state and configuration changes?
(b.1) Extend existing uStreamer launch script to also write the Janus config file and restart the Janus systemd service
(b.2) Create new privileged script that’s run as one-off on changes directly from the Python app
(b.3) Create our own Janus systemd service, that has its own launcher similar to what we do with uStreamer

In regards to (b), I think it might depend a bit how tight we want to couple uStreamer and Janus on our end. (b.1) would couple tighter but would also better ensure “atomicity”, whereas (b.2) keeps things more separate, and is therefore “cleaner” in terms of code structure. I’m not sure (b.3) is an option, since as far as I’m aware the Janus systemd config is currently not in our control. I’d probably tend towards an independent privileged script for simplicity – like (b.2) –, but I also need to look more into it, and get more familiar with the whole topic. If you have thoughts or preferences here, I’d be curious to hear.

I don't think I have a lot of extra context, but my preference is b.2. It just seems simpler to have a standalone script with a simple job as opposed to bundling it with the uStreamer launcher or creating our own systemd service.

I’m wondering whether we’d even need a separate “advanced” view of the video settings, or if we’d get away with an inlined solution. Below a rough mock-up of what would be possible – just as a starting point for further exploration. Would something like this be an option for us worth to consider, or would you prefer to stash the STUN-related things away in a separate place? (As they might be a bit more “special” compared to the other settings.)

Yeah, I'd definitely like to bury them. Probably 80% of users won't have to use them, and I think less than 3% of users will understand what the settings even mean, so I want to avoid throwing confusing settings at people until they ask.

It's similar to how we bury the "mount mode" for virtual media:

The suggested dropdown list is natively supported by browsers, it might also have multiple entries. I’m wondering, however, whether it’s a good idea to hard-code external addresses in our UI. They ultimately aren’t in our control and might change over time. In this case, our UI could get stale, or worse, appears to be broken. If we’d be worried about that, we could consider to e.g. either link to a support page on tinypilotkvm.com with a list of public STUN servers, or we maintain curated addresses via a “static JSON file” hosted at tinypilotkvm.com/stun-server-list.json that we then fetch in the UI.

Let's hardcode it for now for simplicity. If we need something more flexible in the future, we can add a server component, but anytime we bring in client-server interaction, it's like an order of magnitude jump in complexity/maintenance cost.

[jotaen4tinypilot]

Based on our discussion above, I was looking into the UI some more, specifically into the question how we can hide away the STUN settings. Despite the ticket being blocked right now, I’d like to present three possible UI/UX approaches that I drafted – it’s just food for thought for now, we can defer the discussion to when we resume this ticket.

Some general notes upfront:

• In all mockups, there is a “text”-button, that subtly and optionally hints at the “advanced settings”.
  • We can still try different locations for that button, though I think the position near the bottom and near the call-to-action buttons feels sensible.
• In all mockups, there is a separate checkbox for enabling / disabling STUN. This wouldn’t be necessary from a purely technical perspective, since we can derive the active state from the presence of a valid STUN address. However, I’d see the following benefits with an explicit checkbox:
  • The intent and state of a checkbox is clear and unambiguous.
  • The user would be able to toggle STUN on and off without having to delete the address value. (On our end, we can enable this behaviour by having a separate boolean parameter in settings.yml, like ustreamer_janus_stun_enabled, in addition to the host and port parameters.)
• In order to keep things simple and self-explanatory around the default STUN server operated by Google, I’d suggest to use a text button underneath the input field, which fills in the address of Google’s STUN server.

(1) Separate View

Screen.Recording.2023-07-24.at.17.49.14.mov

In this design, we’d transition the entire dialog to a new “state”/view. While this results in a clear visual division of the settings, it also poses some UX challenges – mainly what happens after you click “Save” of “Apply”:

• If you made unrelated changes in the main dialog, e.g. by having adjusted the frame rate slider before going to the advanced settings, would it also save those? (I.e., even if they are out of sight?) Or would it only save the advanced settings and disregard the main settings?
  • Both behaviours could be considered unexpected, and in both cases there are weird edge-cases.
• Would clicking “Save”/“Apply” terminate the dialog, or would the dialog transition back to the main video settings?
  • Transitioning back would make more sense here, similar to how we do it in the Security dialog when adding new users. However, we cannot apply the video settings atomically anymore then (e.g., via a single API call, so the backend logic would become more complex.)

(2) Expand / Collapse new section

Screen.Recording.2023-07-24.at.17.49.56.mov

In this design, we’d only collapse / expand the advanced video settings, where they are just a sub-section of the main video settings. In expanded view, this design looks more crowded, but the advantage is that the control flow in regards to saving/applying is obvious and transparent.

(3) Expand / Collapse STUN section

Screen.Recording.2023-07-24.at.18.30.19.mov

Similar to (2), but slightly less crowded in expanded view, would be to use the checkbox as means to toggle the STUN input controls. It’s a bit more clean overall, but the “STUN” topic is also more prominent in collapsed mode. The fact that the STUN config is considered to be an “advanced setting” also wouldn’t be as explicit anymore.

---

I’d prefer (2) or (3), despite the fact that it would make the video settings dialog look quite “comprehensive” in expanded view. (It probably would be our most complex and advanced view in the entire app.)

[jotaen4tinypilot]

(Update: it occurred to me that there is also a third option, which I added above.)

[cghague]

Please excuse the unprompted feedback! I note that @mtlynch mentioned placing this under the “Video” settings, but pragmatically it’s a network and privacy setting. Would it therefore make more sense to put it below the “Connection” header under “Security” settings instead?

[mtlynch]

Cool, these look good. I'd like to move forward with (2).

I think (1) is unexpected from a user's perspective since the flow would be different from most of our other dialogs.

(3) is pretty similar to (2), but I like the look of (2) better.

We also need the dialog to cover a few other things:

• 1:1 mapping - This is a separate option, and there are instances where a 1:1 mapping works when STUN doesn't, so I'd like for users to control both settings in this dialog.
• Explanations - Outside of the maybe 0.5% of TinyPilot users who also happen to be WebRTC experts, nobody is going to understand what these settings do. The dialog should give a brief (1-2 sentences) explanation of when the user might need that setting.
• Privacy disclosure - I don't want to scare users into not enabling STUN if they need it, but we should find some way to link to documentation we'll host about what the privacy implications are of a third-party STUN server.

I'd also like the Google STUN server to be the default if the user enables STUN rather than something they have to do extra. For probably 95% of users, they'll choose the Google server anyway, so we want to make it easy and obvious.

What do you think of instead of a text field, it's primarily a dropdown where the first option is the Google STUN server, then maybe a couple other public ones, and then the last option is "Custom" and we only show a free text field if the user selects "Custom?"

pragmatically it’s a network and privacy setting. Would it therefore make more sense to put it below the “Connection” header under “Security” settings instead?

Oh, good point.

I still think users will view this as a video settings feature rather than a security feature. It's a video settings feature that has privacy implications, but the reason that users would use this setting is to fix problems with their video rather than to adjust their security. Contrast this with something like user auth where the purpose of the feature is to adjust security rather than to achieve something else that has security implications.

[jotaen4tinypilot]

Cool, thanks for your feedback!

1:1 mapping - This is a separate option

Oh, the mapping somehow fell through the cracks… 🤔

I had one technical question about the STUN<>1:1 duality, which also would have UI implications.

Apart from that, I saw that the Janus nat_1_1_mapping config option allows for multiple, comma-separated IP addresses. Do we also want to support that, and if so, would it be worth to build a more convenient UI for that? E.g., one input field per address, with a + button or so to add more.

I’m not sure how common this scenario is (this is the original PR, for context), but I was wondering whether we should draw a line somewhere in regards to complexity on our end, or whether we should go all in with the flexibility that Janus provides here.

What do you think of instead of a text field, it's primarily a dropdown where the first option is the Google STUN server, then maybe a couple other public ones, and then the last option is "Custom" and we only show a free text field if the user selects "Custom?"

That sounds good!

[jotaen4tinypilot]

I’ve continued to work on this, and wanted to discuss a few things about the WIP state.

UI iteration

Screen.Recording.2023-08-16.at.18.11.20.mov

• The help text and general wording is only for demo purposes – I haven’t put much thought into it for now.
• Since we concluded that STUN and 1:1 mapping are mutually exclusive options, we could offer a dropdown button to toggle between the 3 possible modes – i.e., “STUN”, “1:1 mapping”, and “Regular” (=“off”).
  • The controls underneath the dropdown button would then be mode-specific. In the screen recording above, I only demonstrate the controls for STUN. For 1:1 mapping, we probably would have a similar input field to type in an address, but without the adjacent arrow button.
• For STUN, the adjacent arrow button next to the input field provides two options: “Google” and “Custom” – along with a guiding help text underneath. In the former case (“Google”), the input field is readonly and shows the predefined stun.l.google.com:19302 address, in the latter case (“Custom”) the input field is cleared and becomes editable.
  • We could consider to only show the input field when the user selects “Custom”, i.e. to hide away the stun.l.google.com:19302 address while “Google” is selected. On the one hand, such a UI would be cleaner, on the other hand, it would become more “bouncy” then, and maybe also less transparent.
• I found this curated list of public STUN servers, but from reviewing it I’m not sure there are other viable candidates for us, that we could provide along with the Google server?

Complexity

I realized that the tree of possible config states and choices has quite some complexity to it, and I think it’s tricky to model this in the UI in a way that is neither overwhelming nor bouncy / dynamic. The logical structure is this right now, where all nodes are conceptually different and mutually exclusive:

• Regular mode (i.e., no STUN, no 1:1 mapping)
• STUN
  • Google’s address (predefined, uneditable)
  • Custom address (editable)
• 1:1 mapping
  • Custom address (technically, Janus would even allow to enter multiple addresses here)

Implementation-wise, I’d anticipate that most effort and code complexity will be on the UI side. I think we should try to keep things tidy, e.g. by putting the advanced settings section into its own component, separate from the <video-settings-dialog> component.

In any event, I wanted to double-check that we are still on track with our original estimate, and the amount of work/time we want to invest into this feature. Potential alternatives could include:

• Reduce the feature-scope for the first iteration, e.g., like, STUN-only, and/or without custom server
• And/Or, to only offer a CLI script instead of a UI, and defer the UI to later

Writing the Janus configs

Since #1496 has landed, we can move forward with the backend side of things. I have created a minimal implementation, but unfortunately I ran into an issue. The configure-janus privileged script would be executed by different users / in two different contexts:

• By user root at Debian-package install time (e.g., on a fresh device)
• By user tinypilot at runtime

This creates two problems:

• (a) The update settings code is relying on the “relative user”, i.e. ~, so during Debian install time it may look at /root/settings.yml instead of /home/tinypilot/settings.yml, if the installation is performed as root user.
• (b) The tinypilot user would need to have write access to the Janus config file in order to regenerate the Janus configs at runtime. The Janus config file, however, resides at /etc/janus/janus.jcfg and is currently owned by root, so the tinypilot user cannot write to it.

For (a), we could change from ~ to /home/tinypilot in our code (see #1571). Regarding (b), I’m actually not sure what the best solution would be, and I was wondering whether we faced a similar situation in the past already? Otherwise, I’d need to think about this some more.

[mtlynch]

Cool, this is looking good so far!

We could consider to only show the input field when the user selects “Custom”, i.e. to hide away the stun.l.google.com:19302 address while “Google” is selected. On the one hand, such a UI would be cleaner, on the other hand, it would become more “bouncy” then, and maybe also less transparent.

Yeah, I'd like to abstract away the host:port except for when they choose "Custom." The user won't really care.

I found this curated list of public STUN servers, but from reviewing it I’m not sure there are other viable candidates for us, that we could provide along with the Google server?

These seem trustworthy:

• stun.ucsb.edu:3478 (we'd call it "University of California, Santa Barbara")
• stun.whoi.edu:3478 (we'd call it "Woods Hole Oceanographic Institution")
• stun.barracuda.com:3478 (we'd call it "Barracuda Networks")
• stun.gmx.de:3478 - You'd know better than I would. Is GMX a reputable brand in Germany?

Writing the Janus configs

The configure-janus privileged script would be executed by different users / in two different contexts:

• By user root at Debian-package install time (e.g., on a fresh device)
• By user tinypilot at runtime

I'm confused by this part. configure-janus is a privileged script, so wouldn't it always run as root? The tinypilot user can execute it with sudo privileges.

[jotaen4tinypilot]

Is GMX a reputable brand in Germany?

GMX has been around for ages in Germany, and they are primarily known for their email offering (@gmx.de), which many people still use (though nowadays it might not be the most popular choice anymore). Not sure how reliable their other services are, but at least the brand itself is somewhat reliable and reputable, as far as I can tell.

I'm confused by this part.

Yeah, me as well 😆 I think I got tripped up by the whole initial confusion stemming from #1571, and then I probably misinterpreted what I saw, or forgot to run the script as sudo or so… 🤦‍♂️ It’s all good now, so never mind!

[jotaen4tinypilot]

I submitted a patch for #1571 (#1572), since this issue here is blocked on the fix. (We have to run the configure-janus script via sudo, and the script currently reads from ~/settings.yml, i.e. it wouldn’t behave correctly since it’s then reading from /root/settings.yml instead of /home/tinypilot/settings.yml.)

[jotaen4tinypilot]

As discussed with @mtlynch in our video call today, we restrict the scope of this issue to STUN-only in the first step. We can add support for 1:1 mapping later. This procedure allows us to iterate in smaller chunks, and the STUN feature brings most value.

[jotaen4tinypilot]

I’ve created #1577 to track the 1:1 mapping functionality separately.

[mtlynch]

With the elimination of 1:1 mapping option, I think we can get rid of the "Connection method" dropdown and just have a single dropdown in the Advanced section called "STUN Server" where the options are:

• None (default)
• Google
• UCSB
• ...

[jotaen4tinypilot]

Just a quick update here, as “fyi”: I’ve been working a branch for the backend part (which is not completely finished yet, though), and also started to work on a frontend branch. I’ll stack the latter onto the former, and then plan to open both PRs at the same time, so that we can review/test/QA the feature in its entirety.

For the frontend, we might want to consider some refactoring in the <video-settings-dialog> component, otherwise – with the addition of the “Advanced Settings” section – the component will become pretty large code-wise. In the first step, I’d put everything into the <video-settings-dialog>, though, because then it’s easier to see how or whether we can split things up in a sensible way.

[jotaen4tinypilot]

@mtlynch could you briefly look over the following demo video regarding the frontend?

Screen.Recording.2023-09-28.at.16.43.33.mov

A few notes/questions from my side:

• I’m not sure about the wording of the introductory help text, could you revise that please?

  If your TinyPilot device resides in a remote network, and you access it from a non-public IP address, you need to facilitate the H.264 connection via an external STUN server. Learn more in the TinyPilot online documentation.

• Do we already have an online help page that we can link to? E.g., we could extend this one, or create a new one.
• The list of options in the dropdown seems slightly lengthy to me – do we want to keep all these external providers shown, or shall we boil it down a little bit? E.g., presenting just 2–3, instead of now 5? In addition to the external providers, we’d always have “Disabled” and “Custom”.

[mtlynch]

Cool, I think this looks really good!

It's slightly jarring that the control changes when the user selects "Custom," since I think it violates expectations a little of how dropdowns work.

The alternative is that picking "Custom" keeps the dropdown as-is, but it makes a new textbox appear, and that might be more confusing.

I don't think it's a huge deal, and it might be the best option we have, but it did throw me a little bit.

I’m not sure about the wording of the introductory help text, could you revise that please?

Sure, here's my suggestion for the text:

<p>A STUN server enables H.264 streaming when your web browser is unable to make a direct network connection to your TinyPilot device.</p>

<a href="https://tinypilotkvm.com/faq/stun-server">Do I need a STUN server?</a>

It oversimplifies things a bit, but we can go into more detail in the FAQ.

Do we already have an online help page that we can link to? E.g., we could extend this one, or create a new one.

I've created a ticket for a new one: https://github.com/tiny-pilot/tinypilotkvm.com/issues/1053

The list of options in the dropdown seems slightly lengthy to me – do we want to keep all these external providers shown, or shall we boil it down a little bit? E.g., presenting just 2–3, instead of now 5? In addition to the external providers, we’d always have “Disabled” and “Custom”.

The list of options in the dropdown seems slightly lengthy to me – do we want to keep all these external providers shown, or shall we boil it down a little bit? E.g., presenting just 2–3, instead of now 5? In addition to the external providers, we’d always have “Disabled” and “Custom”.

My thinking is that each one appeals to a slightly different type of user:

• Google: The recognizable, trustworthy choice for most people
• UCSB: For people who are skeptical of tech companies but trusting of academia.
• WHOI: For people who are skeptical of tech companies but trusting of academia.
• Barracuda Networks: For people who are okay with tech companies but distrustful of Google specifically
• GMX: For people who are distrustful of US companies and institutions

Maybe we can cut out WHOI since it's less recognizable than UCSB but serves the same purpose?

I guess the hypothetical user who's skeptical of Google but supports Barracuda is probably pretty rare too.

Let's cut out WHOI and Barracuda.

[jotaen4tinypilot]

I’m close to finishing the STUN feature, and just set up 3 PRs which are almost ready-to-go:

• Support for specifying STUN server: input validation (1/3) #1645
• Support for specifying STUN server: backend logic (2/3) #1646
• Support for specifying STUN server: frontend (3/3) #1647

Unfortunately, there are still some minor issues…

Public STUN servers

I could have noticed that earlier, but it appears that the aforementioned list of “public” STUN servers isn’t up-to-date anymore. Neither UCSB, WHOI, nor Barracuda appear to have a functioning STUN server anymore, at least I wasn’t able to connect to them.

I found another curated list of “public” STUN servers (see the valid_hosts.txt file specifically), which is automatically updated every hour. It seems to confirm that those 3 options aren’t available anymore.

I deliberately put “public” in quotes, because I came to realize that these STUN servers are not necessarily intended for usage by the “general public”. My impression is rather that some random people just gather random STUN servers that they happen to find somewhere, and compile them into these lists. The respective STUN providers, however, might not even be aware that their STUN server is used by random people on the internet. There is hence not really any reliability guarantee, and it’s also debatable whether it’s okay to just use and promote these servers only because they are publicly accessible.

So I think we might want to reconsider which “public” STUN servers we want to put into our UI. Google and GMX seem to be somewhat reliable and okay-to-use, but other than those, I’m not sure there are viable candidates? We can also research this topic further, and/or back-paddle with the 3rd-party STUN servers for now?

Systemd restart on failure

→ #1648

With this issue, it might happen that the user enters a “stuck” state, where (once the Janus service enters failed state), they are unable to apply any H264 settings, or even use H264 streaming, for 3 minutes.

[jotaen4tinypilot]

Just discussed with @mtlynch in person:

• Public STUN servers: we keep Google and GMX for now, and that’s it. We can still add more later.
• Systemd restart on failure: we work on that fix in parallel to the PRs, and we don’t consider it severe enough to block the release.

[jotaen4tinypilot]

@cghague:

The implementation work for integrating STUN into TinyPilot is almost done, so we have everything in place now for the user to specify a STUN server in the video settings dialog. The UI is not yet released, though, so the feature is not available to end-users. While we have tested everything inside the TinyPilot system (e.g., that our Python backend correctly applies the STUN parameters to the Janus config), we weren’t able to verify the external communication with the STUN server, and that our implementation is really fixing H.264 streaming in a scenario where STUN is actually required.

You know how to create a network setup where STUN is required for H.264 video streaming, right? Would you be able to carry out the following check on your end, to verify that our STUN implementation works “in the wild”? The testing procedure would be this:

1. Install this bundle on device: tinypilot-community-20231013T1550Z-1.9.1-49+b264dd7.tgz
   • I’d assume you know how to install nightly bundles on device, but just in case, here are our instructions.
   • Only for the records, the bundle was built off this branch, which contains all pending changes to make the STUN feature available through TinyPilot’s web UI.
2. Turn on H.264 video streaming in the video settings, but ensure that STUN is disabled in the “Advanced Settings” section of the video settings. (Save and apply settings.)
3. Check that your remote screen is unable to stream via H.264.
4. Now open the video settings again, and enable STUN by selecting the “Google” STUN server. (Save and apply settings.)
5. Check that your remote screen is now able to stream via H.264.
6. Repeat the test, but this time with the “GMX” STUN server.

You wouldn’t need to check anything else than that, as all pending changes have already passed code review and general QA.

[cghague]

@jotaen4tinypilot - Sure thing, I'd be happy to get this tested! For reference, you can test STUN without needing a complex network if you have two Internet connections with a public IPv4 address each. It should be as simple as port forwarding a TinyPilot device on one connection and then accessing it using the other. The H.264 connection should fail at first but work after enabling STUN. This scenario is a typical example of what a customer might face, so it's a good test to try.

@mtlynch - I don't have access to a second connection, so we used a temporary cloud server as a relay last time. I looked at the Exploding Servers tool, but I don't think the provisioned servers have any ports open to the Internet other than SSH. Can you please set the cloud server back up so I can test this?

[mtlynch]

@cghague - This is the type of scenario I built Exploding Servers to address, so it'd be good if we can make the tool work here so that you and Jan can self-provision. Are there specific ports that needs to be exposed, or do all the ports need to be exposed?

[mtlynch]

@cghague - From my experiments, servers on Exploding Servers already have all ports open. I just tried this:

sudo apt update && sudo apt install -y ncat

Then I tried listening on 8585 and then hitting the IP+ port from my browser, and the request came through fine:

$ nc -l 8585
GET / HTTP/1.1
Host: 51.159.140.152:8585
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0

Where did port listening fail for you?

[cghague]

Thanks @mtlynch, I must admit I didn't check if the ports were open, as it's standard for cloud servers to require users to allow ports explicitly, and I expected the cloud servers created by Exploding Servers would be the same. I couldn't see any options to allow ports in the web interface, so I assumed it still needed to be implemented.

@jotaen4tinypilot - I started working on this but realized that, as the bundle isn't for TinyPilot Pro, it doesn't support user authentication or HTTP encryption. I'm not comfortable exposing devices on my home network directly to the Internet without those features, so could you please provide a TinyPilot Pro build of this for me to test?

[mtlynch]

@cghague - Can you try generating the Pro bundle? Jan provided feature branch. If you merge those changes into a branch on Pro, it will generate a Pro bundle for you.

Keep in mind that we can't link a Pro bundle/build in this thread, as it's public.

[cghague]

Thanks @mtlynch. I've tried that and successfully merged these changes into a test branch on tinypilot-pro. I've given it a very quick smoke test - it appears to work, and the STUN changes are present. I should be able to test STUN in full tomorrow - I'll let you both know how it goes!

[cghague]

I've tested this, and the results are as follows:

Format	STUN	Result
MJPEG	-	MJPEG works
H.264	Disabled	H.264 fails
H.264	Google	H.264 works
H.264	GMX	H.264 fails
H.264	Custom (stun.l.google.com:3478)	H.264 works
H.264	Custom (<my isp stun server>:3478)	H.264 fails

I can see in the service logs that Janus is correctly using both the GMX STUN server and the STUN server provided by my ISP when selected. They return the correct IP address when queried during the Janus startup procedure. This result suggests that the server side is working correctly.

However, the client fails to identify the correct candidate using anything other than the Google STUN servers. I'm unsure why this is, and there's nothing helpful in about:webrtc or the browser console.

Is there anything else I can test here?

[mtlynch]

@cghague - The client and server can use separate STUN servers, right? I'm wondering if there's some kind of client bug that accidentally pins to Google even if we specify a different server, but that wouldn't explain why it breaks if the server uses a different STUN server.

@jotaen4tinypilot - Any ideas?

[jotaen4tinypilot]

@cghague thanks for testing and providing such a detailed report!

Are you sure that your ISP’s STUN server is usually working, though? We had already discovered earlier that only because a STUN server is advertised somewhere it doesn’t mean that it is also functioning correctly, or generally available at all. The same might be the case for GMX – we don’t have any definite verification so far that GMX’s STUN server is usable for our purposes, except that the server is reachable.

Could you do me a favour and give the following STUN servers a try? According to this random STUN connection testing website that I found, they seem promising.

• freestun.net + 3479 (not 3478!)
• foxy.tel + 3479 (ditto)
• stun1.l.google.com + 19302 (i.e., stun1. instead of stun.)

Should these fail, could you also double-check in a private browser window? Just so that we can rule out any client-side quirks, such as caching.

Thoughts on further investigation

I’d think it would be worthwhile to wait whether any of the 3 aforementioned alternatives will be successful. My level of trust in regards to publicly available STUN servers is slightly damaged, based on our earlier findings about their availability or reliability.

Another factor is that there are two RFCs for the STUN protocol (3489 and 5389), and from what I’ve read, a STUN connection can also fail if the server still uses the old RFC while the client uses the new one.

If it continues to only work with Google, then I think we should investigate this more deeply. It might then probably make sense for me to produce a STUN scenario on my end. Otherwise/additionally, it might also be an option for me and @cghague to connect for a debugging session.

Regarding the client library: Google’s STUN server indeed sticks out strangely right now, but I think we should also gather more data points before making conclusions. In any event, in the Janus client JS library, stun.l.google.com:19302 is the hard-coded default. However, I just checked that #1657 (which is included in the branch for end-to-end testing) correctly overrides the iceServers variable; there is also no other mention of the Google server address in the Janus client library. So I’d currently say it should work in theory…

[cghague]

I carried on looking into this after my last comment. I had many of the same thoughts as @jotaen4tinypilot, and I've ruled out several possible problems and made a small amount of progress.

I started by setting a few breakpoints using the debugger in the browser's developer tools, and the correct value appears to be getting set and propagated correctly. For a belt-and-braces approach, I also tried intentionally mangling the default value in janus.js, which didn't change the outcomes, so we can safely assume that's not the issue.

My ISP markets itself at technical users (they're genuinely excellent and are even "xkcd/806" compliant), so I'd be surprised if their STUN server didn't work. However, to be safe, I verified all of the STUN servers I used with the stuntman-client tools, and they all responded as I expected. Each STUN server identified the correct external IP address and established a viable port number that WebRTC could use.

I finally tried connecting from Safari on my iPhone, and it did work with the STUN server provided by my ISP. My iPhone uses the same Internet connection as my desktop, so I have no explanation for the different outcomes. I tested Safari on my desktop to rule out a browser issue, and it behaved identically to Firefox (which I used for the earlier tests due to the excellent about:webrtc utility).

The above findings suggest that STUN makes a difference and that the changes made by @jotaen4tinypilot work and are configuring Janus correctly. Consequently, I speculate that the failures are likely due to the testing environment we're using. I'll try a few more things tomorrow, including the STUN servers @jotaen4tinypilot suggested.

[jotaen4tinypilot]

@cghague Thanks for looking into this so deeply!

I'll try a few more things tomorrow, including the STUN servers @jotaen4tinypilot suggested.

Just wondering, did you get a chance already to try the other STUN servers?

In general, I’m thinking that if we have evidence that the STUN feature does the job even with custom servers, then we should be good to proceed with releasing this. I’m sure there are a few things left for us to be learned about the whole STUN topic, though, so maybe/hopfully we are be able to keep refining the feature as needed.

[cghague]

Thanks @jotaen4tinypilot! I tested the alternate servers, but initially, none worked, so I dug into this further. I tried:

• Using multiple alternative STUN servers.
  • Only the default Google server worked somewhat reliably.
  • Even the Google server failed occasionally.
  • Other servers worked, but only very rarely.
• Trying private browsing mode.
  • This variable had no noticeable effect.
• Testing with different browsers.
  • Safari and Firefox both behaved the same.
• Enabling full trickle STUN mode.
  • This setting shares candidates from the client with the server. According to the Janus documentation, this setting can help if both ends of a WebRTC connection are behind NAT.
  • There was no noticeable impact.
• Locking Janus to the tun0 interface.
  • This change prevented the spurious back channel candidates from appearing but didn't appear to fix the issue overall.
• Disabling host candidates in Firefox.
  • This change prevented unwanted local host addresses from being tried but didn't help the overall issue.
• Disabling host obfuscation in Firefox.
  • This change didn't impact the number of successful connections, but it did make it easier for me to identify where the connection failures were happening.
• Connecting the client via a cellular hotspot.
  • I could not establish any H.264 connectivity over a cellular connection, but a quick speed test revealed horrendously poor ping and throughput rates, so this test probably isn't reliable.
• Disabling IP address tracking prevention in macOS.
  • This change eliminated the nonsensical candidates being suggested but didn't seem to help obtain valid candidates.
• Testing in a virtual machine.
  • This test was a false start, as I soon discovered H.264 decoding doesn't work in UTM virtual machines on macOS.

During these tests, I observed the about:webrtc page to see where the failures occurred. While there were occasional failures to provide candidate ports on the remote side (i.e., the TinyPilot device), the overwhelming majority were happening on the client side.

Based on my findings, we can deduce that the issue is affecting the client side of the connection, but we don't have enough information to say why. It's plausible that the complex testing environment required to force STUN usage on my home network is causing problems.

I'd suggest that our next step would be to try and arrange a less complicated STUN testing environment. The ideal would be for us to implement the following basic scenario:

TinyPilot -> LAN <- Router (port forward) -> Internet <- Router -> LAN <- Client

I don't have access to two suitable connections, so this isn't something I can try at home. Do either of you have the resources to attempt this?

[jotaen4tinypilot]

Oh ok, interesting findings… 🤔

I’ll try to set up port forwarding in my home network, and then try to access my device through a remote connection (e.g., my mobile carrier). Maybe/Hopefully that works out 🤞

[jotaen4tinypilot]

I’m afraid to report that I wasn’t able to produce a network setup on my end where my device is publicly exposed to the internet. I couldn’t figure out what the problem is exactly, but I wasn’t able to get my router to forward the ports of my device, and to exempt them from the firewall. So when I tried reaching my device (connected to my home network) at it’s IP address via my mobile carrier’s network, it couldn’t establish a connection.

The router which I own is an average consumer model provided by my ISP, so it’s nothing advanced, and it also only offers limited customization options. I’m therefore not sure it’s worth to pursue this route any further, as I think I’ve exhausted all config options and combinations I could think of.

I’m wondering how best to proceed with this – current thoughts:

• @mtlynch would you be able (or have capacity) to arrange a suitable network setup for verifying the STUN functionality by using the end-to-end bundle build?
• I could try to debug and investigate my network/router setup further, maybe there is something that I overlooked after all. I’m still puzzled why it wouldn’t work on my end, but as mentioned above, I don’t know what else to try, or what I possibly could have done wrong.
• I could try to set up a proxy server to expose my device and access it remotely. I haven’t done that before, however, so it might take a while for me to get things going. I’m not sure this is a worthwhile option, though, since @cghague already tried with a proxy setup, and reported lots of complications.
• We move forward with the release based on the evidence that we have so far. As I understood @cghague, the STUN functionality seems to work in general, and the problems might indicate network-related flakiness/reliability issues rather than a fundamental problem with our implementation. As far as I see, our STUN configuration with Janus is also “by the book”. There’d still be some risk, though.
  • As a potential middleground-solution, we could also temporarily limit the STUN dropdown to “Disabled” and “Google” for now (e.g., by simply hiding the “GMX” and “Custom” option from the UI). We have evidence that the “Google” option works, so maybe it’s safe for us to release that. In the meantime, we could try to tackle the testing situation internally, and defer the decision how to proceed with the “GMX” and “Custom” option when we have a better picture.

[cghague]

My take on this is that in scenarios where H.264 already works, it will continue to do so regardless of whether STUN is enabled, and in situations where H.264 doesn't currently work, enabling STUN will either fix the problem or have no effect. It therefore follows that there is no downside to providing the STUN functionality. We should, however, continue investigating this to understand why some servers don't work, as that will allow us to provide better customer support.

[mtlynch]

Yeah, I think we should move forward with the current implementation. It seems strange that there would be something specific about Janus' client implementation that would make Google the only valid STUN server.

Given the cost of testing the STUN setup and the possibility that the effect we're seeing is just testing error, I think the current implementation is worth moving forward with.

@cghague - Can you share more detail about the debugging steps you took in case we want to re-do that in the future? Was it all through about:webrtc or were there other interfaces? I'm curious about how to do these parts:

• Enabling full trickle STUN mode.
• Locking Janus to the tun0 interface.
• Disabling host candidates in Firefox.
• Disabling host obfuscation in Firefox.

[cghague]

I primarily based my investigations on the output of about:webrtc and the Janus logs. The Janus logs are accessible via journalctl -u janus. I performed the following actions with information from the Janus logs:

• Confirmed that the STUN server was accessible.
• Checked a valid external IP and port combination was found.
• Read status messages for individual connections.

From the about:webrtc tool, the most helpful information was:

• The list of remote candidates.
• The list of local candidates.
• Overall WebRTC connection status.
• The browser's WebRTC logs.

In terms of tweaking the behavior of Janus, I made all changes in janus.jcfg. I enabled full trickle mode by adding full_trickle = true to the nat section and locked Janus to the tun0 interface by adding ice_enforce_list = "tun0", also to the nat section. Note that editing the STUN server via TinyPilot will revert these changes.

Regarding Firefox, the browser allows you to override many STUN/TURN settings in about:config. The setting titles are self-explanatory, but given how many are available, you might find it helpful to filter for either stun or ice to narrow the list down. The two specific options I used were:

• media.peerconnection.ice.obfuscate_host_addresses
• media.peerconnection.ice.no_host

Any active overrides also conveniently show up in about:webrtc.

[mtlynch]

@cghague - Thanks, that's helpful!