feat(webcam): add support for go2rtc webrtc #1651

meteyou · 2023-11-12T20:09:34Z

Description

This PR add support for go2rtc webrtc streams.

Related Tickets & Documents

fixes #1396

Mobile & Desktop Screenshots/Recordings

none

[optional] Are there any post-deployment tasks we need to perform?

add this streamer to the docs

Signed-off-by: Stefan Dej <meteyou@gmail.com>

github-actions · 2023-11-12T20:10:22Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	1	0
en.json	0	0

vajonam · 2023-11-13T20:00:42Z

Thanks for this! Amazing work as always!

meteyou · 2023-11-14T01:50:48Z

@vajonam how do you find the stream? i could only test it with a shaky "file stream". but it didn't run in the go2rtc view either, so i'm still waiting for feedback from @mryel00.

vajonam · 2023-11-14T13:05:47Z

@meteyou I haven't tried this yet, my build environment is still stuck on 2.4.1 when I did my last set of PRs. Do can you send me an alpha build with this in I can test it out for you?

Thanks

mryel00 · 2023-11-14T13:21:34Z

The builds for PRs should be available inside the checks. Just hit Show all checks and select Details for the Build/build run. Then go to summary (on the left). Direct link
I tested this with reencoding of a mjpg stream and directly using an USB cam. The reencoding worked flawlessly, just one error on the console. The direct use didn't work great for me, but there were problems with the go2rtc stream too, so not a problem directly affecting this service.

vajonam · 2023-11-14T14:07:03Z

@mryel00 thanks I did not know this! wow! testing now!

vajonam · 2023-11-14T14:12:22Z

@meteyou @mryel00

I am using frigate / go2rtc behind an ssl, so need it load wss. at the moment when I try this I am seeing.

Mixed Content: The page at 'https://trident.domain.com/' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://frigate.domain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio'. This request has been blocked; this endpoint must be available over WSS.

the url I that I set in the settings was

https://frigate.domain.com/live/webrtc/stream.html?src=trident&mode=webrtc

vajonam · 2023-11-14T15:35:32Z

moving to the direct URL.

http://figate.domain.com/live/webrtc/stream.html?src=trident seems to work! The stream works really well on both web and mobile ios!

I think there is still an issue when using HTTPS or WSS

vajonam · 2023-11-14T15:47:56Z

Also just sitting there streaming I am seeing this.

Seems like its terminating after a few minutes, if I am using the stream.html that go2rtc provides I dont see such a problem.

[WebRTC go2rtc] connecting to ws://frigate.domain.com/live/webrtc/api/ws?src=trident&media=video+audio
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] connecting to ws://frigate.domain.com/live/webrtc/api/ws?src=trident&media=video+audio
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] connecting to ws://frigate.domain.com/live/webrtc/api/ws?src=trident&media=video+audio
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] connecting to ws://frigate.domain.com/live/webrtc/api/ws?src=trident&media=video+audio
WebrtcGo2rtc-f31a43b1.js:1 [WebRTC go2rtc] open

vajonam · 2023-11-14T15:58:10Z

on the left is the mainsail window, on the right is the same stream URL loaded in a browser directly.
you can see it disconnects and reconnects.

Untitled.Project.mp4

vajonam · 2023-11-14T16:38:03Z

@vajonam how do you find the stream? i could only test it with a shaky "file stream". but it didn't run in the go2rtc view either, so i'm still waiting for feedback from @mryel00.

@meteyou
You can find the streams on the go2rtc dashboard, it looks like

clicking on the links takes you to

Signed-off-by: Stefan Dej <meteyou@gmail.com>

github-actions · 2023-11-14T23:36:13Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	1	0
en.json	0	0

meteyou · 2023-11-14T23:37:00Z

@vajonam thx for the report. i pushed a fix for that. please can you doublecheck, if it works with and without ssl? you only have to open mainsail with ssl and mainsail will also switch the webcam url to ssl (both have to be SSL or without SSL. mixed is not allowed in the browser)

vajonam · 2023-11-15T11:35:32Z

@meteyou thanks the commit has fixed the ssl/non ssl issue. Both works

However the periodic disconnection is still an issue. This is a non issue when using the html from go2rtc. I suspect there is some other issue that closes the websocket? Other webrtc instances (same stream on home assistant, or frigate itself) do not show this disconnection. I had posted a video about the disconnects.

vajonam · 2023-11-15T15:05:55Z

This is what the console logs show during the Werbrtc socket disconnections.

[WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] connecting to wss://frigate.mydomain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-343c6b27.js:1 [WebRTC go2rtc] open

src/components/webcams/streamers/WebrtcGo2rtc.vue

Signed-off-by: Stefan Dej <meteyou@gmail.com>

meteyou · 2023-11-15T19:10:19Z

@vajonam i pushed a fix. pls double-check it, if it fix the disconnecting issue.

github-actions · 2023-11-15T19:10:21Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	1	0
en.json	0	0

vajonam · 2023-11-15T19:19:44Z

testing it now.

vajonam · 2023-11-15T19:28:06Z

@meteyou still one error. but the disconnects and reconnects still seem to be around that has not fixed the issue.

Browser: chrome 119.0.0, OS: Linux
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] connecting to wss://frigate.domain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
vuetify-69452629.js:5 TypeError: Cannot set properties of undefined (setting 'srcObject')
    at o.start (WebrtcGo2rtc-aae87515.js:1:1870)
    at o.expandChanged (WebrtcGo2rtc-aae87515.js:1:1416)
    at Bt (vuetify-69452629.js:5:24599)
    at t.$watch (vuetify-69452629.js:5:43538)
    at Ns (vuetify-69452629.js:5:43083)
    at ou (vuetify-69452629.js:5:42970)
    at tu (vuetify-69452629.js:5:41676)
    at t._init (vuetify-69452629.js:5:43886)
    at new a (vuetify-69452629.js:5:45148)
    at zh (vuetify-69452629.js:5:37820)
er @ vuetify-69452629.js:5
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] connecting to wss://frigate.domain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] open
index-8cab4652.js:27 PWA registration error: DOMException: Failed to register a ServiceWorker for scope ('https://mainsail.domain.com/') with script ('https://mainsail.domain.com/sw.ts'): The script has an unsupported MIME type ('text/html').
onRegisterError @ index-8cab4652.js:27
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] connecting to wss://frigate.domain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] open
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] close
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] terminating
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] connecting to wss://frigate.domain.com/live/webrtc/api/ws?src=trident&mode=webrtc&media=video+audio
WebrtcGo2rtc-aae87515.js:1 [WebRTC go2rtc] open

vajonam · 2023-11-15T20:54:32Z

I am re testing. Sometimes the PWA is cached am messed up my testing. I am still testing to see if the connections cause the stream to restart.

Signed-off-by: Stefan Dej <meteyou@gmail.com>

github-actions · 2023-11-15T21:35:22Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	1	0
en.json	0	0

vajonam · 2023-11-15T21:53:16Z

@meteyou. The tag for the video tag worked. However the disconnection of the stream is still happening. I will contact you on discord, as I would like to send you the video privately.

vajonam · 2023-11-16T02:35:56Z

@meteyou I was doing some more digging and I can see the following errors on go2rtc only from mainsail.

ERR github.com/AlexxIT/go2rtc/internal/api/ws/ws.go:101 > host=frigate.domain.com origin=https://mainsail.domain.com error="websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header"

Apparently missing the Upgrade websocket header?

meteyou · 2023-11-16T07:06:09Z

Websocket is a browser internal package/lib and I use the identical implementation as in the example file.

vajonam · 2023-11-28T21:48:50Z

Okay. I will investigate. There are 2 reverse proxies, one in the frigate software itself that proxies go2rtc, then for me to apply ssl to that connection. as far as I know until know I have enabled WSS support for both (otherwise it wouldn't even connect) I will keep searching!

mryel00 · 2023-11-28T21:59:39Z

@vajonam I just checked the go2rtc version that frigate is using. It seems like they are still on v1.2 that got released in february. They will update it with frigate v0.13.
Could you check if the problem still exists with go2rtc v1.8.4? That is currently the newest release and the version my tests are based on.

vajonam · 2023-11-28T22:00:55Z

I am running 1.8.4 part of beta 13, on the nightly builds for that. but I am now trying to eleminate the frigate nginx that has keepalive_timeout 65, to see if that will help.

meteyou · 2023-11-28T22:12:26Z

ok. that sounds like the 1 minute cycle and could be the issue.

vajonam · 2023-11-28T22:26:30Z

I bypassed that 1st frigate provided nginx (less control over), and then on my SSL reverse proxy added

proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;

so far up for a more than a minute or so.

this might be a good idea to point out timeout / disconnection issues in the docs.

meteyou · 2023-11-28T22:41:07Z

@vajonam i don't think that these settings in the second reverse proxy can fix the first one. i think these settings have to be made directly in the first one.

but I just realized that I still have to write the docs in general...

vajonam · 2023-11-28T22:44:17Z

@meteyou I think you misunderstood me (we are both saying the same time) the only way to get it work with frigate provided go2rtc is bypass its nginx, because it too late if that is used. (expose port directly on that container) , and then if doing any other reverse proxy maybe apply the settings proxy_*_timeout 7d might be extreme by maybe an hour or so.

Signed-off-by: Stefan Dej <meteyou@gmail.com>

github-actions · 2023-12-09T10:46:47Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	16	0
en.json	0	0

pedrolamas · 2023-12-09T11:02:30Z

FWIW, in Fluidd I ended up porting the full WebRTC functionality in stream.html to add support for this, which seems to have fixed the issues @vajonam was experiencing.

I am considering porting the rest of the code so that it would support not just WebRTC but the whole thing (that would mean keeping "webrtc-go2rtc" camera type as legacy and adding a new "go2rtc", or something like that).

That way all users would need to do is supply the full stream.html address and we will handle it on our side.

meteyou · 2023-12-09T11:20:06Z

FWIW, in Fluidd I ended up porting the full WebRTC functionality in stream.html to add support for this, which seems to have fixed the issues @vajonam was experiencing.

I am considering porting the rest of the code so that it would support not just WebRTC but the whole thing (that would mean keeping "webrtc-go2rtc" camera type as legacy and adding a new "go2rtc", or something like that).

That way all users would need to do is supply the full stream.html address and we will handle it on our side.

thx for this information. but the issue from @vajonam is a wrong reverse proxy (which i said from the beginning, but he rather switched to fluidd and left out functions to avoid his misconfig, pls check the comments above).

I will now merge this streamer-client so that we can use it to run tests for crowsnest. I can't say at the moment whether we will then go deeper into go2rtc or drop it in the future... The situation with the RPI5 without HW encoder is very bad and we have to look in all directions...

vajonam · 2023-12-09T11:26:56Z

To be clear, yes the reverse proxy was an environment issue, which did cause reconnects that I was reporting. But I found other issues specific to iOS and Safari that were addressed in the rewrite.

meteyou · 2023-12-09T11:38:43Z

@vajonam if you mean the reconnect issue in your second PR in fluidd (fluidd-core/fluidd#1257), that's exactly why i have the reconnect in my implementation. that's why i didn't want to remove it...

if you mean another issue, pls explain it...

vajonam · 2023-12-09T11:44:00Z

This was manifested as an issue that was only happening on iOS, where the connection would stall and freeze and not reconnect.

It was specific to how the state change of the peer connection socket was handled on disconnected state.

I looked at the stream.html that didn't have this issue and started looking for the differences, I did add a commit for this on the branc that addressed this issue. This was then refactored by @pedrolamas

vajonam · 2023-12-09T11:47:56Z

Specifically this is what helped the issue on iOS.

fluidd-core/fluidd@14a8f80

meteyou · 2023-12-09T12:06:09Z

Specifically this is what helped the issue on iOS.

fluidd-core/fluidd@14a8f80

thx for this link! it looks like i missed this in this implementation. i also use this in the "camera-streamer" implementation. i think i should add this also here...

Signed-off-by: Stefan Dej <meteyou@gmail.com>

…ected streams Signed-off-by: Stefan Dej <meteyou@gmail.com>

Signed-off-by: Stefan Dej <meteyou@gmail.com>

github-actions · 2023-12-09T13:02:06Z

Language file analysis report:

File	Missing Keys	Unused Keys
de.json	16	0
en.json	0	0

meteyou · 2023-12-09T13:02:27Z

@vajonam i added the support of this here: 701fe6d

thx again for the hint!

meteyou added 2 commits November 12, 2023 21:06

feat(webcam): add support for go2rtc webrtc

423a909

Signed-off-by: Stefan Dej <meteyou@gmail.com>

locale(en): add english locale

1534f23

Signed-off-by: Stefan Dej <meteyou@gmail.com>

fix: use ssl websocket connection when the interface is secure

a251760

Signed-off-by: Stefan Dej <meteyou@gmail.com>

vajonam reviewed Nov 15, 2023

View reviewed changes

src/components/webcams/streamers/WebrtcGo2rtc.vue Outdated Show resolved Hide resolved

fix: stop scheduledRestart after opening the websocket

2adbfc7

Signed-off-by: Stefan Dej <meteyou@gmail.com>

fix: fix issue when video tag not exists

a828a1d

Signed-off-by: Stefan Dej <meteyou@gmail.com>

rackrick approved these changes Nov 29, 2023

View reviewed changes

vajonam approved these changes Nov 29, 2023

View reviewed changes

meteyou added 2 commits December 9, 2023 11:36

Merge branch 'develop' into feat/webcam-go2rtc

853484c

feat: add option to enable/disable audio

565fc72

Signed-off-by: Stefan Dej <meteyou@gmail.com>

meteyou added 3 commits December 9, 2023 13:28

fix: check wrong url before converting it

6f3466c

Signed-off-by: Stefan Dej <meteyou@gmail.com>

feat: add support for connectionstatechange to restart failed/disconn…

701fe6d

…ected streams Signed-off-by: Stefan Dej <meteyou@gmail.com>

feat: restart stream if enableAudio changes

2b7bbc2

Signed-off-by: Stefan Dej <meteyou@gmail.com>

rackrick approved these changes Dec 9, 2023

View reviewed changes

meteyou merged commit 7939357 into mainsail-crew:develop Dec 10, 2023
10 checks passed

meteyou deleted the feat/webcam-go2rtc branch December 10, 2023 10:02

4rnoP pushed a commit to 4rnoP/mainsail that referenced this pull request Dec 13, 2023

feat(webcam): add support for go2rtc webrtc (mainsail-crew#1651)

6fc531a

4rnoP pushed a commit to 4rnoP/mainsail that referenced this pull request Dec 13, 2023

feat(webcam): add support for go2rtc webrtc (mainsail-crew#1651)

5ef6164

Xelinor mentioned this pull request Dec 26, 2023

Add support for go2rtc Clon1998/mobileraker#304

Closed

feat(webcam): add support for go2rtc webrtc #1651

feat(webcam): add support for go2rtc webrtc #1651

Conversation

meteyou commented Nov 12, 2023 • edited

Description

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings

[optional] Are there any post-deployment tasks we need to perform?

github-actions bot commented Nov 12, 2023

vajonam commented Nov 13, 2023

meteyou commented Nov 14, 2023

vajonam commented Nov 14, 2023

mryel00 commented Nov 14, 2023

vajonam commented Nov 14, 2023

vajonam commented Nov 14, 2023 • edited

vajonam commented Nov 14, 2023 • edited

vajonam commented Nov 14, 2023

vajonam commented Nov 14, 2023

vajonam commented Nov 14, 2023 • edited

github-actions bot commented Nov 14, 2023

meteyou commented Nov 14, 2023

vajonam commented Nov 15, 2023 • edited

vajonam commented Nov 15, 2023

meteyou commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

vajonam commented Nov 15, 2023

vajonam commented Nov 15, 2023

vajonam commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

vajonam commented Nov 15, 2023

vajonam commented Nov 16, 2023

meteyou commented Nov 16, 2023

vajonam commented Nov 28, 2023

mryel00 commented Nov 28, 2023

vajonam commented Nov 28, 2023 • edited

meteyou commented Nov 28, 2023

vajonam commented Nov 28, 2023

meteyou commented Nov 28, 2023

vajonam commented Nov 28, 2023 • edited

github-actions bot commented Dec 9, 2023

pedrolamas commented Dec 9, 2023

meteyou commented Dec 9, 2023

vajonam commented Dec 9, 2023

meteyou commented Dec 9, 2023

vajonam commented Dec 9, 2023 • edited

vajonam commented Dec 9, 2023

meteyou commented Dec 9, 2023

github-actions bot commented Dec 9, 2023

meteyou commented Dec 9, 2023

meteyou commented Nov 12, 2023 •

edited

vajonam commented Nov 14, 2023 •

edited

vajonam commented Nov 14, 2023 •

edited

vajonam commented Nov 14, 2023 •

edited

vajonam commented Nov 15, 2023 •

edited

vajonam commented Nov 28, 2023 •

edited

vajonam commented Nov 28, 2023 •

edited

vajonam commented Dec 9, 2023 •

edited