I end up with an end-to-end latency of around 800 milliseconds

[vguenther]

I tried to get this running, but found several issues, which might or might not be related to each other. Any idea what's going wrong?

Issues:

  1. I end up with an end-to-end latency of around 800 milliseconds – far beyond the figures in the description. I found this out by by directing the camera to the display and at the same time to a stopwatch app and taking a photo of the setup.
2. The output on the display shows checkerboard-like compression artefacts and is virtually unusable.
3. Though I can move the cursor on the RPI-screen with the mouse, I cannot change the configuration-settings.
4. gs shows error messages on startup (I added LOGE("{}", SDL_GetError()); in the SDLCHK macro)

pi@fpvbusterpi:~/esp32-cam-fpv/gs $ sudo -E LD_LIBRARY_PATH=/usr/local/lib ./gs
(I) src/PI_HAL.cpp: 125: Initializing pigpio
(I) src/PI_HAL.cpp: 283: Drivers: 4
(I) src/PI_HAL.cpp: 286: Driver 0: x11
(I) src/PI_HAL.cpp: 286: Driver 1: KMSDRM
(I) src/PI_HAL.cpp: 286: Driver 2: dummy
(I) src/PI_HAL.cpp: 286: Driver 3: evdev
(I) src/PI_HAL.cpp: 295: Mode 0: 1280x1024
(I) src/Comms.cpp: 718: Radiocap header size: 11, IEEE header size: 24
(I) src/Comms.cpp: 581: Opening interface wlan1 in monitor mode
(I) src/Comms.cpp: 282: DLT_IEEE802_11_RADIO Encap
(I) src/Comms.cpp: 581: Opening interface wlan2 in monitor mode
(I) src/Comms.cpp: 282: DLT_IEEE802_11_RADIO Encap
(I) src/Video_Decoder.cpp: 160: SDL window: 31254080
(E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161
(E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS)
(I) src/Video_Decoder.cpp: 160: SDL window: 31254080
(E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161
(E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS)
(I) src/Video_Decoder.cpp: 160: SDL window: 31254080
(E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161
(E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS)
(I) src/Video_Decoder.cpp: 160: SDL window: 31254080
(E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161
(E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS)

What's my setup and what did I do?

 
• I’m using an RPI4 with a Buster image and TL-WN722 v2 adapters together with appropriate driver from https://github.com/aircrack-ng/rtl8188eus.
• I provided the prerequisites as described
• I fixed the obvious issue in Makefile in order to find all headers
• I had to change Comms.cpp ( pcap_set_rfmon(pcap.pcap, 0) < 0) ) in order to work around a PCAP problem (pcap_activate … too many arguments)
• WLAN1 und WLAN2 are successfully brought up in monitoring mode before startup of gs
• I made sure ESP32-CAM was running continuously
• I accessed RPI by ssh to start gs
• I did serial monitoring on ESP32

Typical output of gs:

…
(I) src/main.cpp: 104: RX len: 77910, RSSI: -65, Latency: 158/158/158
(I) src/main.cpp: 104: RX len: 99960, RSSI: -69, Latency: 144/144/144
(I) src/main.cpp: 104: RX len: 104370, RSSI: -47, Latency: 168/168/168
(I) src/main.cpp: 104: RX len: 113190, RSSI: -81, Latency: 162/162/162
(I) src/main.cpp: 104: RX len: 110250, RSSI: -47, Latency: 130/130/130
(I) src/main.cpp: 104: RX len: 108780, RSSI: -65, Latency: 140/140/140
(I) src/main.cpp: 104: RX len: 104370, RSSI: -69, Latency: 144/144/144
(I) src/main.cpp: 104: RX len: 107310, RSSI: -47, Latency: 167/167/167
(I) src/main.cpp: 104: RX len: 98490, RSSI: -47, Latency: 160/160/160
(I) src/main.cpp: 104: RX len: 108780, RSSI: -81, Latency: 195/195/195
(I) src/main.cpp: 104: RX len: 97020, RSSI: -65, Latency: 126/126/126
(I) src/main.cpp: 104: RX len: 102900, RSSI: -95, Latency: 180/180/180
(I) src/main.cpp: 104: RX len: 101430, RSSI: -95, Latency: 130/130/130
(I) src/main.cpp: 104: RX len: 104370, RSSI: -95, Latency: 141/141/141
…

HAL widget on DISPLAY reports <5ms/frame and >200fps

Typical output of esp32-cam:

…
MEMORY Before Loop:
Heap summary for capabilities 0x00000004:
At 0x3f800000 len 4194303 free 979859 allocated 3211264 min_free 979859
largest_free_block 966656 alloc_blocks 257 free_blocks 1 total_blocks 258
At 0x3ffae6e0 len 6432 free 4 allocated 4136 min_free 4
largest_free_block 0 alloc_blocks 35 free_blocks 0 total_blocks 35
At 0x3ffb7920 len 165600 free 4 allocated 162832 min_free 4
largest_free_block 0 alloc_blocks 153 free_blocks 0 total_blocks 153
At 0x3ffe0440 len 15072 free 4 allocated 12876 min_free 4
largest_free_block 0 alloc_blocks 10 free_blocks 0 total_blocks 10
At 0x3ffe4350 len 113840 free 6888 allocated 104528 min_free 6872
largest_free_block 6784 alloc_blocks 68 free_blocks 2 total_blocks 70
Totals:
free 986759 allocated 3495636 min_free 986743 largest_free_block 966656
WLAN S: 29520, R: 70, E: 0, D: 0, % : 99 || FPS: 1, D: 68846 || D: 0, E: 0
Wifi rate changed from 9 to 13
Camera resolution changed from 3 to 0
Target FPS changed from 30 to 0
Camera quality from 8 to 63
WLAN S: 119556, R: 280, E: 0, D: 0, % : 97 || FPS: 48, D: 537492 || D: 0, E: 0
WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 123820 || D: 0, E: 0
WLAN S: 119556, R: 350, E: 0, D: 0, % : 95 || FPS: 54, D: 124757 || D: 0, E: 0
WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 124719 || D: 0, E: 0
WLAN S: 121032, R: 210, E: 0, D: 0, % : 95 || FPS: 54, D: 125311 || D: 0, E: 0
WLAN S: 119556, R: 350, E: 0, D: 0, % : 97 || FPS: 53, D: 125379 || D: 0, E: 0
WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 125704 || D: 0, E: 0
WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 53, D: 123120 || D: 0, E: 0
WLAN S: 121032, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 122297 || D: 0, E: 0
WLAN S: 119556, R: 210, E: 0, D: 0, % : 95 || FPS: 53, D: 124083 || D: 0, E: 0
WLAN S: 119556, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 125109 || D: 0, E: 0
WLAN S: 119556, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 124289 || D: 0, E: 0
WLAN S: 119556, R: 350, E: 0, D: 0, % : 97 || FPS: 53, D: 124572 || D: 0, E: 0
WLAN S: 121032, R: 350, E: 0, D: 0, % : 95 || FPS: 53, D: 123323 || D: 0, E: 0

…

[jeanlemotan]

Hi
I don't know what could be going on honestly. It looks like something on the PI side - it's buffering a lot of frames for some reason.
The GL errors could be relevant for this.
I cannot debug this on my side as I don't have time ATM, but I'll check with the first opportunity.

Regarding the mouse, it's probably an issue on the ImGUI -> SDL interaction, where it doesn't register the mouse as a valid input device. Probably trivial to fix if you can debug the issue.

[jeanlemotan]

What Pi are you using for the GS?

[vguenther]

Rpi4 4GB with Buster

[JulesThuillier]

Hello ! I am getting the same kind of error (Video_Decoder line 161), black appears with a few control sliders in a corner. I see a frame rate updating for a few seconds then I have a sigHandler message in the terminal. Have you managed to fix it ?

[vguenther]

…. Have you managed to fix it ?

No, I haven‘t.

[jeanlemotan]

I managed to compile and setup the pi to debug this issue, I will check tomorrow in detail.