Constant "Basic output test failed" errors from Sway if WayVNC is running when SDL2 program is started

[eero-t]

Setup

Fedora 33 package versions:

• sway v1.5
• wlroots v0.11.0
• wayvnc v0.3.0

Problem

Use-case:

• Start: sway
• Start: WAYLAND_DISPLAY=wayland-1 wayvnc
• Start: WAYLAND_DISPLAY=wayland-1

Expected results:

• No errors as they don't happen when wayvnc isn't running
• Any error output is rate-limited to something sane

Actual result:

• VNC client output of Sway screen and the SDL2 application looks fine, so error doesn't seem very serious
• Following from Sway approximately at rate given by WayVNC "--max-fps" option:

00:00:03.677 [types/wlr_output.c:567] Basic output test failed
00:00:03.694 [types/wlr_output.c:567] Basic output test failed
00:00:03.711 [types/wlr_output.c:567] Basic output test failed
00:00:03.727 [types/wlr_output.c:567] Basic output test failed

Notes

This looks similar to an earlier wlroots issue, but that was fixed clearly before wlroots v0.11.0 release, so I assume Fedora version to contain that fix: #2164

I tried one another SDL2 program with this, but that didn't trigger the issue, so I assume problem to be one of the SDL options used by this program:

window = SDL_CreateWindow(name, x, y, w, h, SDL_WINDOW_FULLSCREEN|SDL_WINDOW_INPUT_FOCUS|SDL_WINDOW_ALWAYS_ON_TOP);
renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_PRESENTVSYNC|SDL_RENDERER_ACCELERATED);
SDL_CreateTexture(renderer, SDL_PIXELFORMAT_NV12, SDL_TEXTUREACCESS_STREAMING, w, h);

And such textures being blit to the screen. Using 16-bit SDL_PIXELFORMAT_UYVY instead of the 12-bit one doesn't help.

(Program is for viewing video streams, that's why it's using video related formats, and unfortunately it's closed one.)

[emersion]

This was mentioned in #2523 (comment)

Please try with the latest Sway and wlroots commits.

[emersion]

Can you try swaywm/sway#5871?

[eero-t]

I haven't built any of these components from sources. Are there API breaks since v0.11.0 wlroots release (one in Fedora) i.e. if I build latest wlroots + that extra patch, do I need to rebuild also Sway & WayVNC?

[emersion]

You need to build both wlroots and Sway from source, no need to install. See https://github.com/swaywm/sway/wiki/Development-Setup#compiling-as-a-subproject

[eero-t]

Ok, thanks. I should have some time for that next week.

[eero-t]

Problem wasn't related to the SDL2 or pixel formats, but fullscreen (as indicated by the linked items & PRs).

While fullscreen doesn't work for glmark2 Wayland version:

$ sway &  # running in a wayland window
$ wayvnc --max-fps 10 &
$ WAYLAND_DISPLAY=wayland-1 glmark2-es2-wayland --fullscreen
Error: eglCreateWindowSurface failed with error: 0x300b
Error: eglCreateWindowSurface failed with error: 0x300b
Error: CanvasGeneric: Invalid EGL state
Error: main: Could not initialize canvas

I can use Xwayland and X version for it, and this does produce the Sway/wlroots errors with WayVNC:
DISPLAY=:2 ./glmark2-es2 --fullscreen

[eero-t]

Unfortunately I don't have time to check the fix (due to app I needed this for not working 100% under Wayland and few other reasons, I've switched to Weston + Xwayland + x11vnc), but you can easily verify the fix with commands in above comment (glmark2 is available in the most Linux distros).