Vulkan based linux port #604

xytovl · 2021-04-16T07:04:44Z

This version replaces the "screen grab" version.

The mechanism is based on frame capture in vrcompositor process, which
are then shared with vrserver process, encoded there and transmitted to
the headset.
A vulkan layer is added to vrcompositor, this layer exposes the
VK_EXT_direct_mode_display, VK_EXT_acquire_xlib_display and some other
related vulkan functions. Functions implemented in the layer add one
display to the list naturally returned by the driver, this display
matches properties in ALVR settings (resolution and refresh rate), and
all functions required by SteamVR are implemented (xlib acquire,
properties enumeration, vsync event, swapchain creation).
The layer then connects to a socket owned by the server, submits the
VkImage creation parameters, and transfers file descriptors
corresponding to the images in the swapchain and associated semaphores.
The, on each frame, a packet is sent on the socket describing the image
that has been submitted.
On server side, vulkan frames are mapped to vaapi surfaces, using
ffmpeg, then converted to a format suitable for encoding, encoded
through vaapi and sent with the usual mechanism.
In order to associate each frame with tracking information, timing
details are requested to the compositor, and tracking history is queried
to find the most suitable tracking index. This mechanism is still
inaccurate and will need significant upgrades.

Co-authored-by: ckie git-525ff67@ckie.dev

Closes #269.

This version replaces the "screen grab" version. The mechanism is based on frame capture in vrcompositor process, which are then shared with vrserver process, encoded there and transmitted to the headset. A vulkan layer is added to vrcompositor, this layer exposes the VK_EXT_direct_mode_display, VK_EXT_acquire_xlib_display and some other related vulkan functions. Functions implemented in the layer add one display to the list naturally returned by the driver, this display matches properties in ALVR settings (resolution and refresh rate), and all functions required by SteamVR are implemented (xlib acquire, properties enumeration, vsync event, swapchain creation). The layer then connects to a socket owned by the server, submits the VkImage creation parameters, and transfers file descriptors corresponding to the images in the swapchain and associated semaphores. The, on each frame, a packet is sent on the socket describing the image that has been submitted. On server side, vulkan frames are mapped to vaapi surfaces, using ffmpeg, then converted to a format suitable for encoding, encoded through vaapi and sent with the usual mechanism. In order to associate each frame with tracking information, timing details are requested to the compositor, and tracking history is queried to find the most suitable tracking index. This mechanism is still inaccurate and will need significant upgrades. Co-authored-by: Ron B <me@ronthecookie.me>

decipher2k · 2021-04-16T07:26:16Z

Following error occures while building:

ALVR-linux-vulkan-port/alvr/server/cpp/platform/linux/ffmpeg_helper.cpp:126: undefined reference to `av_vk_frame_alloc'
          collect2: error: ld returned 1 exit status

xytovl · 2021-04-16T07:40:37Z

Following error occures while building:

ALVR-linux-vulkan-port/alvr/server/cpp/platform/linux/ffmpeg_helper.cpp:126: undefined reference to `av_vk_frame_alloc'
          collect2: error: ld returned 1 exit status

This means your ffmpeg does not include vulkan support.

Build instructions update is in progress...

decipher2k · 2021-04-16T07:53:16Z

Following error occures while building:
ALVR-linux-vulkan-port/alvr/server/cpp/platform/linux/ffmpeg_helper.cpp:126: undefined reference to `av_vk_frame_alloc'
          collect2: error: ld returned 1 exit status
This means your ffmpeg does not include vulkan support.

Build instructions update is in progress...

thanks a lot!

Do not build ffmpeg module inside alvr_common

decipher2k · 2021-04-18T06:06:36Z

it builds the server, and the client builds using android studio.
when connecting, a timeout appears in the client app, and the server throws the following error:

[ERROR] At alvr/common/src/sockets/stream_socket/tcp.rs:46: Connection refused (os error 111)"}],"executionContextId":1,"stackTrace":{"callFrames":[{"columnNumber":20,"functionName":"addLogLine","lineNumber":344,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"},{"columnNumber":16,"functionName":"","lineNumber":48,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"}]},"timestamp":1618725597456.958,"type":"log"}}
Message: {"method":"Runtime.consoleAPICalled","params":{"args":[{"type":"string","value":"07:59:57.557406474 [WARN] Handshake: At alvr/common/src/sockets/control_socket.rs:213: Transport endpoint is not connected (os error 107)"}],"executionContextId":1,"stackTrace":{"callFrames":[{"columnNumber":20,"functionName":"addLogLine","lineNumber":344,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"},{"columnNumber":16,"functionName":"","lineNumber":48,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"}]},"timestamp":1618725597557.7542,"type":"log"}}

i tried different protocols (tcp/trhtld udp/udp) but without success.
when changing the port to a low port, the client throws a permission denied error.
running setcap 'cap_net_bind_service=+ep' ./alvr_server was without success.

EDIT: reason was that i have disabled audio streaming

decipher2k · 2021-04-18T15:09:54Z

after a dozen audio errors, i can get it to this point using jack and pulseaudio.
then, a threading error in gtk appears:

Message: {"method":"Runtime.consoleAPICalled","params":{"args":[{"type":"string","value":"17:07:44.235454744 [ERROR] Message: \"Attempted to initialize GTK from two different threads.\"\nBacktrace:\n 0: alvr_common::logging::set_panic_hook::{{closure}}\n 1: std::panicking::rust_panic_with_hook\n at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/std/src/panicking.rs:595:17\n 2: std::panicking::begin_panic::{{closure}}\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n 4: std::panicking::begin_panic\n 5: gtk::rt::init\n 6: msgbox::linux::create\n 7: std::sys_common::backtrace::__rust_begin_short_backtrace\n 8: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/alloc/src/boxed.rs:1521:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/alloc/src/boxed.rs:1521:9\n std::sys::unix::thread::Thread::new::thread_start\n at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/std/src/sys/unix/thread.rs:71:17\n 10: start_thread\n 11: clone\n"}],"executionContextId":1,"stackTrace":{"callFrames":[{"columnNumber":20,"functionName":"addLogLine","lineNumber":344,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"},{"columnNumber":16,"functionName":"","lineNumber":48,"scriptId":"15","url":"http://127.0.0.1:8082/js/app/monitor.js"}]},"timestamp":1618758471280.811,"type":"log"}}

In order to get the correct pose associated to an image, search the stack for the variable, then copy it and send to the server.

This will allow the user to not have to modify `vrenv.sh` which is quite error-prone.

decipher2k · 2021-04-19T04:52:05Z

the launcher is looking for the openvr paths config at ~/.cache
on some systems, the file is at ~/.config

xytovl · 2021-04-19T06:08:18Z

the launcher is looking for the openvr paths config at ~/.cache
on some systems, the file is at ~/.config

this has been changed long ago, it is in alvr/common/src/commands.rs

    #[cfg(windows)]
    let path = trace_none!(dirs::cache_dir())?.join("openvr/openvrpaths.vrpath");
    #[cfg(target_os = "linux")]
    let path = trace_none!(dirs::config_dir())?.join("openvr/openvrpaths.vrpath");

Where do you get the issue?

decipher2k · 2021-04-19T06:12:32Z

the launcher is looking for the openvr paths config at ~/.cache
on some systems, the file is at ~/.config

this has been changed long ago, it is in alvr/common/src/commands.rs
    #[cfg(windows)]
    let path = trace_none!(dirs::cache_dir())?.join("openvr/openvrpaths.vrpath");
    #[cfg(target_os = "linux")]
    let path = trace_none!(dirs::config_dir())?.join("openvr/openvrpaths.vrpath");
Where do you get the issue?

it's in /alvr/launcher/src/commands.rs

another problem i found in the launcher is that it removes openvrpaths when clicking "reset drivers", then it throws an error about not being able to load it ^^
when restarting steamvr manually, this file gets recreated.

also, the XDG path environment variable of wrapper.sh points to some weird directory on my system, whereas the .txt file it searches for is in /tmp/

decipher2k · 2021-04-19T06:19:01Z

sorry for bothering you again,
currently the client throws an error about a connection not being establishable (error 111).
taking a look at the sources, it seems to be an error when the server establishes a connection to the client.
(line 46 in alvr/common/src/sockets/stream_socket/tcp.rs)

it occures on the self-compiled apk of this release, as well as on the main release of alvr.
firewall is off. apk version is that of the quest 1

trying to change from tcp to udp results in the apk crashing on the quest.

This reverts commit 3cff822.

ckiee · 2021-04-20T18:09:40Z

trying to change from tcp to udp results in the apk crashing on the quest.

@dennisheine You have to turn off audio streaming as that part isn't implemented.

FeckingPotato · 2021-04-21T13:31:17Z

For some reason, neither the debug nor the release version of the driver doesn't want to load. SteamVR web console shows this error:
Unable to load driver shared library alvr_server from /home/potato/sources/ALVR/build/alvr_server_linux/bin/linux64/driver_alvr_server.so. Error Code (1) ASSERT: "DriverManager alvr_server load error 1" at /data/src/common/vrcommon/drivermanager.cpp:404. Unable to load driver alvr_server because of error VRInitError_Init_FileNotFound(103). Skipping. Unable to load driver shared library alvr_server from /home/potato/sources/ALVR/build/alvr_server_linux/bin/linux64/driver_alvr_server.so. Error Code (1) ASSERT: "DriverManager alvr_server load error 1" at /data/src/common/vrcommon/drivermanager.cpp:404. Unable to load driver alvr_server because of error VRInitError_Init_FileNotFound(103). Skipping.

Meanwhile, the launcher floods the console with that:
DEBUG [ureq::stream] connecting to 127.0.0.1:8082 at 127.0.0.1:8082

Kierek · 2021-04-21T13:34:51Z

@FeckingPotato
What distro? On Arch I have to use steam-native instead of steam-runtime. Just launching steam before ALVR works fine.

FeckingPotato · 2021-04-21T13:49:48Z

@Kierek

Thank you, switching from steam-runtime to steam-native worked. But now SteamVR complains about a key component not working properly (Error code 307).

Edit 3: for some reason, I had the AMD driver installed alongside with the NVIDIA driver. Removing the AMD driver fixed the problem.

stableversion · 2021-04-21T15:37:42Z

Edit 3: for some reason, I had the AMD driver installed alongside with the NVIDIA driver. Removing the AMD driver fixed the problem.

This woks with an Nvidia GPU??

decipher2k · 2021-04-21T15:47:55Z

Edit 3: for some reason, I had the AMD driver installed alongside with the NVIDIA driver. Removing the AMD driver fixed the problem.

This woks with an Nvidia GPU??

it is based on vulkan, so it should work with nvidia.
https://developer.nvidia.com/vulkan-driver

this project is the last thing i am missing on linux. no need for windows anymore.

xytovl · 2021-04-21T16:02:50Z

Edit 3: for some reason, I had the AMD driver installed alongside with the NVIDIA driver. Removing the AMD driver fixed the problem.

This woks with an Nvidia GPU??

it is based on vulkan, so it should work with nvidia.
https://developer.nvidia.com/vulkan-driver

this project is the last thing i am missing on linux. no need for windows anymore.

It goes one step further for nvidia, the video stream is now captured by the server, but hardware encoding only supports vaapi (AMD/Intel) for the moment.

decipher2k · 2021-04-21T16:04:24Z

Edit 3: for some reason, I had the AMD driver installed alongside with the NVIDIA driver. Removing the AMD driver fixed the problem.

This woks with an Nvidia GPU??

it is based on vulkan, so it should work with nvidia.
https://developer.nvidia.com/vulkan-driver
this project is the last thing i am missing on linux. no need for windows anymore.

It goes one step further for nvidia, the video stream is now captured by the server, but hardware encoding only supports vaapi (AMD/Intel) for the moment.

oh, maybe that's why i couldn't get it running

wangling12 · 2021-04-22T02:10:55Z

It goes one step further for nvidia, the video stream is now captured by the server, but hardware encoding only supports vaapi (AMD/Intel) for the moment.

Maybe we can try https://github.com/freedesktop/vdpau-driver (a VDPAU-based backend for VA-API) to support vdpau?

xytovl · 2021-04-22T06:07:02Z

It goes one step further for nvidia, the video stream is now captured by the server, but hardware encoding only supports vaapi (AMD/Intel) for the moment.

Maybe we can try https://github.com/freedesktop/vdpau-driver (a VDPAU-based backend for VA-API) to support vdpau?

vdpau is a decode only api, the encoding one is nvenc, but as I don't have the hardware to test it someone else has to develop the feature.

There are also the new provisional vulkan extensions which provide hardware h264 encoding, I don't know the timeline for their finalization, but that would be a much better solution as it will certainly be implemented by all vendors in the end.

wangling12 · 2021-04-22T09:58:04Z

There are also the new provisional vulkan extensions which provide hardware h264 encoding, I don't know the timeline for their finalization, but that would be a much better solution as it will certainly be implemented by all vendors in the end.

It seems that nvidia's bate driver already supports this new feature. So maybe we can use vulkan encoding on nvidia. When mesa supports this new extension, we can give up vaapi and use vulkan entirely.

xytovl · 2021-04-22T10:10:06Z

Sure, but vulkan encoding is a provisional extension, so it is only enabled on beta drivers and this will be the case until the extension is finalized.
This extension currently supports h264 only.

I do not have the hardware to test the feature, you can submit patches but I don't think this is an ideal solution for short-term, as khronos discourages shipping products that rely on provisional extensions.

We therefore request that driver vendors not ship production drivers supporting these provisional extensions, and that ISVs not use these provisional extensions in their production applications

zarik5 · 2021-04-22T10:59:56Z

Thanks @xytovl and @ckiee for working on this ❤

wangling12 · 2021-04-22T15:07:23Z

Thank you for your working ❤️

I do not have the hardware to test the feature, you can submit patches but I don't think this is an ideal solution for short-term, as khronos discourages shipping products that rely on provisional extensions.

The code is too hard for me, I just can't wait to use this on linux. 🤣

leo60228 · 2021-04-27T16:36:02Z

Is there a blocker for implementing NVENC support through FFmpeg? If I'm reading the ALVR code and FFmpeg docs correctly, it should just be a matter of changing VAAPI to CUDA in a few places (presumably behind a configuration option).

xytovl · 2021-04-27T16:48:21Z

Is there a blocker for implementing NVENC support through FFmpeg? If I'm reading the ALVR code and FFmpeg docs correctly, it should just be a matter of changing VAAPI to CUDA in a few places (presumably behind a configuration option).

Not really, there are a few differences, you can't map frames between vulkan and cuda and apparently color conversion doesn't work the same, you have to transfer them instead.
The main reason it is not implemented is that I don't have the hardware to test it myself.

amvidalrc · 2021-05-11T20:49:14Z

Hello and very thanks for this linux feature. Now, if this works, I will leave Windows for a long time.

I have Arch, compiled succesfully. Using steam-native. When I open alvr, it detect the headset (With the latest APK 15.2.1) and I can see on the headset "Streaming will start shortly, please wait". I can see packages being sent to the Quest, but nothing more... I can't see the room...

Is there any APK or Special configuration on the server? I have 60Hz h264... Thanks!

--- Update

I have installed the latest NIGHTLY APK and now I have video!! 👍 Im very happy with this incredible step on linux VR gaming...
I have no sound. I have tried to enable a loopback device "sudo modprobe snd_aloop" and set the audio device on "pavucontrol" for play the games but I can't listen nothing.

My card is an Nvidia RTX2070 Super with 465.27 and I have a good quality (h264, 60hz) but is not so smooth... I think because the VAAPI not supported. I will check new changes asap you publish.

Thanks again.

ckiee · 2021-05-12T15:40:44Z

Hi @amvidalrc!

I have no sound.

Audio streaming hasn't been implemented yet and is turned off by default to avoid the missing codepath.

but is not so smooth... I think because the VAAPI not supported.

ALVR does not currently use hardware encoding on nVidia as CUDA is annoying to work with and no one has put the effort into it yet. (it uses software encoding on the CPU instead)

Also, for the future, we develop ALVR on the discord server and that is where we usually discuss this kind of stuff. (you will find more activity there, but personally I am not against using GitHub too)

xytovl added 3 commits April 16, 2021 23:47

include pose age in timing calculation

fcd3d78

linux: correctly set timestamps for bitrate

4ac4c3b

fix uninitialized variable, layer metadata

a11c744

xytovl force-pushed the linux-vulkan-port branch from e9b5af8 to a11c744 Compare April 17, 2021 12:07

xytovl and others added 3 commits April 17, 2021 16:02

fix warning

92ee62a

CI: fix missing vulkan headers

c3b2091

CI: Do not build alvr_server for linux

0ebf9f6

Do not build ffmpeg module inside alvr_common

use stack inspection in layer to find pose

65988a4

In order to get the correct pose associated to an image, search the stack for the variable, then copy it and send to the server.

ckiee mentioned this pull request Apr 18, 2021

Linux port: vulkan layer init & intergration #569

Closed

Ron B added 3 commits April 18, 2021 22:35

Automatically wrap vrcompositor-launcher

5c0309f

This will allow the user to not have to modify `vrenv.sh` which is quite error-prone.

update vulkan-layer shell.nix

d5f96da

run cargo fmt (oops)

3e2e880

Ron B and others added 4 commits April 19, 2021 19:28

add missing <sstream> include

dcb8b8b

Update CI

3cff822

Revert "Update CI"

8d0b58a

This reverts commit 3cff822.

make the compiler happy (unused Result)

6d8f134

zarik5 merged commit 64288f4 into alvr-org:master Apr 22, 2021

xytovl deleted the linux-vulkan-port branch April 27, 2021 16:53

ckiee mentioned this pull request May 17, 2021

VR driver rewrite / Linux support #269

Closed

ckiee mentioned this pull request Jul 18, 2021

ALVR Server not loading in ubuntu #669

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vulkan based linux port #604

Vulkan based linux port #604

xytovl commented Apr 16, 2021 •

edited by ckiee

decipher2k commented Apr 16, 2021

xytovl commented Apr 16, 2021

decipher2k commented Apr 16, 2021

decipher2k commented Apr 18, 2021 •

edited

decipher2k commented Apr 18, 2021

decipher2k commented Apr 19, 2021

xytovl commented Apr 19, 2021

decipher2k commented Apr 19, 2021 •

edited

decipher2k commented Apr 19, 2021 •

edited

ckiee commented Apr 20, 2021

FeckingPotato commented Apr 21, 2021

Kierek commented Apr 21, 2021

FeckingPotato commented Apr 21, 2021 •

edited

stableversion commented Apr 21, 2021

decipher2k commented Apr 21, 2021

xytovl commented Apr 21, 2021

decipher2k commented Apr 21, 2021

wangling12 commented Apr 22, 2021

xytovl commented Apr 22, 2021

wangling12 commented Apr 22, 2021

xytovl commented Apr 22, 2021

zarik5 commented Apr 22, 2021 •

edited by ckiee

wangling12 commented Apr 22, 2021

leo60228 commented Apr 27, 2021

xytovl commented Apr 27, 2021 •

edited

amvidalrc commented May 11, 2021 •

edited

ckiee commented May 12, 2021

Vulkan based linux port #604

Vulkan based linux port #604

Conversation

xytovl commented Apr 16, 2021 • edited by ckiee

decipher2k commented Apr 16, 2021

xytovl commented Apr 16, 2021

decipher2k commented Apr 16, 2021

decipher2k commented Apr 18, 2021 • edited

decipher2k commented Apr 18, 2021

decipher2k commented Apr 19, 2021

xytovl commented Apr 19, 2021

decipher2k commented Apr 19, 2021 • edited

decipher2k commented Apr 19, 2021 • edited

ckiee commented Apr 20, 2021

FeckingPotato commented Apr 21, 2021

Kierek commented Apr 21, 2021

FeckingPotato commented Apr 21, 2021 • edited

stableversion commented Apr 21, 2021

decipher2k commented Apr 21, 2021

xytovl commented Apr 21, 2021

decipher2k commented Apr 21, 2021

wangling12 commented Apr 22, 2021

xytovl commented Apr 22, 2021

wangling12 commented Apr 22, 2021

xytovl commented Apr 22, 2021

zarik5 commented Apr 22, 2021 • edited by ckiee

wangling12 commented Apr 22, 2021

leo60228 commented Apr 27, 2021

xytovl commented Apr 27, 2021 • edited

amvidalrc commented May 11, 2021 • edited

ckiee commented May 12, 2021

xytovl commented Apr 16, 2021 •

edited by ckiee

decipher2k commented Apr 18, 2021 •

edited

decipher2k commented Apr 19, 2021 •

edited

decipher2k commented Apr 19, 2021 •

edited

FeckingPotato commented Apr 21, 2021 •

edited

zarik5 commented Apr 22, 2021 •

edited by ckiee

xytovl commented Apr 27, 2021 •

edited

amvidalrc commented May 11, 2021 •

edited