Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
PsychOpenXR[Core]: Add Monado "metrics" timestamping hack.
Monado has a built-in system for logging presentation "metrics" info
to a file. During each XR compositor work cycle it can write some
data records about which OpenXR client submitted a new frame for
future presentation at a specific target time, when such a frame
gets latched into the compositor for its next XR display cycle,
when the composed frame gets submitted for pageflip onto the XR
display devices associated video output, and when the flip for a
present completed, iow. stimulus onset. It uses the Google protobuffer
protocol, via nanopb, to write such logs on user request.
We now optionally (ab)use this as a side-feedback channel to
retrieve info that allows us to reconstruct which XR frames were
displayed (vs. dropped), and when actual image onset happened vs.
when it was requested, ie. tOnset vs. tWhen in a typical Flip...
[tvbl, tOnset] = Screen('Flip', win, tWhen);
... invocation.
How does this work, if enabled?
For setup the user has to:
1. Install an enhanced version of current Monado git main branch
with a patch from myself, to fflush(g_file) the metrics output
file after each write of a metrics record. Otherwise default
stream buffering will increase latency between 'Flip' (aka
xrEndFrame()) and reported flip completion and performance will
go down the gutter.
2. Install an enhanced Mesa Vulkan driver version with google
timestamping protocol support. [Not in official Mesa upstream yet].
3. Choose a gpu with good open-source Mesa Vulkan drivers and enough
performance. Iow. a discrete AMD gpu, maybe one of the new discrete
Intel Arc gpu's?
4. Create a fifo pipe as file, e.g.,
"sudo mkfifo /usr/local/framequeue.protobuf"
or similar.
For use the user has to:
1. Start monado-service and use the created fifo file as output file
for the metrics log, e.g., via
"XRT_METRICS_FILE=/usr/local/framequeue.protobuf monado-service"
This will launch the monado-service OpenXR compositor, enable its
metrics logging into the fifo, and block its startup until a reader
opens the fifo file as well.
2. Start a PTB session, also with XRT_METRICS_FILE environment variable
specified to the same fifo file location, e.g.,
"XRT_METRICS_FILE=/usr/local/framequeue.protobuf octave" or
"XRT_METRICS_FILE=/usr/local/framequeue.protobuf matlab"
Once the PsychOpenXR driver has detected that a Monado XR server is
running, that the fifo file exists and is accessible, it opens that
fifo for read access, which will let monado-service fully start up
and get ready to serve OpenXR clients.
3. During runtime, after each 'Flip' aka xrEndFrame() our PsychOpenXRCore
then reads+parses+processes all the info that Monado sends as metrics
data, using nanopb as parser, and from that learns about and uses the
frame timestamps, returning them as [tVbl, tOnset] = Screen('Flip',..)
timestamps with tVbl == tOnset = Zero for dropped frames, and greater
than zero for visual stimulus onset timestamps.
Assumption is currently that stimulus onset = flip timestamp + 4 msecs,
as current Monado hard-coded the flip-completion/start-of-active-scanout
to photon emission latency as 4 msecs. A particular value that does not
make too much sense to me right now, but maybe matched the behaviour of
the HMD originally used for Monado development? 4 msecs would be about
right for mid-display onset on a 120 Hz HMD with no shutter or rolling
shutter and no pixel latency. A better assumption could be to add one
active scanout duration ~ 0.95 video refresh cycle durations, assuming
a zero-latency instant OLED panel or LCD panel with flashing backlight
and global shutter at end of scanout.
This timestamping approach is obviously not a clean and nice solution, but
it is the only currently existing solution if one wants trustworthy, robust,
precise stimulus onset timestamps on current operating systems and HMDs, as
afaik all existing OpenXR implementations, including extensions, do not
support proper timestamping.
It has various downsides:
1. Linux + hacked Monado + hacked Mesa + AMD or Intel dGPU only atm.
2. Fiddly setup and operation.
3. Not future-proof. If Monado changes its metrics implementation, things
may break.
4. No way forward to a proper OpenXR solution.
5. The current code adds substantial latency to each Flip, cutting possible
framerate down. Atm. I achieve at most 1/3 rd video refresh rate, e.g.,
30 fps on a 90 Hz Oculus Rift CV-1 HMD. However, my AMD iGPU is too weak
to drive the Rift properly, so numbers might improve with an appropriately
performant gpu for the task.
6. The current code needs multi-threaded presentation mode, even on Monado,
which normally can do without it, thereby adding overhead which may
contribute to poor performance. This because the Monado server constantly
streams data into a fifo of fixed capacity, and the fifo must never fill
up completely or Monado will lock-up and then PTB will lock-up and we have
a beautiful deadlock situation. So the thread must periodically (each
compositor work cycle, e.g., 90 times/second on a 90 Hz HMD) submit new
frames and pulll data out of the fifo.
It is possible to run single-threaded for a performance boost, but then
the user script must carefully manage timing to never let the fifo run
full. Current max fifo capacity is 1 Megabyte, with about 25 kb/s streaming,
so any pause between Flips of more than ~40 seconds will end in deadlock!
=> This still needs tweaking and actual testing against hw reference for
timestamping, but is the best one can do with the current OpenXR spec,
Monado, etc.
=> Not great, but better than nothing.
=> A proper OpenXR extension will need to be drafted, prototyped and implemented
at least for Monado. This is TODO for future work.- Loading branch information
