Lag in second frame of movie on Linux system

[CameronTEllis]

Describe the bug
The duration of time between the call of the OpenMovie and PlayMovie commands is added to the duration between frame 1 and frame 2 of the movie once PlayMovie is called. This behavior can make timing difficult to control: if you want to trigger the movie (e.g., fMRI experiments), then you want to listen for the trigger after OpenMovie, but doing that will add lag due to this bug. The alternative is to listen to the trigger before OpenMovie but that adds I/O speed lag that is unpredictable.
This didn't happen on previous Linux OS/PTB versions (circa 2019).

To Reproduce
Steps to reproduce the behavior:

1. Open SimpleMovieDemo
2. On line 44 (i.e., between the OpenMovie and PlayMovie commands), add: WaitSecs(2);
3. Run SimpleMovieDemo. There will be a 2s pause after the window is made (as expected), but then the first frame will hang for 2s before starting.
4. Setting the SetMovieTimeIndex to zero after the OpenMovie call does not fix this behavior.

Expected behavior
There shouldn't be a lag since the movie shouldn't 'begin' until the PlayMovie command.

Desktop (please complete the following information):
System showing the bug:

• OS: Ubuntu 22.04 LTS
• Runtime: Matlab 2023a
• Hardware: AMD Ryzen 7 6800U with Radeon Graphics
• Version: 3.0.19

System showing the correct behavior

• OS: MacOS 13.4
• RunTime: Matlab 2022b
• Hardware: M1
• Version: 3.0.18

Additional context
Feel free to reach out if you have any questions

[kleinerm]

Indeed, I was able to verify this problem on my Ubuntu 22.04 LTS system, following your repro steps. Thanks for the bug report.
Diagnosing the bug, finding the root cause and coming up with potential workarounds took over 2 hours of billable work time.
Therefore I'd ask you to post your paid support authentication token (help PsychPaidSupportAndServices) to pay at least for part of that work time, before I can divulge possible solutions. Otherwise this has to wait until somebody else pays for it. Thank you.

Also note that macOS on Apple Silicon Macs is not yet a currently supported platform, and stimulation timing will likely be wrong for Psychtoolbox and any other vision science toolkit, at least if frame accurate timing and time-stamping is required.

[CameronTEllis]

@kleinerm Thanks for looking into it. Here is my authentication token: VJZ7R4KY-2023912144811:98b0fafc433b89e17639ea0ccf26b802e2b30ff1497a037ebef07410adb26bd5

Let me know what else you need.

[kleinerm]

Is this from a just purchased license or paid via some slow wire transfer? So far the corresponding license key for the token is not in my database, which could mean a processing delay somewhere.

[CameronTEllis]

It was purchased 2 days ago via a credit card. Let me know if you would like me to forward you the purchase receipt.

[kleinerm]

Hmm, still not in our database. This either means the payment via credit card didn't come through, or something else went wrong. Could you please send the receipt via e-mail to ***@***.*** ?

…

On Thu, Sep 14, 2023 at 10:30 PM Cameron T Ellis ***@***.***> wrote: It was purchased 2 days ago via a credit card. Let me know if you would like me to forward you the purchase receipt. — Reply to this email directly, view it on GitHub <#814 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIXMNTH5TWIJ476LOZLEM3X2NSMVANCNFSM6AAAAAA4I4NWCM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kleinerm]

Ok, mystery solved with the help of our sales person. Apparently you entered a wrong OrderId/Order No. when creating the authentication token, creating an invalid token, therefore the entry could not be found in our database. With the proper Order No. I found the entry. For future reference. The correct OrderId would have been SLL6APY4, to be found on your invoice according to our instructions here - see yellow highlighted region: https://www.psychtoolbox.net/license-key/

So to the topic:

The reason for this lag is a bug in version 16 of the Pulseaudio sound server which ships with Ubuntu 22.04-LTS up to at least upcoming Ubuntu 23.10 (Pulseaudio bug report at https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/issues/1383 GStreamer bug report at https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1735).

Whenever Pulseaudio is used as default audio backend for sound output during movie playback via Psychtoolbox GStreamer movie playback engine on Linux, that Pulseaudio bug will screw up timing in the way you described.

The bug has been properly fixed for the upcoming Pulseaudio version 17 (https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/commit/8fe50bbc31e11abf2f30864f1e2dbdaa16d0e1c3), but version 17 is not yet released, and therefore this bug is in all Ubuntu versions since 22.04 LTS. Hopefully the bug fix will end in a future Ubuntu 24.04-LTS release next year.

Therefore we are left with workarounds for this issue:

• Ubuntu 20.04.6 LTS has an older Pulseaudio, so would work. Downgrading is generally not recommended though,as we will cancel support for 20.04 in the foreseeable future with future PTB releases. We don't have the funding to keep multiple major Ubuntu versions properly tested and supported, so very little testing with Ubuntu 20.04 happens anymore.

• If you don't need sound during movie playback, you can call Screen('OpenMovie', ..., specialflags1); with a specialflags1 setting of 2 to disable any sound playback from the movie -> No sound -> No Pulseaudio -> No problem.

• If you do need to play back sound from your movie files, then we need to direct PTB to use a different sound backend than the default direct Pulseaudio backend: For that, the optional movieoptions parameter in Screen('OpenMovie', ..., movieoptions); needs to be set to choose a different sound backend or parameters:

  • Setting movieoptions as the string 'AudioSink=alsasink' will reroute sound through the ALSA backend, which itself by default will use Pulseaudio for sound output, but this indirection will side-step the problem. This works, but has somewhat increased audio latency wrt. the visual stimulus.
  • Setting movieoptions as the string 'AudioSink=alsasink device=hw:0,0' would also reroute sound through the ALSA backend, but force it to choose an audio hardware device directly, in this example 'hw:0,0', which provides lower latency. The specific number depends on your specific installed sound card though, e.g., it could also be 'hw:0,1' or 'hw:2,0' or such. A list of devices can be shown via d = PsychPortAudio('GetDevices', 8); PsychPortAudio('Close'); -> d.DeviceName lists the available devices, some will have a "hw:..." in their name.

• Theoretically you could also download the latest pulseaudio builds and install them manually, or you could switch to the pipewire sound server on Ubuntu 22.04 LTS following howto's on the internet. Or maybe the latest Ubuntu 23.04, which uses pipewire instead of pulseaudio as sound server, will not have this bug. UPDATE: Pipewire on Ubuntu 22.04 - 23.10 suffers the same bug, most likely inherited from Pulseaudio (cfe. https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/3399), so this is not an option. But this manual upgrade to pre-release pulseaudio 17 builds is an untested option by myself and not supportable by me right now.

[kleinerm]

As of 4th March 2024, the bug has been fixed in Pulseaudio version 17, but so far this is not a part of current Debian or the upcoming Ubuntu 24.04-LTS. I've pinged Ubuntu about still upgrading to Pulseaudio 17 for upcoming LTS:
https://answers.launchpad.net/ubuntu/+source/pulseaudio/+question/709476

And we got the response that it is too late to include a modern Pulseaudio 17 package which would solve this and other problems. Instead it was recommended to me to file an important bug report, just asking for backport of that one bug fix for that one bug that affects us here, which I did:

https://bugs.launchpad.net/pipewire/+bug/2056079
https://bugs.launchpad.net/ubuntu/+source/pulseaudio/+bug/2057684

Fingers crossed that this will still make it into the upcoming April 2024 release of Ubuntu 24.04-LTS.