Vertical videos are rotated horizontally

[dronnikovigor]

Describe the bug
Vertical videos are rotated horizontally during playback

To Reproduce
Steps to reproduce the behavior:

1. Open Memories
2. Open any vertical video in History / Folders
3. Start it
4. Video will be rotated to horizontal

Screenshots

Platform:

• OS: MacOS, Windows, Android, iOS
• Browser: Chrome
• Memories Version: 5.4.1
• Nextcloud Version: 27.0.2 AIO
• PHP Version: 8.1.22

Additional context

• No errors in the JS console
• No errors in the Nextcloud server logs

[pulsejet]

1. Are you using transcoding?
2. Hardware acceleration?
3. Did you turn on the "transpose workaround" in the admin panel of memories?

[dronnikovigor]

Thanks, after enabling transpose workaround it was fixed.

[ChildLearningClub]

When Transcoding alone is enable things work fine.

When Transcoding and VA-API HW Acceleration I'm seeing this issue of vertical videos being played back horizontally.

When Transcoding, VA-API HW Acceleration, and "transpose workaround" enabled: I get (Transcode failed: Transcoder returned 408) and the video does not play.

[pulsejet]

@ChildLearningClub

1. There should be further logs in /tmp/go-vod/.... Can you find these?
2. What hardware is this?
3. Are you using an external transcoder? Possibly your ffmpeg could be too old.

[ChildLearningClub]

It does say in the logs that VA-API doesn’t support transpose, it’s running on an older laptop with i7-4800mq cpu.

[pulsejet]

It does say in the logs that VA-API doesn’t support transpose, it’s running on an older laptop with i7-4800mq cpu.

Ah, that's unfortunate. Hopefully FFmpeg fixes the broken autorotation in VA-API in the future.

[Floppik]

When Transcoding alone is enable things work fine.

When Transcoding and VA-API HW Acceleration I'm seeing this issue of vertical videos being played back horizontally.

When Transcoding, VA-API HW Acceleration, and "transpose workaround" enabled: I get (Transcode failed: Transcoder returned 408) and the video does not play.

Same issue. Try build ffmpeg from source, but when enable "transpose workaround" - got Transcode failed: Transcoder returned 408. If "transpose workaround" disable - vertical video rotate on 270 degrees

/usr/src/ffmpeg-build-script/workspace/bin/ffmpeg -encoders |  grep -i 265
ffmpeg version 6.0.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-nonfree --enable-gpl --enable-openssl --enable-libdav1d --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libvpx --enable-libxvid --enable-libvidstab --enable-libaom --enable-libzimg --enable-lv2 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libtheora --enable-libfdk-aac --enable-libwebp --enable-libfreetype --enable-libsrt --disable-ffnvcodec --enable-amf --disable-debug --disable-shared --enable-pthreads --enable-static --enable-version3 --extra-cflags='-I/usr/src/ffmpeg-build-script/workspace/include -I/usr/src/ffmpeg-build-script/workspace/include/lilv-0' --extra-ldexeflags= --extra-ldflags=-L/usr/src/ffmpeg-build-script/workspace/lib --extra-libs='-ldl -lpthread -lm -lz' --pkgconfigdir=/usr/src/ffmpeg-build-script/workspace/lib/pkgconfig --pkg-config-flags=--static --prefix=/usr/src/ffmpeg-build-script/workspace --extra-version=
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
 V....D libx265              libx265 H.265 / HEVC (codec hevc)
 V....D hevc_vaapi           H.265/HEVC (VAAPI) (codec hevc)

/usr/src/ffmpeg-build-script/workspace/bin/ffmpeg -encoders |  grep -i 264
ffmpeg version 6.0.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-nonfree --enable-gpl --enable-openssl --enable-libdav1d --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libvpx --enable-libxvid --enable-libvidstab --enable-libaom --enable-libzimg --enable-lv2 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libtheora --enable-libfdk-aac --enable-libwebp --enable-libfreetype --enable-libsrt --disable-ffnvcodec --enable-amf --disable-debug --disable-shared --enable-pthreads --enable-static --enable-version3 --extra-cflags='-I/usr/src/ffmpeg-build-script/workspace/include -I/usr/src/ffmpeg-build-script/workspace/include/lilv-0' --extra-ldexeflags= --extra-ldflags=-L/usr/src/ffmpeg-build-script/workspace/lib --extra-libs='-ldl -lpthread -lm -lz' --pkgconfigdir=/usr/src/ffmpeg-build-script/workspace/lib/pkgconfig --pkg-config-flags=--static --prefix=/usr/src/ffmpeg-build-script/workspace --extra-version=
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
 V....D libx264              libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (codec h264)
 V....D libx264rgb           libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 RGB (codec h264)
 V....D h264_amf             AMD AMF H.264 Encoder (codec h264)
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 encoder wrapper (codec h264)
 V....D h264_vaapi           H.264/AVC (VAAPI) (codec h264)

[pulsejet]

Same issue.

Just to reiterate, there's nothing we can do about this. The VA-API encoder doesn't autorotate the videos and not all devices support rotating the videos in hardware (newer Intel processors would work, AMD never works). In other words, this needs to be fixed either in the VA-API drivers or by ffmpeg.

[dronnikovigor]

Hi @pulsejet !
I'm facing the same issue with 'VA-API driver doesn’t support transpose' on my new Ryzen 5600g. I've read above that it's issue of ffmpeg / vaapi driver. Ok, question isn't about this. But I just wanted to point, that in Immich / PhotoPrism transcoding of vertical videos works also with use of ffmpeg & vaapi and videos are shown correctly. I'm not sure how everything works in Memories, but maybe it's possible to rotate videos basing on video's metadata on frontend or with some software postprocessing?

[pulsejet]

But I just wanted to point, that in Immich / PhotoPrism transcoding of vertical videos works also with use of ffmpeg & vaapi and videos are shown correctly

AFAIK neither of these support adaptive streaming. The problem in case is that players (e.g. browsers) don't honor the rotation metadata in videos when using adaptive streaming, which requires the video to be correctly oriented to begin with.

I'm not sure how everything works in Memories, but maybe it's possible to rotate videos basing on video's metadata on frontend

This is actually what we used to do earlier. Unfortunately it's very prone to weird failures and doesn't work on the native app.

The next version will have an option to force the transpose to happen in software; this will be slower and I can't guarantee it'll work, but it's worth a shot. At least on my (newer) Intel, it allows rotating correctly without using transpose_vaapi at all.

[pulsejet]

Please try this option with v7.3 and report back