ffmpeg: segfault on x86 CPUs with less instruction features.

[DazzyWalkman]

Maintainer: @thess
Environment: x86_64,, OpenWrt snapshot r9874

Description:
The following code:
ffprobe ./test.avi
will generate a segfault:
kern.info kernel: [49036.983202] ffprobe[28883]: segfault at 7f693b4de320 ip 00007f693a891317 sp 00007ffcc5547330 error 7 in libavcodec.so.57.107.100[7f693a870000+8cb000]
in syslog and program aborts.

It appears since
295f6b3
ffmpeg: update to 3.4.5

Previous version 3.2.12 works flawlessly.

BUILD_PATENTED true or false does not matter, and whatever codecs of test.avi do not seem to matter either.

Tests I have run so far reveal that It may be related to the CPU type on which OpenWRT is running, particularly to SSE4 support.

Works on Sandybridge, Hasswell, Segfault on Atom D525 or kvm64.

Real world impacts for me:

1. The requirement is not inline with OpenWRT mininmum CPU requirement: Pentium MMX, I cannot run ffmpeg 3.4.5 or later on older CPUs like Atom D525.
2. ffmpeg 3.4.5 or later can't be run on kvm guest featuring generic kvm64 CPU. When such guest running on top of clusters with various CPUs, it's not feasible to set CPU to anything but kvm64.

I don't know if it's a new requirement imposed by newer versions of ffmpeg, or there are related build flags.

Modifying Makefile to disable asm or any instructions newer than ssse3 does not work.
And in fact ffmpeg already enables 'runtime-cpudetection' as default.

At present I stick to ffmpeg 3.2.12 on the affected platform.

Update:
I tried compiling a build with --CPU=nocona, as suggested by @diizzyy. The CPU flag is quite close to D525. Still segfault.

Compiled on a Ubuntu 18.10 Virtualbox guest with CPU as host: Intel(R) Pentium(R) CPU G640 (Sandy Bridge).
Test run on Virtualbox OpenWRT guest with CPU as host: G640 : Works.
Test run on QEMU OpenWRT guest with CPU as host: Intel(R) Celeron(R) CPU J1900 (Bay Trail): Segfault.

Version String:
ffprobe version 3.4.6 Copyright (c) 2007-2019 the FFmpeg developers
built with gcc 8.3.0 (OpenWrt GCC 8.3.0 r9482-b2bf3745ff)
configuration: --enable-cross-compile --cross-prefix=x86_64-openwrt-linux-musl- --arch=x86_64 --cpu=nocona --target-os=linux --prefix=/usr --pkg-config=pkg-config --enable-shared --enable-static --enable-pthreads --enable-zlib --disable-doc --disable-debug --disable-lzma --disable-vaapi --disable-vdpau --disable-outdevs --enable-lto --enable-x86asm --enable-avresample --enable-hardcoded-tables --enable-libmp3lame
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100

I compared flags of G640 and J1900, not so much difference. It's not an SSE4 issue as I formerly speculated.

[diizzyy]

https://github.com/openwrt/packages/blob/master/multimedia/ffmpeg/Makefile#L484
Long shot but try adding --cpu=nocona at the end of the line, ffmpeg should be able to figure out cpu spport on its own.

[DazzyWalkman]

Thanks for the tip. However, it's doesn't work. I've updated my OP to include this test.

An unrelated side note:
I have to edit
https://github.com/openwrt/packages/blob/master/multimedia/ffmpeg/Makefile#L416
to change the cpu flag.
CPU flag in Line 484 would be overridden by Line 416.

[diizzyy]

Does ffmpeg work or does it also segfault?

[DazzyWalkman]

ffmpeg also segfault.
run:
ffmpeg -loglevel panic -i "./test.mp4" -y -frames 1 -q:v 1 -vf "select=not(mod(n,)),scale=-1:360,tile=5x20" "/root/test.mp4.jpg
syslog:
kern.info kernel: [69540.808961] ffmpeg[14594]: segfault at 7f7839445320 ip 00007f783880e30e sp 00007ffc25642bf0 error 7 in libavcodec.so.57.107.100[7f78387f1000+8b2000]
My usage is all about generating video contact sheets. So only libavcodec is involved.

[diizzyy]

Hmm... If you do a debug build you could use gdb and hopefully figure out why it segfaults.

Additionally depending on how much you want to spend on this you could try my ffmpeg4 branch which slightly changes the way ffmpeg is being built although I doubt that's the cause. The last commit is untested but "it should work"(TM).
https://github.com/diizzyy/packages/tree/ffmpeg4/multimedia/ffmpeg4
At least you should be able to tell if still segfaults, if you want ffprobe you need to remove the switch and add it to the install section at least. Works on ppc, arm and mips at least.

[neheb]

I'll just mention that ffmpeg on i486_pentium does not compile:

https://downloads.openwrt.org/snapshots/faillogs/i386_pentium/packages/ffmpeg/full/compile.txt

The issue is that the nasm binary built as part of the toolchain is not assembling correctly. objdump and strip give unrecognized format errors. The simplest fix would be to disable x86asm for the legacy x86 target.

[DazzyWalkman]

@neheb
In my case,
--disable-x86asm
or even an additional overkill not intended for x86 --disable-asm --disable-inline-asm still segfault.

@diizzyy
I am interested to test your ffmpeg4 builds, however pulling in ffmpeg4 directory and remove old ffmpeg under multimedia feeds then menuconfig triggered a couple of complaints about unsatisfied dependencies, and ffmpeg4 did not get added. Maybe I would read again the basics of adding packages. Or do you have some quick and dirty way of build a single package? I have already got a buildroot in place.

As for the time to spend, I am just an ffmpeg user and happy addicted OpenWRT snapshot tester. And I think my use case and environment combination are unpopular, if not rare among OpenWRT users.

The present situation in no way will jeopardize my daily operation, for I have a working good old version in hand, and more, since I started to try virtualizing OpenWRT on nodes and cluster, it does not necessarily confine the use of ffmpeg on OpenWRT. If someday my needs call for a newer version of ffmpeg, a container of some distro with working ffmpeg builds would be an easy alternative.

I tried reading through commits of upstream ffmpeg repo, but the effort seems to be prohibitive for me.
If things really go deep into assembly, the best bet I will try is comparing building outputs of 3.2.12 and newer version, to see if something comes up. I am not familiar with gdb, I can give it a try though.

It will take time, but I am not in a hurry. Maybe before I solve this mystery, the old hardware will eventually be phased out and the problem solves itself.

[diizzyy]

@DazzyWalkman
Place the ffmpeg4 directory in feeds/packages/multimedia
./scripts/feeds update -i
./scripts/feeds install -a
make menuconfig
Select ffmpeg4 (you need to enable patented code for now in Global build settings) and deselect ffmpeg and potential selected libraries libffmpeg.

[DazzyWalkman]

@diizzyy
I have tested your ffmpeg4 branch. It works like charm on x86_64. No more segfaults on previously mentioned platforms, even I have to enable ffprobe and png support to suit my needs.

I notice you may have disabled some rare used modules to shrink the total package(s) size from 8.5 mega bytes of 3.4.6 full to 4.6 megabytes of ffmpeg4.

Other things have happened before this test:

1. I have skipped ahead, switched OpenWRT to 4.19 kernel,
2. Building host has been updated to Ubuntu 19.04,
3. musl has been updated to 1.1.22, however, without make dirclean performed. (openwrt/openwrt@8262537)

I have not tested if 3.4.6 clean built under these circumstances also works ok yet. So I'd like to leave this issue open for now, until some further tests conducted.

It is nice to be back on track with latest ffmpeg, and do not have to stay with the soon-to-be-EOL 3.2 branch.

Thank you.

[diizzyy]

@DazzyWalkman
Many thanks for testing!

The idea is to only include what you'd normally need in one of these scenarios:

• Serving media using DLNA using miniDLNA/ReadyDLNA and Gerbera with support for possible transcoding of audio as hardware is too weak for video transcoding.
• Remuxing and transcoding IP(TV)/ATSC/DVB-streams for serving using nginx etc
• Remuxing and encoding audio streams
• Full functionality using youtube-dl
• Maintain good performance on plaforms where storage space isn't extremely limited as I'd like to avoid extroot if possible.
• Support most common audio and video formats for extraction, transcoding and "edit".

...and somehow try to fit all this into "one size fits all" to avoid a zillion of variants and options ;-)

[DazzyWalkman]

There is no change of situation concerning ffmpeg 3.4.6.
As I have already moved to ffmpeg 4, no further test would be conducted.
I am closing this issue. Thank you all.

[diizzyy]

@DazzyWalkman
If you have time and/or interest, can you have a look at https://github.com/diizzyy/packages/tree/ffmpeg4/multimedia/ffmpeg4 again?

[DazzyWalkman]

As of diizzyy@0ea14d0
Works on X86_64, J1900 CPU with png encoder manually enabled in the Makefile. No segfaults.
Things to note:
I cannot verify the x86 fix for I have obsoleted x86 long ago.
Although I enabled everything in menuconfig let alone openssl since it's mutually exlusive to mbetls, I cannot test modules outside my routine video contact sheet generation. In particular, anything not belong to video decode, png encode or ffprobe is untested.
Hope it helps.
@diizzyy

[diizzyy]

Thanks, I added ffprobe and png because you asked about it last time ;-)
I missed that ffmpeg offered an encoder. I'll fix that.

[diizzyy]

@DazzyWalkman
There you go, PNG encoder added. No need to manually hack the Makefile (I hope) :)