Decide on FFMPEG support

[dmik]

Mozilla provides built-in support for FFMPEG (H264/AAC) video and includes a copy of the ffmpeg library used for that. However, this support is always disabled in Windows, OS X and Android by default. I disabled it on OS/2 as well in 9732dbb but we need to re-evaluate this after the initial build of esr31 is done. We have a working ffmpeg port (hobbes) so we may benefit from it too (if there is really some benefit).

A quick search suggests that it may be disabled on Win and OS X due to patent issuses. I also want to find what exactly this support gives.

[dbanet]

Well, hopefully, it will enable MP3 playback (#67). Also, an RPM-packed FFMPEG version would be handy.

[dryeo]

On 04/24/15 03:26 AM, Dmitriy Kuminov wrote:

Mozilla provides built-in support for FFMPEG (H264/AAC) video and
includes a copy of the ffmpeg library used for that. However, this
support is always disabled in Windows, OS X and Android by default. I
disabled it on OS/2 as well in 9732dbb
9732dbb
but we need to re-evaluate this after the initial build of esr31 is
done. We have a working ffmpeg port (hobbes) so we may benefit from it
too (if there is really some benefit).

The obvious benefit is being able to play H264 videos (in theory most
any audio/video format, even IBM Ultra-motion but the FF support has to
be there). Need HTML5 audio support in FF as well.

A quick search suggests that it may be disabled on Win and OS X due to
patent issuses. I also want to find what exactly this support gives.

I doubt that we have to worry about patent issues, especially if the
FFmpeg binaries are hosted somewhere besides the States.
The current way to enable H264 is through http://www.openh264.org/ where
Cisco builds and hosts the binaries, Firefox downloads them and Cisco
covers the licensing. This may be possible by giving Cisco a eCS license
along with a development environment.
Current OS/2 FFmpeg support,
http://fate.ffmpeg.org/history.cgi?slot=x86.os2.446. Note that to pass
all the tests takes OS/2 Warp v4 (avoid the FPU bug), Libc065 (libc066
breaks MP2 encoding, see above link for failures down the page) and GCC
4.4.6 (newer GCCs all are broken when it comes to high optimizations.

[dmik]

Thank you guys for commenting on that! We will take it into account.

[dmik]

Some clarifications. The mozilla tree doesn't actually include the FFMPEG library itself — only headers from version 0.8.9. The lbrary is dynamically loaded by content/media/ffmp4/ffmpeg/FFmpegRuntimeLinker.cpp (as already mentioned by @komh and @dryeo in other tickets). I took the libav (ffmpeg fork) DLLs from the Dave's build of 0.8.17 from #16 and modified FFmpegRuntimeLinker.cpp to work on OS/2. However, the test from www.quirksmode.org/html5/tests/video.html still shows H.264/MP4 is not supported and I don't even see that this FFmpegRuntimeLinker code is ever executed. Something else needs to be done/enabled/triggered, apparently. I will find what.

[dmik]

The strange thing is that Mozilla doesn't even try to find a decoder for video/mp4 (as it does for video/webm and video/ogg, for instance) when I load the above test. Note that judging from the code, you have to have the following prefs set to true:

media.fragmented-mp4.enabled
media.fragmented-mp4.ffmpeg.enabled

to get libav involved. Note that the second perf is for some reason set to false by default (while the first one is true). I have no idea why. Anyway, the execution doesn't even reach these checks at the moment. So it seems that MP3/H.264/FFMPEG is disabled at some other level.

@dryeo btw, I don't see any connection between FFMPEG and MP3 in Mozilla so far. And the tests show MP3 doesn't work.

[LewisR]

@dmik Looks like the above prefs came about here:

https://bugzilla.mozilla.org/show_bug.cgi?id=886196

Also notes of some possible interest:

https://oneanddone.mozilla.org/en-US/tasks/89/

[LewisR]

Also, I believe that as of FF 35, media.fragmented-mp4.enabled now defaults to true. My reading of some of the *nix support threads seems to indicate that at the time this was implemented, playback of fragmented streams was problematic, so as a safety feature this was left disabled until the supporting libs improved.

[dryeo]

@dmik MP3 (and FLAC?) playback uses gstreamer on Linux and the system codecs on Windows and up until recently on OSX which is moving to gstreamer. No idea how hard it would be to use the system codec on OS/2 though most people have it installed.

[abwillis]

Lewis Rosenthal wrote:

@dmik Looks like the above prefs came about here:

https://bugzilla.mozilla.org/show_bug.cgi?id=886196

Also notes of some possible interest:

https://oneanddone.mozilla.org/en-US/tasks/89/

---

Reply to this email directly or view it on GitHub:
#102 (comment)
I haven't followed the ffmpeg support previously but that second link
suggests we may need a later release:
"First make sure you have the latest version of Nightly (Firefox 37)."

[dmik]

What you say guys makes sense becaus my findings through the code show that MP4 support is not complete in ff31. I guess I know how to enable it but I will be able to try it only on Monday. Thank you for the links.

[dryeo]

@abwillis odds are that version 37 is when they finally decided to enable the experimental feature. Seen the same with Gstreamer, worked fine for a long time before being officially enabled. @dmik Seems that in https://bugzilla.mozilla.org/show_bug.cgi?id=908503 they imported libstagefright to enable all types of MP4 playback instead of just fragmented. Kind of a surprising choice as usually libstagefright is used for hardware decoding on Android. According to https://source.android.com/devices/media.html stagefright does include software decoders. Bug 908503 was targeted at FFv32 so I'd guess it could easily be backported and it does include changes to the fmp4 backend.

[dmik]

I've enabled the MP4 decoder instantiation and made it successfully load the LIBAV dlls but it still doesn't give the MP4 video playback. Some changes that I had to make suggest that in ff31 we have something very preliminary that was never properly tested even on a linux desktop. For example, the original code attempts to import av_dict_get from libavcodec while it's in fact located in libavutil. And they never trigger the MP4 decoder activation because they never check for if it supports video/mp4...

I will do some more research.

[abwillis]

Rather than back porting to FF31, would it not make sense to tackle FF38 first where backport is not necessary?

[dmik]

@dryeo I couldn't find any obvious differences in content/media/fmp4 between FF 32.0.3 and 31.8.0 that could potentially solve the problem. Even the "wrong" export is still there. There are also some (more significant) changes in the above directory which pull things from other parts of the mozilla tree and therefore can't be easily back ported. Given that in FF38 and up the layout of these classes has been heavily changed (for instance, content/media was moved to dom/media, many new files were added, support for different libav versions), I think it makes little sense to deal with that right now.

We will postpone it till we work on 38 ESR (which is already out BTW).

Nevertheless, I committed my changes in 29677e4 just for historical purposes.

[dbanet]

Doooh. :((

And I initially was hoping to have SoundCloud working without Flash with FF17...

[dryeo]

@dmik Thanks for committing your work. I've applied the patches from https://bugzilla.mozilla.org/show_bug.cgi?id=908503 and the associated BSD fixes and will test, probably still won't work but the patches apply cleanly including changing the demuxer. I'll also change to exports by name in libav so the next releases won't need patching (currently by ordinal due to FFmpeg used to not have releases). Question, should the PROTMODE line be removed from the def files? Wlink complains about it.

[dmik]

@dryeo Your last build of libav (0.8.17) already exports everything by name — I used it for my testing. Or was it just a quick hack? Re PROTMODE, you may safely remove it, it's by default anyway. Please report if you get any success with backporting things from the ticket.

[dmik]

Note that I re-enabled FFPMEG on OS/2 by default in d7c232a because libstagefright needs GetDemuxerLog expected to be defined by MP4Reader.cpp in our case which is only build when FFMPEG is enabled.

This means that I didn't port the new libav loading code to OS/2 but simply enabled the unix code to build (it is relatively harmless as it will not find the right dlls anyway ATM). Porting it should be easy but I will do that (and test FFMPEG) once we get esr38 running.

Note also that 29677e4 is fully reverted now (with f8df811) because the same thing is already done in esr38 upstream (a few lines above). Which also proves that in esr31 it was not complete and that my experimental change in that place was in the right direction, BTW.

[dmik]

I adapted the DLL loading code to OS/2 in 6acd51f and got H.264/AAC immediately working with the latest Dave's build of libav v10.7 (https://bitbucket.org/dryeo/libav/downloads/libav_10_7.7z), which can be see both here http://www.quirksmode.org/html5/tests/video.html and here http://html5test.com/.

I've put the test XUL build here: http://rpm.netlabs.org/test/tx2.zip (note that you will also need t.zip for the rest of 38.6.1).

In order to enable FFMPEG, the following preferences must be set to true (they are false by default):

media.fragmented-mp4.exposed
media.fragmented-mp4.ffmpeg.enabled

Note that for now we only support two sets of libav DLLs: libav 0.8.7 (avform53.dll etc) and libav 10.7 (avform55.dll etc). You should have either of them in LIBPATH to make it work.

Note that if you run into problems, you may enable FFMPEG logging with NSPR_LOG_MODULES=FFmpegDecoderModule:9 to see what went wrong.

Please test.

[dryeo]

Testing with SeaMonkey, MP4 playback seems fine, sound works unlike the test webm videos of VP8/VP9 and Vorbis/Opus that I've transcoded. I'll try transcoding various bitrates and such later and do more testing and also test using FFmpeg rather then Libav though they've diverged quite a bit since the fork. For an RPM, libav could be linked with fontconfig, freetype, libvpx, libx264, libmp3lame, libxvid, libopus, libtheora, SDL and decided whether to build it as LGPL or GPL. I think the MPL allows linking to GPL.

[dmik]

Yes, trying ffmpeg instead of libav is a good idea. Wikipedia says that libav was pushed by Debian/Ubuntu all these years but now they switched back to ffmpeg because the libav team is not as good at closing vulnerabilities as the ffmpeg team. I wonder if the ffmpeg DLLs are ABI compatible with libav ones. This need checking (will do on Monday).

[dryeo]

For a while after the fork, FFmpeg maintained a libav compatible switch but eventually they diverged too much. I have a build of FFmpeg v3.0.1 in Hobbes incoming libavutil 55. 17.103 / 55. 17.103 libavcodec 57. 24.102 / 57. 24.102 libavformat 57. 25.100 / 57. 25.100 libavdevice 57. 0.101 / 57. 0.101 libavfilter 6. 31.100 / 6. 31.100 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.101 / 2. 0.101 libpostproc 54. 0.100 / 54. 0.100 The DLL names of libav and FFmpeg (with the same major version number) conflict.

[dmik]

Then we will perhaps need to wait until the Mozilla team switches back to ffmpeg in their code.

[dryeo]

Feeding SM various testcases, it quickly got to the point where it stopped playing any video, possibly due to not liking some of the file types. Using Big Buck Bunny (freely downloadable) as a test case, the video would often freeze before finishing and would no longer seek or play. After closing the video tab, the video file stays locked and Seamonkey hangs while closing, forcing a reboot before I can restart it. It does write the session restore stuff before hanging so cleanly starts up again.

[dmik]

Actually, I have a feeling after reading some internet that ffmpeg dlls will work with Firefox as a drop-in replacement of libav. I need to build ffmpeg and check this myself before deciding how to go.

Note that they also enabled ffmpeg support by default in FF44+, see https://bugzilla.mozilla.org/show_bug.cgi?id=1207429.

[dmik]

The latest build of FFMpeg 3.0.1 (57.x ABI) by Dave lacks _avcodec_get_frame_defaults used by Firefox. This function was deprecated in 55.x ABI and seems to be still there in 56.x ABI (v2.8 branch). And 56.x ABI is supported by Firefox, according to the code. So I will build the latest release of FFMpeg 2.8 myself and see if it works with Firefox (I believe it does). If so, we will choose FFMpeg over libav and I will create RPMs FFMpeg and all its requirements (that is, a newer libvpx, lame, libogg, libtheora, opus, x624 and xvidcode — these are already on Hobbes built by either Dave or KOMH so it should be relatively easy to create RPMs for them).

@dryeo btw, your README.OS2 in FFMpeg 3.0.1-r2 zip misses the ogg0.dll zip reference. And the statement that if you have Mozilla installed then all requirements are there doesn't appear to be correct either. I understand it's a PITA to maintain dependencies by hand and this is where RPM/YUM wins big time :)

[dryeo]

On 04/11/16 01:48 PM, Dmitriy Kuminov wrote:

@dryeo https://github.com/dryeo btw, your README.OS2 in FFMpeg
3.0.1-r2 zip misses the ogg0.dll zip reference. And the statement that
if you have Mozilla installed then all requirements are there doesn't
appear to be correct either. I understand it's a PITA to maintain
dependencies by hand and this is where RPM/YUM wins big time :)

Yes, it was a quick readme and I realized I'd forgotten about ogg (which
FFMpeg itself handles fine) right after I uploaded it.
The Mozilla requirements were just for Fontconfig and Freetype.
Note that make install will also install the long named DLLs, which I've
just been manually removing

[dmik]

I finally built FFMpeg 2.8.6 and it works well with Firefox - at least not worse than libav. So we will definitely stick with FFMpeg from now on. I will create the necessary RPMs etc.

[dmik]

Hmm, I've built RPMs using RPMFusion .spec, but it traps Firefox immediately. Very strange! Must be some extra options. Have to experiment more.

[dmik]

Hmm, even pmdll traps immediately when I try to open avform56.dll. This must be lxlite or something like that.

[dmik]

Okay, that was lxlite 1.2.1. With lxlite 1.3.3 all works ok (we need to deal with that once as 1.3.3 has at least one case that doesn't work, XUL.DLL IIRC). The ffpmeg RPMs are uploaded and seem to work well with Firefox. Only the ffmpeg-libs RPM is needed, the rest is not necessary for H.264/AAC playback.

Note that the current build lacks support for lame, x624, vorbis, theora, opus and xvid as we don't have the respective RPMs yet. Once I make RPMs for these libs, I will release a new ffmpeg set. This, however, doesn't influence the current Firefox functionality so it's postponed (we may need mp3 support in ffmpeg later, once Mozilla drops gstreamer support completely in favor of ffmpeg and their own mp3 demuxer, see #67).

[dmik]

In 64bdbff I enabled support for FFMPEG 2.8.x (56 ABI) which is needed to see the ffmpeg DLLs from RPMs.

In 0483c98 I enabled media.fragmented-mp4.exposed and media.fragmented-mp4.ffmpeg.enabled by default on OS/2 (these settings are dropped in FF 44+ but we need them until then). This issue should be closed now.

[dryeo]

On 04/15/16 06:17 am, Dmitriy Kuminov wrote:

Okay, that was lxlite 1.2.1. With lxlite 1.3.3 all works ok (we need to
deal with that once as 1.3.3 has at least one case that doesn't work,
XUL.DLL IIRC)

I'm using Version 1.3.5 shl, which Steven fixed for xul.dll (used to
fail due to 32 MB obj limit) and it works fine for FFmpeg and Mozilla.

[dmik]

@dryeo thanks! The link is http://home.earthlink.net/%7Esteve53/betas/lxlt135-shl-20121201.zip, right? I didn't know that he did it (back in 2012 it seems). What a shame. I think I will make an lxlite RPM out of it if it works well for me.

[dryeo]

Yes, that looks like it. I just took it for granted that you were using that version as most fail to compress xul.dll IIRC.

[dmik]

No, 1.3.3 (that comes with eCS 2.2 by default) worked well until some recent XUL growth (esr38 IIRC). Then I switched to 1.2.1 that handled the latest XUL well... I've installed 1.3.5, let's see how it behaves.

[dryeo]

It was debug builds of xul.dll (Firefox 10ESR) that lxlite was failing on.

[StevenLevine]

right? I didn't know that he did it (back in 2012 it seems). What a
shame. I think I will make an lxlite RPM out of it if it works well for
me.

I could swear I did this for you as part of the wlink debug data
enhancments. :-)

[StevenLevine]

It's hard to say if 1.3.5 will work any better than 1.3.3. It should not
trap on large debug data. Lxlite does everything in memory and Virtual
Pascal has no highmem support so it's pretty easy to run out of memory.
We had a similar problem with AOO's swooo.dll.

If you would like me to work on the lxlite failure, zip up a testcase and
upload it to

ftp://ftp.warpcave.com

using

userid: upload2
password: 4steven

Include the failing xul.dll and a .cmd file the invokes lxlite with the
switches you want to use. Once I have this, either I will be able to fix
it or I will be able to estimate what it will take to make lxlite handle
something of this size.

You can do the same for the pmdll failure. If pmdll is running out of
memory, this might be easy to fix. Arthur used DosAllocMem, so adding
OBJ_ANY and some code to accomodate really old kernels, should be
sufficient.

[dmik]

Okay I must have forgotten it then :) Now I recall we had a discussion about that but given that I didn't try 1.3.5 before looks that I lost its result somehow. Okay, I will report you if I succeed with 1.3.5 or not etc.