[FFmpeg] version bump to n1.2 (rev e820e3a)

[FlyingRat]

FFmpeg version bump to n1.2 (rev e820e3a).

This solves a bunch of more or less serious problems that we currently struggle with in version 0.10.2, for instance preventing users to watch certain content types, hangs or causes stuttering video playback. Also, n1.2 is the last ffmpeg version that supports the old api. Essentially, this upgrade contains the following files and directories:

• dir: lib/ffmpeg
  FFmpeg n1.2 fork (rev e820e3a) including all necessary xbmc add on patches.
• dir: lib/ffmpeg/patches
  This directory contains all the XBMC add-on patches applied to ffmpeg n1.2.
• file: lib/ffmpeg/patches/README-patches
  Brief description of the xbmc add on patches.
• file: lib/ffmpeg/ffmepg/patches/obsolete-patches
  Contains obsolete xbmc custom patches that were NOT applied to n1.2.
• file: lib/ffmpeg/build_xbmc_win32.sh
  Win32 build script that generates the include file DllPaths_generated_win32_ffmpeg.h and is called from buildmingwlibs.sh.
• file: lib/DllAvFormat.h
  Deals with the decrepated version av_read_frame_flush -> ff_read_frame_flush
• file: lib/Makefile.in
  When ffmpeg is configured for osx then remove duplicate log2_tab.o from libavcodec.a, libavformat.a and libswresample.a
• file: xbmc/cores/dvdplayer/DVDCodecs/Video/DVDVideoCodecFFmpeg.cpp & xbmc/cores/dvdplayer/DVDCodecs/Video/DVDVideoCodecFFmpeg.h
  Resolves dllAvFilter dependencies of LibPostProc that previously required LD_LIBRARY_PATH
• General adjustments to n1.2 (files)
  file: xbmc/cores/dvdplayer/DVDCodecs/Video/DVDVideoCodecLibMpeg2.cpp
  file: xbmc/cores/dvdplayer/DVDCodecs/Video/DXVA.cpp
  file: xbmc/cores/dvdplayer/DVDCodecs/DVDCodecs.h
  file: xbmc/cores/dvdplayer/DVDAudio.h
  file: xbmc/cores/dvdplayer/DVDDemux.h
  file: xbmc/cores/dvdplayer/DVDPlayerAudio.h
  file: xbmc/cores/dvdplayer/DVDStreamInfo.h
• file: tools/depends/native/gas-preprocessor-native/gas-preprocessor.pl
  Updated version that manages libavcodec/arm assembler optimizations. Source: github.com/mansr/gas-preprocessor, rev 76a72f00e (Dec 03, 2012).
• file: project/Win32BuildSetup/BuildSetup.bat
  Win32 build script adjusted to ffmpeg n1.2
• file: project/Win32BuildSetup/buildmingwlibs.sh
  Win32 build script that generates the include file DllPaths_generated_win32_ffmpeg.h.
• file: xbmc/DllPaths_win32.h
  DllPaths_win32.h is now modified to include the auto-genrerated file "DllPaths_generated_win32_ffmpeg.h".

[wsoltys]

could you please separate the changes to lib/ffmpeg and the rest to different commits? would make it easier to review the changes to the XBMC core.

[FlyingRat]

Done, commit splitted in two parts.

[FlyingRat]

pushed the close button by accident...

[elupus]

The patches directory would be nice if it was named the same as before. Also i really want the ffmpeg part of the update a separate commit from the addition of the patches.

[FlyingRat]

• bookmarks and TODO:s cleaned and wiped!
• renamed: XBMC-patches -> patches
• updated: patches/README-patches
• renamed: patches/OBSOLTE -> patches/obsolete-patches
• Commit: "[FFmpeg] version bump to n1.2 (rev e820e3a) - lib/ffmpeg"
  lib/ffmpeg n1.2 including xbmc patches.
• Commit: "[FFmpeg] version bump to n1.2 (rev e820e3a) - xbmc files"
  Designated xbmc files...

[wsoltys]

nice work on the ffmpeg bump. I'm a little unsure about all the batch file changes. most will get lost when switching to jenkins since I already rewrite them but probably I can get some nicer ways to handle it from your scripts.
I have mixed feeling about the automatic generation of the header includes. For the two times in a year (if ever) we bump ffmpeg I'm able to change the files by hand and all the error prone changes wouldn't be needed.

[FlyingRat]

Thanks! The intention was to make the build process work smoother but you are of cource welcome to disregard or/and restructure the any of the win32 build script whichever way you see fit best for the new Jenkins build environment.

[aballier]

Very nice work. However, I fear ffmpeg 1.2 (as opposed to eg 1.0.6) could break some things by introducing planar audio formats.
I'm not sure that these files will still work:
xbmc/cores/AudioEngine/Encoders/AEEncoderFFmpeg.cpp
xbmc/cdrip/EncoderFFmpeg.cpp

for example, ffmpeg aac encoder now seems to expect planar float which xbmc has no clue about

[FlyingRat]

@aballier, please provide some media samples we can verify with that also can work as a reference if we need to alter stuff. /Thanks in advance, Lars.

[aballier]

well, those are encoders so basically any sample would work ;)
its just a matter of triggering that code path, but note that i'm not saying that based on my experiences but rather based on my reading of the code so it's possible I'm wrong; I'm just saying to be careful about these two encoders :)

[FlyingRat]

@popcornmix, referenced this pull request in huceke/omxplayer: (github.com/huceke/omxplayer/issues/150)

"Issue #150 MKV video plays at double speed (FFmpeg-1.2)"

I'm sorry, but what has OMXplayer and "Built a recently checked out omxplayer against external ffmpeg-1.2" anything to do with this pull request??

Btw, as this is not a discussion forum please move further discussion to: http://forum.xbmc.org/showthread.php?tid=156303&action=lastpost . /Thanks in advance, Lars.

[FlyingRat]

@aballier commented:

well, those are encoders so basically any sample would work ;) its just a matter of triggering that code path,
but note that i'm not saying that based on my experiences but rather based on my reading of the code so
it's possible I'm wrong; I'm just saying to be careful about these two encoders :)

Ok, thanks. Since there are no actual problems reported I'll consider this particular case closed for the moment. You are more then welcome to get back if you find anything specific issues that should be solved but in the mean time let's move more general discussions to: http://forum.xbmc.org/showthread.php?tid=156303&action=lastpost /Thanks, Lars.

[wsoltys]

@FlyingRat: please bring back the copy of the pdb file as we need it that jenkins can fetch that too. Then I'm fine with it.

[ghost]

@FlyingRat, "apologize" to popcornmix, all he did was open an issue on his repo. github is responsible for the noise..

[FlyingRat]

cptspiff commented

@FlyingRat, "apologize" to popcornmix, all he did was open an issue on his repo. github is responsible for the >noise..

Well, sorry about that but I had no idea about this "feature". I'll know github is good for many things but... anyhow, now I know.

@popcornmix good 👍 github bad (well, surprises..) 👎

[FernetMenta]

@FlyingRat How much testing does this require on Linux with vdpau and vaapi?

@sraue You already run external ffmpeg 1.2 with OE, right? Are there any tweaks you have done which should be considered here?

[FlyingRat]

@FernetMenta commented:

@FlyingRat How much testing does this require on Linux with vdpau and vaapi?

I've done quite some testing on my own but only with vmware fusion. There are some other folks on the following thread that I'm quite sure have done some more extensive testing and can probably answer any of your questions:
"[WIP] FFmpeg v1.1.x + XBMC add-on patches."

@sraue You already run external ffmpeg 1.2 with OE, right? Are there any tweaks you have done which
should be considered here?

Check this post: http://forum.xbmc.org/showthread.php?tid=156303&pid=1391318#pid1391318

[FlyingRat]

@wsoltys commented:

@FlyingRat: please bring back the copy of the pdb file as we need it that jenkins can fetch that too. Then I'm
fine with it.

Ok, will fix. Want a rebase or will a separate commit do?

[sraue]

@FernetMenta no, we build with external 0.10.6 with all XBMC patches applied which is similar to the included in xbmc-frodo/master (0.10.2 or so)

[FernetMenta]

I did a couple of tests on Linux with vdpau. Looks good so far.
Transcoding True HD to AC3 seemed to work as well.

[vicbitter]

This thread discusses the ffmpeg testing across the various platforms supported by OpenELEC

http://www.openelec.tv/forum/130-video-decoders/62608-xbmc-ffmpeg-1-1-2

[Alphix]

@FlyingRat The mkv embedded subtitle issue you've hit upon is something I've already fixed in my pull request (#2525)

[vicbitter]

@Alphix, I have applied your fix and subtitles are working again! Thanks

[FlyingRat]

@Alphix commented

The mkv embedded subtitle issue you've hit upon is something I've already fixed in my pull request (#2525)

Excellent work dude! 👍 👍 👍 Saved me a lot of headache, thank you!

[FlyingRat]

@cptspiff: is there a way to create a dependency to another PR (like this one: #2525) or how is this supposed to work?

[FlyingRat]

My findings regarding the common.mak issue:

• Brief summary:
  Currently Mingw-Make does only work when ffmpeg is checked out with core.autocrlf=false under win32 (as most already may have guessed!) It's quite odd that mingw-make can't cope with crlf endings since I guess this was not an issue with the old 0.10.2 ffmpeg.

  Are we missing a "miniwg" settings file or something like that?

• Background:
  The build env that was used is based on a ffmeg repo cloned from the ffmpeg n1.2 fork. Since it was a git repo it was also possible to utilizes local config settings like core.autocrlf=false. As soon as the PR made it into the xbmc master repo, ffmpeg just became a regular directory which makes it impossible to use local git config settings.

• Temporary workaround:
  A very temporary solution I've tested is to re-checkout ffmpeg with core.autocrlf=false but that is obviously a rather ugly solution!

• Possible solutions?
  Does it exist any platform settings (local or global) in the minigw environment that enables Mingw-Make to accept crlf instead of just lf? Can someone with deeper skills in this area please explain if anything like that is possible?

  Other ideas?

I'll continue to dig deeper into this issue and see if i may find anything in the old 0.10.2 environment. Any type of feedback or ideas is of course welcome...

[aballier]

@FlyingRat commented
What has changed in regards to the previous discussion on this topic? Trifle or not, please motivate and explain why. /Thanks in advance, Lars.

I should rather ask you this question. You agreed that this hunk shouldn't be merged, but it has been merged and you deleted the branch and all the comments with it...

FYI, again: the dllavformat changes merged with this PR makes xbmc use an internal ffmpeg symbol that may change its ABI (so that xbmc will crash) or even be removed (so that xbmc wont work) for absolutely no reason nor gain. Everything was safe before.

[FlyingRat]

Regarding buildsetup.bat

I had the same oddity with buildsetup.bat that behaved very strangely in the same way as reported in the forums but I couldn't locate any defects when i debugged it. When a renamed it and checked out the file again, everything worked as normal(!) That made me suspect that buildsetup.bat somehow was checked in with the wrong line endings.

I made a successful build of master (rev c01c97d) on win32 by first re-check out buildsetup.bat and then re-check lib/ffmpeg with git core.autocrlf=false (only used for ffmpeg)

Uploaded: XBMCSetup-20130412-c01c97d-dx.exe to https://www.dropbox.com/sh/a7tsbe6o80rrv2a/S2NszQXp7H

[FlyingRat]

@aballier: it might be a misunderstanding but pls let me get back to you when we have solved win32 problems. /Thank you, Lars.

[FlyingRat]

Worth a try: #2604

[FernetMenta]

here is a quick fix for AE transcoding: FernetMenta@ca16827

I leave it to @fritsch to make this nice :)

[fritsch]

@FernetMenta:
We don't have that many options, as AE internally works completely on AE_FMT_FLOAT as a basis for encoding. Changing the internal format would be the other option :-)

So we have to die one cpy death. PR it if you please.

[wsoltys]

@FlyingRat: short update. ffmpeg compiles with gnu make 3.82 (already committed to master). unfortunately libdvdnav and libdvdread won't compile anymore.
Whereas gnu make 3.81 seems to work fine with VPATH gnu make 3.82 just won't find the source files anymore. if I adapt the src paths in the Makefile directly libdvdread will compile but libdvdnav won't due to a missing rule.
Does any Makefile expert has an idea if this is a bug in gnu make or if there's something wrong with the libdvd makefiles?

[FlyingRat]

2013/4/13 wsoltys notifications@github.com

@FlyingRat https://github.com/FlyingRat: short update. ffmpeg compiles
with gnu make 3.82 (already committed to master). unfortunately libdvdnav
and libdvdread won't compile anymore.
Whereas gnu make 3.81 seems to work fine with VPATH gnu make 3.82 just
won't find the source files anymore. if I adapt the src paths in the
Makefile directly libdvdread will compile but libdvdnav won't due to a
missing rule.
Does any Makefile expert has an idea if this is a bug in gnu make or if
there's something wrong with the libdvd makefiles?

Morning! I've probably built 12-14 win32 envs the last couple of days and
everyone has worked with the "built in" msys-make that is downloaded to
BuildDependencies i.e:

k:\src\xbmc\project\BuildDependencies\msys\mingw\bin\make.exe -v
GNU Make 3.82
Built for i386-pc-mingw32
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Btw, how is v3.81 involved in this case, i mean do you have your own local
msys/mingw installed somewhere that may conflict with the other? This is
just speculations, but you don't think this happens to be another
core.autocrlf symptom as
well?

[wsoltys]

v3.81 was our current make until today when I update make to 3.82. and again I try to build ffmpeg with autoctrl=true as like I already explained we can't use false.

[FlyingRat]

Sorry, forgot to ask if you read my other mail. You'll hopefully reconsider
your opinion regarding autoctrl because I'm quite convinced that the
correct (and default) repository setting is after all
core.autoctrl=false. I rest my case (or at least until proven wrong) :D

ce8efc8#commitcomment-3005705

2013/4/13 wsoltys notifications@github.com

v3.81 was our current make until today when I update make to 3.82. and
again I try to build ffmpeg with autoctrl=true as like I already explained
we can't use false.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2554#issuecomment-16332782
.

[Voyager1]

@elupus @FlyingRat
Patch 0028 is still needed. With great thanks to @wsoltys I could finally get ffmpeg compiled on Win32, and immediately tested the dvds with still menus + audio. Same issue as before that patch.
Line 709 of libavformat/utils.c needs to be patched as follows. They've also wrapped AVPROBE_SCORE_MAX/4 as a new define...

C:\DATA\git\xbmc\lib\ffmpeg\libavformat\avformat.h (1 hit)
    Line 345: #define AVPROBE_SCORE_RETRY (AVPROBE_SCORE_MAX/4)
C:\DATA\git\xbmc\lib\ffmpeg\libavformat\utils.c (4 hits)
    Line 709:             if(    (st->codec->codec_id != AV_CODEC_ID_NONE && score > AVPROBE_SCORE_RETRY-1)

[wsoltys]

@FlyingRat : the autoctrl thing was a team decision that helps us working in a mixed environment (win/Linux). Switching would lead to huge problems which was the trigger to switch to autoctrl=true on windows systems.

[MartijnKaijser]

@FlyingRat
core.autoctrl=false is a no go end of story. Period!

@Voyager1
What did you else change to get it compiled?

[Voyager1]

@MartijnKaijser - I just redownloaded & updated the mingw environment, using DownloadMingwBuildEnv.bat. After wsoltys' update it now includes gmake 3.82. This one works ok with the windows line endings. The patch 0028 story is unrelated to getting it compiled (it is related to DVD with menus)

[FlyingRat]

Ok, i hear and understand what you say. But here are the facts:
core.autoctrl=false is the default repository setting and works without any
problems. If you don't believe me, please read .git/config and tell me what
you see.

Bottom line: regardless if it was the team decision to alter this flag for
Jenkins you need to find a another solution to deal with this problem.
Please take my advice regarding Jenkins, don't start altering code and
other configs to satisfy things but rather fix the Jenkins setup instead
with for example plugins, etc. But do whatever you like, i'm just trying to
make a point :)

2013/4/13 wsoltys notifications@github.com

@FlyingRat https://github.com/FlyingRat : the autoctrl thing was a team
decision that helps us working in a mixed environment (win/Linux).
Switching would lead to huge problems which was the trigger to switch to
autoctrl=true on windows systems.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2554#issuecomment-16333254
.

[MartijnKaijser]

this has nothing to do with Jenkings!
your point is not valid and I don't believe you. core.autoctrl=false won't happen.

[Voyager1]

@FlyingRat - I may be wrong but you seem to have built the ffmpeg libs 10-12x with gmake 3.82 and it worked fine. It works fine with 3.82 on my set up too, with windows CRLF line endings! so I think you're chasing the wrong problem here...

[FlyingRat]

Hey Martijn, did you actually read my mail reading the git config?? Well, 3.82
might be a lifesaver for win32 after all... but here are the facts (again):

$ type .git\config
[core]
repositoryformatversion = 0
filemode = false
bare = false
logallrefupdates = true
symlinks = false
ignorecase = true
hideDotFiles = dotGitOnly
autocrlf = false
[remote "origin"]
url = git://github.com/xbmc/xbmc.git
fetch = +refs/heads/:refs/remotes/origin/
[branch "master"]
remote = origin
merge = refs/heads/master
[branch "Frodo"]
remote = origin
merge = refs/heads/Frodo
[remote "upstream"]
url = git://github.com/xbmc/xbmc.git

fetch = +refs/heads/:refs/remotes/upstream/

Please stop blaming ffmpeg or anything else since this obviously a pure
CRLF problem that you brought in yourselves when
you started to override the default setting with core.autocrl=true.

2013/4/13 Martijn Kaijser notifications@github.com

@FlyingRat https://github.com/FlyingRat
core.autoctrl=false is a no go end of story. Period!

@Voyager1 https://github.com/Voyager1
What did you else change to get it compiled?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2554#issuecomment-16333262
.

[MartijnKaijser]

no i am ignoring that false statement

Edit:
and since you obviously did not pay attention due to chasing some ghost we already have ffmpeg build but are now stuck at libdvdread

[FlyingRat]

Well, ignore or override, the end result just became the same. OO for now...

2013/4/13 Martijn Kaijser notifications@github.com

no i am ignoring that false statement

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2554#issuecomment-16333918
.

[FlyingRat]

and since you obviously did not pay attention due to chasing some ghost we already have ffmpeg build
but are now stuck at libdvdread

Since i'm not using a build server like Jenkins (where core.autocrlf=true) i don't have this problem. Like i said, i'm using the standard git setting that comes with the xbmc repo and it works perfectly well because the effective default setting of core.autocrlff is false which compiles everything without any problems (with default gmake 3.81).

So I'm sorry, but you are on your own on this one.

OO & AWFK - definitely this time since my wife is shouting now! :)

[Montellese]

@FlyingRat: You should read the "Git Usage" article on our wiki: http://wiki.xbmc.org/index.php?title=Git_Usage
It clearly states that you have to set core.autocrlf to true. It doesn't matter whether you use jenkins or anything else, every dev committing something to XBMC needs to set core.autocrlf to true because otherwise you run into problems when people from different platforms (Windows vs. linux/osx) commit/pull stuff. Even github has a page stating that anyone using git on Windows should set core.autocrlf to true.

[FlyingRat]

Yes i know, but that's not the recommended settings for ffmpeg and libav on
Windows. I've also posted about this issue on the forums:
http://forum.xbmc.org/showthread.php?tid=156303&pid=1363840#pid1363840. But
you are probably correct, and the following guys are obviously wrong and
don't know what they are talking about regarding core.autocrlf ;-) [?]

http://stackoverflow.com/questions/2825428/why-should-i-use-core-autocrlf-true-in-git
http://ffmpeg.org/git-howto.html
http://libav.org/download.html

http://libav-users.943685.n4.nabble.com/Libav-user-makefile-problem-on-win32-MinGW-td4657083.html

But when/if you find an alternative solution you could do them a favour and
post that solution on their forums. Need get back to the family again.
Later dudes!

2013/4/13 Sascha Montellese notifications@github.com

@FlyingRat https://github.com/FlyingRat: You should read the "Git
Usage" article on our wiki: http://wiki.xbmc.org/index.php?title=Git_Usage
It clearly states that you have to set core.autocrlf to true. It doesn't
matter whether you use jenkins or anything else, every dev committing
something to XBMC needs to set core.autocrlf to true because otherwise you
run into problems when people from different platforms (Windows vs.
linux/osx) commit stuff. Even github has a page stating that anyone using
git on Windows should set core.autocrlf to true.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2554#issuecomment-16334816
.

[elupus]

Sadly you are missing the point, the auto setting has nada zilch to do with
ffmpeg. It's xbmc as a whole.

It may very well be that it's bad for ffmpeg to have it on, but that is
beside the point. It's needed for the rest of xbmc.

We have gone through a whole lot of grief regarding this already.

That said, it's obviously not your fault that we have issues now, nor your
job to fix it... , unless you have a magic wand that solves it magically :-)

[ghost]

it isnt very hard to see the problem here. usually windows wants to be a type writer. this logic breaks down when windows wants to pretend it is unix. either the tools need to find some common ground, which was the problem that caused the autocrlf in the first place or you deal with this through some variant of (conditional) prebuild patchery. likely not as a patch but through a call to dos2unix.

[Voyager1]

I think prebuild patchery à la dos2unix is a bad idea as this would cause modification of source files. I believe that where we landed now, using a gnu make that deals with it properly is much better. It seems that for now, this issue is solved, and we can refocus on getting ffmpeg 1.2 a good round of testing...

[ghost]

it is no more changing the source than what git does in the first place.. that is after all what the setting discussed inbhere does... anyways i agree a make that handles it is better. what i didnt get was all the confusion that seemed to be around what was happening.

[magnyld]

Wouldn't this be easily fixed by adding text eol=lf for ffmpeg in the .gitattributes? Instead of depending on a make that hopefully handles crlf correctly on windows.

[FlyingRat]

I need to supply a video sample for a patch submitted upstreams as a rfc to
ffmpeg. Does anyone know where to find the video file "brokenStream.mpg"
(or similar) that is related to patch 0015 below?

https://github.com/xbmc/xbmc/blob/master/lib/ffmpeg/patches/0015-fixed-memleak-in-mpegts-demuxer-on-some-malformed-mp.patch

From b83c9a2505338cdf021dd499c26686e82bcbc066 Mon Sep 17 00:00:00 2001
From: Joakim Plate elupus@ecce.se
Date: Fri, 26 Nov 2010 20:56:48 +0000
Subject: [PATCH 15/24] fixed: memleak in mpegts demuxer on some malformed
(??) mpegts files with too large pes packets

at-visions sample file brokenStream.mpg