Integrated libx264 for higher quality mp4 output #1564

chaosphere2112 · 2015-09-22T22:54:53Z

Fixes #1118, and gives us much nicer quality animation output.

durack1 · 2015-09-22T23:01:46Z

@chaosphere2112 nice! How much nicer are the outputs? And are they also smaller?

doutriaux1 · 2015-09-22T23:31:03Z

@chaosphere2112 looks like you're missing some files.

doutriaux1 · 2015-09-22T23:31:57Z

@chaosphere2112 you might as well use cdat_confiugre_step for x264 as well.

chaosphere2112 · 2015-09-23T00:13:13Z

@doutriaux1 I just did a dumb thing (removed the ffmpeg_configure_step.cmake.in and x264_configure_step.cmake.in without removing the configure_file command, despite them not being in use). Also forgot to update the ffmpeg() call on canvas to actually take advantage of this stuff.

@durack1 A lot better. And, yes, smaller as well.

Codec	Size	Quality
mpeg2	2 MB	Low
mpeg2	6.3 MB	High
h264	3.6 MB	High

doutriaux1 · 2015-09-23T00:33:57Z

@chaosphere2112 yasm stuff

durack1 · 2015-09-23T04:33:47Z

@chaosphere2112 great - thanks for doing this.. I also note that the h264/*.mp4 encoded files play directly in any modern browser, whereas the mpeg2/*.mp4 files can be painful with codecs (and don't just play in FF41).. This is a great jump forward!

chaosphere2112 · 2015-09-23T14:36:07Z

@doutriaux1 On the system requirements page we list YASM as a requirement for Ubuntu, though neither of the other platforms. Hm... I can just wrap the configure args to check if YASM is installed and disable it otherwise?

durack1 · 2015-09-23T16:25:44Z

Packages/vcs/Lib/Canvas.py

-            cmd += ' -r %s ' % rate
-        if bitrate is not None:
-            cmd += ' -b:v %sk' % bitrate
+        cmd += " -pix_fmt yuv420p "


@chaosphere2112 are h264 encoded files now the new default for the x.ffmpeg() and the x.animation() functions (where x = vcs.init())?

Yup, FFMPEG defaults to using h264 when libx264 is installed. That's the reason for the -pix_fmt yuv420p option (they default to a pixel format that quicktime doesn't support but most other players do, so I went with the lowest common denominator).

chaosphere2112 · 2015-09-23T16:56:27Z

Let's hang off a sec on merging this, @doutriaux1; it turns out Cisco provides a binary version of H264 that is properly licensed for anyone to use, so I'm going to try linking against that with ffmpeg and see what happens.

chaosphere2112 · 2015-09-23T16:56:43Z

@doutriaux1 also there's all of the build failures, but that's probably not my fault.

aashish24 · 2015-09-23T17:29:22Z

Looks like we are running into CMake version issue. This is new but I don't know why we didn't see this before.

chaosphere2112 · 2015-09-23T21:11:24Z

@doutriaux1 Rather than porting to OpenH264's source, I'm content just sticking with libx264. It's better supported by FFMPEG, and we already made it work here. I'll push a fresh commit to restart these... not sure why they all failed.

durack1 · 2015-09-23T22:26:21Z

@chaosphere2112 there are a bunch of master tweaks to get cmake working and other things, a rebase might be useful to pull all these little tweaks and get your branch passing all the buildbot tests..

doutriaux1 · 2015-09-24T03:22:32Z

@chaosphere2112 looks like Ubuntu is hang. Will try on my machine before giving it the final go. Did you try under Ubuntu?

durack1 · 2015-09-24T03:52:38Z

@chaosphere2112 to fix travis-ci/push you'll need to merge in #1565, it seems garant has a standing issue with vcs_test_click_info this has repeated in a couple of standing PRs #1540, #1560, #1557

doutriaux1 · 2015-09-24T05:19:50Z

@durack1 I believe this PR is already in master so if the bot started after it went in there is no need for @chaosphere2112 to merge it in, the bots automatically merge master in.

doutriaux1 · 2015-09-24T05:20:40Z

ooops @durack1 I take it back... the bots WILL merge it in from master, but travis is not a bot...

doutriaux1 · 2015-09-24T05:22:38Z

@durack1 I think the click_info is an artefact of the baselines being updated for the resize merge and that code actually not making it into garant because it is not a bot.

chaosphere2112 · 2015-09-24T16:45:41Z

@doutriaux1 @durack1 Rebased off of master, hopefully we get a clean build this time 😄

durack1 · 2015-09-24T17:15:09Z

@doutriaux1 @chaosphere2112 seems like garant just isn't getting up to date source.. Everything else is passing, I wonder if this issue is the same problem with vcs_test_click_info which keeps occurring, just on the one machine (garant)?

chaosphere2112 · 2015-09-24T17:24:27Z

@durack1 Dunno; I can't seem to get to the cdash page to see the results.

durack1 · 2015-09-24T17:26:55Z

@chaosphere2112 try safari, FF is getting blocked by the lab..

jbeezley · 2015-09-24T17:28:03Z

What are your issues with the cdash page? We have been reconfiguring SSL on all of our servers. If there are specific problems, I can try to work them out with our sysadmins.

durack1 · 2015-09-24T17:31:42Z

@jbeezley it's likely an issue specific to our side (and firewall, and browser - safari works!), @doutriaux1 has an open ticket with the network folks to stop blocking our connections to https://open.cdash.org

chaosphere2112 · 2015-09-24T17:37:30Z

@durack1 @jbeezley I'm unable to get to it from any of my web browsers (or wget).

--2015-09-24 10:34:33--  http://open.cdash.org/
Resolving open.cdash.org... 66.194.253.26
Connecting to open.cdash.org|66.194.253.26|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://open.cdash.org/ [following]
--2015-09-24 10:34:34--  https://open.cdash.org/
Connecting to open.cdash.org|66.194.253.26|:443... connected.
Unable to establish SSL connection.

Odds are solid you guys upgraded to an SSL version that's too new for our firewall (same thing happened to http://djangoproject.com a while back).

Amusingly, it just happened when I tried to connect to https://OpenSSL.org

Here's the openssl s_client errors:

CONNECTED(00000003)
60321:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52.6.4/src/ssl/s23_clnt.c:593:

chaosphere2112 · 2015-09-25T18:49:41Z

Indeed! You don't see me trying to port our build to anything else 😄

durack1 · 2015-09-25T21:12:51Z

@chaosphere2112 want to refresh/rebase this PR again so those tests get re-issued? I think we're close to an all green! The "Update branch" button/link below might just do it for you..

chaosphere2112 · 2015-09-25T21:33:50Z

@durack1 Doing so now; fingers crossed 🎱

doutriaux1 · 2015-09-28T16:42:44Z

@aashish24 @sankhesh I get this:

CMake Warning (dev) at CMake/cdat_modules/ffmpeg_external.cmake:13 (set):
  Policy CMP0053 is not set: Simplify variable reference and escape sequence
  evaluation.  Run "cmake --help-policy CMP0053" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  For input:

    '--disable-yasm^^--enable-gpl^^--enable-libx264^^--extra-cxxflags=@ffmpeg_source@^^--enable-shared^^--enable-zlib'

  the old evaluation rules produce:

    '--disable-yasm^^--enable-gpl^^--enable-libx264^^--extra-cxxflags=/home/doutriaux1/build_x64/build/FFMPEG^^--enable-shared^^--enable-zlib'

  but the new evaluation rules produce:

    '--disable-yasm^^--enable-gpl^^--enable-libx264^^--extra-cxxflags=@ffmpeg_source@^^--enable-shared^^--enable-zlib'

  Using the old result for compatibility since the policy is not set.
Call Stack (most recent call first):
  CMakeLists.txt:671 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

We can fix this in another branch later.

doutriaux1 · 2015-09-28T16:43:57Z

@chaosphere2112 on ubuntu I get:

   Command failed: 1
   '/usr/local/bin/cmake' '-DINSTALL_DIR=/home/doutriaux1/build_x64/install/Externals' '-DWORKING_DIR=/home/doutriaux1/build_x64/build/X264' '-DCONFIGURE_ARGS=--disable-asm;--enable-shared' '-DBASH_CONFIGURE=ON' '-P' '/home/doutriaux1/build_x64/CMake/cdat_configure_step.cmake'
   From /home/doutriaux1/build_x64/X264-prefix/src/X264-stamp/X264-configure-out.log

  -------------------------------------------------

  -------------------------------------------------

   From /home/doutriaux1/build_x64/X264-prefix/src/X264-stamp/X264-configure-err.log

  -------------------------------------------------

  [INFO] ADDITIONAL CFLAGS

  CONFIGURE_ARGS IS --disable-asm;--enable-shared

  LD_ARGS IS
  /home/doutriaux1/build_x64/install/lib:/home/doutriaux1/build_x64/install/Externals/lib:/home/doutriaux1/build_x64/install/Externals/lib64:/home/doutriaux1/build_x64/install/Externals/lib/paraview-.:/lgm/uvcdat/2015-09-16/lib:/lgm/uvcdat/2015-09-16/Externals/lib:


  CFLAGS : -I/home/doutriaux1/build_x64/install/Externals/include
  -I/home/doutriaux1/build_x64/install/Externals/lib/libffi-3.1/include

  bash: configure: No such file or directory

  Config Errors detected:





  CMake Error at
  /home/doutriaux1/build_x64/CMake/cdat_configure_step.cmake:25 (message):

    Error in config

I think it needs ./configure

Will try it now.

chaosphere2112 · 2015-09-28T16:54:30Z

@doutriaux1 I have seen the first warning message; I think all that needs to be done is convert the @ffmpeg_source@ to ${ffmpeg_source}. As for the second... that's weird. I wonder if the PWD isn't getting set up properly?

chaosphere2112 · 2015-09-28T19:57:18Z

@doutriaux1 I'm not having any trouble building libx264 or ffmpeg, but I do run into problems with pip. My ubuntu VM needs a pile of updates, so I'll run those and try again.

chaosphere2112 · 2015-09-28T20:12:47Z

Weird. Mine tried to install pip before setuptools. Worked when I tried again. Building as we speak.

doutriaux1 · 2015-09-28T20:16:21Z

@chaosphere2112 can you please check that pip_deps refers setuptools_pkg

chaosphere2112 · 2015-09-28T20:17:36Z

It does, which is why I found it odd.

doutriaux1 · 2015-09-28T22:25:37Z

@chaosphere2112 configure is not in the source path:
it is at:

/home/doutriaux1/build_x64/build/X264/x264-snapshot-20150921-2245-stable/

one extra directory.

doutriaux1 · 2015-09-28T23:38:10Z

@chaosphere2112 on my ubuntu I have this extra directory. But not on the bots... I have no idea why. @aashish24 @sankhesh could it be a cmake version difference?

chaosphere2112 · 2015-09-28T23:55:13Z

@doutriaux1 libx264 and ffmpeg both built fine on my ubuntu VM (14.04), though I think I'm missing some prereqs that are breaking the rest of the build. Unfortunately, that VM is on the computer I don't have a property pass for, so I'll have to finish debugging it tomorrow.

chaosphere2112 · 2015-09-28T23:55:38Z

(that's before your commit)

doutriaux1 · 2015-09-29T00:00:05Z

@chaosphere2112 don't know what's going on here. It's not crucial for RC1 (but is for 2.4) so if the other 2 PR pass I will tag RC1 tonight. Unless you get a push that fixes the bots. I won't worry about my machine for now if all bots pass your thing.

doutriaux1 · 2015-09-29T08:07:40Z

@chaosphere2112 let's try to fix this early tomorrow morning so we can tag rc1

Integrated libx264 for higher quality mp4 output

chaosphere2112 added Build VCS labels Sep 22, 2015

chaosphere2112 added this to the 2.4 milestone Sep 22, 2015

durack1 reviewed Sep 23, 2015
View reviewed changes

chaosphere2112 force-pushed the add_libx264 branch from 6bfac03 to e24113b Compare September 24, 2015 16:45

durack1 mentioned this pull request Sep 24, 2015

Fix vector plots #1560

Merged

Sam Fries and others added 7 commits September 25, 2015 14:33

Integrated libx264 for higher quality mp4 output

b238942

Fixed small issues, added missing -pix_fmt call to make ffmpeg() work

3e7297c

Detects presence of YASM for FFMPEG/X264 builds

9f8fc0a

Made annoying python build folders go away

74a1224

Switched to bash instead of sh

15d027a

Made bash be an option that's toggleable at the configure step

e933822

Should work now

f77a5c3

chaosphere2112 force-pushed the add_libx264 branch from f58af23 to f77a5c3 Compare September 25, 2015 21:33

got x264 to build again

5ac6d06

Removed extra x264 configure path arg

12aaaa7

doutriaux1 added a commit that referenced this pull request Sep 29, 2015

Merge pull request #1564 from UV-CDAT/add_libx264

b4b469f

Integrated libx264 for higher quality mp4 output

doutriaux1 merged commit b4b469f into master Sep 29, 2015

doutriaux1 deleted the add_libx264 branch September 29, 2015 19:57

chaosphere2112 added the Enhancement label Sep 30, 2015

Integrated libx264 for higher quality mp4 output #1564

Integrated libx264 for higher quality mp4 output #1564

Conversation

chaosphere2112 commented Sep 22, 2015

durack1 commented Sep 22, 2015

doutriaux1 commented Sep 22, 2015

doutriaux1 commented Sep 22, 2015

chaosphere2112 commented Sep 23, 2015

doutriaux1 commented Sep 23, 2015

durack1 commented Sep 23, 2015

chaosphere2112 commented Sep 23, 2015

durack1 Sep 23, 2015

Choose a reason for hiding this comment

chaosphere2112 Sep 23, 2015

Choose a reason for hiding this comment

chaosphere2112 commented Sep 23, 2015

chaosphere2112 commented Sep 23, 2015

aashish24 commented Sep 23, 2015

chaosphere2112 commented Sep 23, 2015

durack1 commented Sep 23, 2015

doutriaux1 commented Sep 24, 2015

durack1 commented Sep 24, 2015

doutriaux1 commented Sep 24, 2015

doutriaux1 commented Sep 24, 2015

doutriaux1 commented Sep 24, 2015

chaosphere2112 commented Sep 24, 2015

durack1 commented Sep 24, 2015

chaosphere2112 commented Sep 24, 2015

durack1 commented Sep 24, 2015

jbeezley commented Sep 24, 2015

durack1 commented Sep 24, 2015

chaosphere2112 commented Sep 24, 2015

chaosphere2112 commented Sep 25, 2015

durack1 commented Sep 25, 2015

chaosphere2112 commented Sep 25, 2015

doutriaux1 commented Sep 28, 2015

doutriaux1 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

doutriaux1 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

doutriaux1 commented Sep 28, 2015

doutriaux1 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

chaosphere2112 commented Sep 28, 2015

doutriaux1 commented Sep 29, 2015

doutriaux1 commented Sep 29, 2015