crash during extraction

[gvoysey]

I got the following error:

C:\Users\clackn\Desktop\src\whisker\src\ffmpeg_adapt.c(236):------------------]
        packet.pts!=0x8000000000000000i64
        Expression evaluated as false.
C:\Users\clackn\Desktop\src\whisker\src\ffmpeg_adapt.c(346):
        ffmpeg_video_next(v,iframe)>=0
        Expression evaluated as false.
C:\Users\clackn\Desktop\src\whisker\src\video.c(253):
        im=get_[k](self->fp,iframe)
        Expression evaluated as false.
C:\Users\clackn\Desktop\src\whisker\src\whisk.c(59):
        im=video_get(v,index,1)
        Expression evaluated as false.
C:\Users\clackn\Desktop\src\whisker\src\whisk.c(168):
        image=load(movie,i,((void *)0))
        Expression evaluated as false.
*** ERROR: Could not read frame 4650 from left_inverted_1.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000000003DE71A0] multiple edit list entries, a/v desync might occur, patch welcome

it's not clear why the path is C:\Users\clackn* (since I assume that's @nclack's machine).

I am processing video with the following features:

• recorded at 240 fps, 720p.
• h.264 encoded, in a MP4 container
• original video was in color; i've reduced it to 8-bit greyscale and adjusted pixel saturations with a combination of avidemux and ffmpeg.

running whisk as follows:
trace left_inverted_1.mp4 li1.whiskers

After the crash, li1.whiskers is about 80 MB.

Running on windows 10. 256GB ram, multi-core Xeon CPU, Nvidia Quadro M6000.

[tvajtay]

Holy smokes, that's a hell of a machine! Did you use the prebuilt binaries(32/64?) or compiled from source using some form of GCC on windows? Is this what you are seeing in terminal? I'm not familiar with this error. But I usually put the whole path name in the file video to analyze and output file locations.
I would try:
trace C:\Users\<name>\<folder>\left_inverted_1.mp4 C:\Users\<name>\<folder>\li1.whiskers

Now my own personal setup uses the batch.py file included in whisk in order to use parallel processing, 1 movie/core. You only need to point to the directory and define the file type.

C:\Python27\python C:\whisk\python\batch.py C:\Users\<name>\<folder> -e trace -f *.tif

I don't know if that would actually make a difference though.

[gvoysey]

@tvajtay i used the prebuilt binaries. That's my terminal output which dumped about 25% of the way through processing a 600,000 frame video recording.

I'll try to specify a full path -- but given that it was finding frames (nominally) correctly before, it seems strange.

I'm all for parallelism -- have you or @nclack explored exploiting GPUs for this?

[tvajtay]

@gvoysey I know I have received errors in the past if the movie is too big and there isn't enough memory but I would be surprised if that's an issue you would face given your specs. Try running a smaller movie as a proof of concept? My application usually deals with much smaller movies (~3000 frames/movie). If you were able to load parameters and start tracking segments in the frames than it isn't an issue with your command, but probably the size/length of your movie(?) Since the real speed of the parallel processor occurs when many files are being processed simultaneously versus just one big one, maybe break apart the movie and then splice the data together after it's analyzed? That is if ultimately size is the issue.

To be honest I have more a hobbyist association with the program than being a developer. @nclack being the main developer, could give you a much more in depth review of the specifics and possible GPU potential.

[gvoysey]

okay. Splitting is a pretty reasonable thing to try first, even given my specs.

I can run the movie.mp4 example provided on the wiki without any problems.

[tvajtay]

Yeah, if you can run the example movie without issue than the size of the video is most likely your issue.

[nclack]

You can also run into that problem if the timebase for the video gets mucked up.

ffmpeg is set up to be able to represent files with multiple audio and video streams each with their own timing information. When I decode the video, I have to use that timing information to determine a frame index. When that gets messed up the frame index can be out of bounds.

If you look at your mp4 with ffprobe it will give you the timing information. This stackoverflow thread is very useful. Make sure the tbn,tbc and tbr all agree.

[gvoysey]

@nclack it is likely that the frame indexes are confused:

$ ~/Projects/mousetracker/preprocess-video/core/resources/ffmpeg-win64/bin/ffprobe.exe  whisking\ with\ movement\ \(1\).MP4
ffprobe version N-82721-g89092fa Copyright (c) 2007-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 68.100 /  6. 68.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'whisking with movement (1).MP4':
  Metadata:
    major_brand     : avc1
    minor_version   : 0
    compatible_brands: avc1isom
    creation_time   : 1970-01-01T00:33:36.000000Z
  Duration: 00:00:19.42, start: 0.000000, bitrate: 61885 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1280x720, 60493 kb/s, 239.76 fps, 239.76 tbr, 240k tbn, 479.52 tbc (default)
    Metadata:
      creation_time   : 1970-01-01T00:33:36.000000Z
      handler_name    :  XiaoYi AVC
      encoder         : Ambarella AVC encoder

This is raw video from the camera. Subsequent manipulations (reduction to greyscale, etc.) do not change the fact that tbn, tbr, and tbc don't agree.

Do you have any advice in how to rectify those values before I process the files?

@tvajtay : I chunked the video up into 1000-frame slices, but still got the same crash, so i expect it's not a size error. The crash happens at roughly the same percentage way through the file regardless of length, so i currently favor @nclacks theory. (which is good, because chunking stuff up was a hassle)

[gvoysey]

@nclack i have used ffmpeg.exe -i whisking\ with\ movement\ \(1\).MP4 -codec:v mpeg4 -r 240 -qscale:v 2 -codec:a copy test.mp4 , which seems to work.

I'm not familiar with ffmpeg enough to know if that's doing something stupid, though.

[nclack]

@gvoysey Nice!