Model Zoo with longer video crashes

[tomgreen66]

Is there an existing issue for this?

• I have searched the existing issues

Bug description

When running topviewmouse with a 15 minute video (resolution 1616 × 1077, H.264) and it crashes with error. Using a shorter video it seems to run (by trimming in the GUI).

Operating System

CentOS 7 Linux, but using container with Ubuntu 20.04.

DeepLabCut version

2.3.5

DeepLabCut mode

single animal

Device type

Tesla V100 16GB

Steps To Reproduce

1. Select Model Zoo from GUI
2. Use video of mouse
3. Crashes with error

Relevant log output

2023-08-20 00:44:12.221784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14626 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:3b:00.0, compute capability: 7.0

/opt/deeplabcut/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  warnings.warn('`layer.apply` is deprecated and '
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 83, in load_and_enqueue
    batch_np = dataset.next_batch()
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/deeplabcut/pose_estimation_tensorflow/datasets/pose_multianimal_imgaug.py", line 520, in next_batch
    batch_images, batch_joints = self.pipeline(
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/imgaug/augmenters/meta.py", line 2008, in __call__
    return self.augment(*args, **kwargs)
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/imgaug/augmenters/meta.py", line 1979, in augment
    batch_aug = self.augment_batch_(batch, hooks=hooks)
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/imgaug/augmenters/meta.py", line 596, in augment_batch_
    batch_norm = batch.to_normalized_batch()
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/imgaug/augmentables/batches.py", line 196, in to_normalized_batch
    images_unaug = nlib.normalize_images(self.images_unaug)
  File "/opt/deeplabcut/venv/lib/python3.8/site-packages/imgaug/augmentables/normalization.py", line 142, in normalize_images
    assert image.ndim in [2, 3], (
AttributeError: 'NoneType' object has no attribute 'ndim'

Anything else?

I have been unsure whether to report this - it seems very specific but advice whether the video format could produce this.

Code of Conduct

• I agree to follow this project's Code of Conduct

[n-poulsen]

Hi @tomgreen66, thanks for reporting this issue! I believe this is due to a bug in our code, but would like to make sure that's the case.

Does this issue happen towards the end, when the video is almost completely analyzed, or in the beginning?

[tomgreen66]

Since the trimmed version worked using the option in the GUI to trim to 30 seconds I just tried converting the whole video:

ffmpeg -i input.mov -vcodec h264 -acodec mp2 test.mp4

And it worked with the test.mp4 file. So it must be something to do with the video processing with the input.mov. It seems to create plot-poses and a h5 and pickle file but then crashes with that error. I can provide more details if it will help but was hitting a wall trying to understand the issue. The video file that has issues shows the following information in QuickTime on my Mac.

[n-poulsen]

I believe the issue was caused by the number of frames in the video being slightly off from what's written in the metadata (which can happen, is not an issue and we should deal with).

This is why I think re-encoding the video fixed the issue (the number of frames written in the metadata was corrected, so no more issues on our side).

I'll work on a fix

[MMathisLab]

@n-poulsen is a fix implemented?

[n-poulsen]

I've worked on it but was not able to replicate the issue with one of my own corrupt videos. I've worked on this again this morning, but again cannot replicate the issue.

I think we can close this issue as it was resolved by re-encoding the video.

[tomgreen66]

Just wanted to comment - seems it might be due to opencv/opencv/issues/24000 where FPS changed definition. Just trying to install with opencv-python-headless<3.8 since 3.8 seems to be the version that the change occurred. Trying to check.