How can I install the cv_reader

[hanzifan]

If I run the OneVision-Encoder/llava_next/examples/training_data_demo/run_training_data_pipeline.sh script, I should install the cv_reader.
However, I can't install the cv_reader

Also, I can't find some information from google.

[YunyaoYan]

Hi! cv_reader is not a pip package on PyPI — it’s our internal package, so pip install cv_reader won’t work.

The easiest way is to follow llava_next/README.md and run via Docker: the image already has cv_reader installed, so you can directly run examples/training_data_demo/run_training_data_pipeline.sh inside the container.

If you really want to install it manually (without Docker), you can run:
cd llava_next/Compressed_Video_Reader && bash install.sh

If you run into any issues, feel free to share your OS/CUDA/Python versions (and the error log) — happy to help 😊

[hanzifan]

Hi! cv_reader is not a pip package on PyPI — it’s our internal package, so pip install cv_reader won’t work.

The easiest way is to follow llava_next/README.md and run via Docker: the image already has cv_reader installed, so you can directly run examples/training_data_demo/run_training_data_pipeline.sh inside the container.

If you really want to install it manually (without Docker), you can run: cd llava_next/Compressed_Video_Reader && bash install.sh

If you run into any issues, feel free to share your OS/CUDA/Python versions (and the error log) — happy to help 😊

Thank you for your prompt reply. I can now use cv_reader normally. However, I have encountered a new issue: I want to read the videos from the LongVideoBench-val dataset using cv_reader and then save their motion vectors as new videos. Unfortunately, I am facing the following problems:
1.The processing time for some videos is excessively long. Even with 64 threads, it takes over a hundred hours to complete the processing.
2.Some videos cannot be read properly, and the following error occurs: [cv_reader] HEVC frame 355: no AV_FRAME_DATA_MOTION_VECTORS side-data

[YunyaoYan]

Thanks for the detailed feedback.

For the two issues you mentioned:

1. Extremely long processing time

Currently, cv_reader reads and decodes the video sequentially frame by frame. Because of this design, increasing the number of threads does not necessarily speed things up significantly. Each worker still needs to go through the decoding pipeline in order, so decoding itself is often the main bottleneck.

In addition, if residual output is enabled, cv_reader will also generate residual data for each decoded frame, which adds extra processing overhead. If you only need motion vectors, you can disable residual output (without_residual=1) to reduce the cost.

Some possible things to try:

• Disable residual output if you only need motion vectors
• Sample frames instead of decoding the entire video if full decoding is unnecessary
• Check whether the bottleneck is in saving/rendering the motion-vector video rather than reading the video itself
• Use a moderate number of workers instead of very high concurrency, since decoding is often limited by CPU and I/O bandwidth

2. [cv_reader] HEVC frame XXX: no AV_FRAME_DATA_MOTION_VECTORS side-data

This warning means that the frame was decoded successfully, but no motion vector side-data was exported for that frame.

This can happen because not every frame in a compressed video actually has motion vectors, for example I-frames. It may also happen for some HEVC frame types where the decoder does not provide MV side-data.

So this message does not necessarily mean that the whole video cannot be read. In practice, such frames can simply be skipped or filled with zero motion vectors if needed.

[hanzifan]

Thanks for the detailed feedback.

For the two issues you mentioned:

1. Extremely long processing time

Currently, cv_reader reads and decodes the video sequentially frame by frame. Because of this design, increasing the number of threads does not necessarily speed things up significantly. Each worker still needs to go through the decoding pipeline in order, so decoding itself is often the main bottleneck.

In addition, if residual output is enabled, cv_reader will also generate residual data for each decoded frame, which adds extra processing overhead. If you only need motion vectors, you can disable residual output (without_residual=1) to reduce the cost.

Some possible things to try:

• Disable residual output if you only need motion vectors
• Sample frames instead of decoding the entire video if full decoding is unnecessary
• Check whether the bottleneck is in saving/rendering the motion-vector video rather than reading the video itself
• Use a moderate number of workers instead of very high concurrency, since decoding is often limited by CPU and I/O bandwidth

2. [cv_reader] HEVC frame XXX: no AV_FRAME_DATA_MOTION_VECTORS side-data

This warning means that the frame was decoded successfully, but no motion vector side-data was exported for that frame.

This can happen because not every frame in a compressed video actually has motion vectors, for example I-frames. It may also happen for some HEVC frame types where the decoder does not provide MV side-data.

So this message does not necessarily mean that the whole video cannot be read. In practice, such frames can simply be skipped or filled with zero motion vectors if needed.

Thank you for your reply. Your solution is very effective, but we still have one unresolved issue. That is, how should I decode the sampled frames? Because I see that the cv_reader API only has two parameters: video_path and with_residual.

[YunyaoYan]

Thanks for the detailed feedback.
For the two issues you mentioned:

1. Extremely long processing time

Currently, cv_reader reads and decodes the video sequentially frame by frame. Because of this design, increasing the number of threads does not necessarily speed things up significantly. Each worker still needs to go through the decoding pipeline in order, so decoding itself is often the main bottleneck.
In addition, if residual output is enabled, cv_reader will also generate residual data for each decoded frame, which adds extra processing overhead. If you only need motion vectors, you can disable residual output (without_residual=1) to reduce the cost.
Some possible things to try:

• Disable residual output if you only need motion vectors
• Sample frames instead of decoding the entire video if full decoding is unnecessary
• Check whether the bottleneck is in saving/rendering the motion-vector video rather than reading the video itself
• Use a moderate number of workers instead of very high concurrency, since decoding is often limited by CPU and I/O bandwidth

2. [cv_reader] HEVC frame XXX: no AV_FRAME_DATA_MOTION_VECTORS side-data

This warning means that the frame was decoded successfully, but no motion vector side-data was exported for that frame.
This can happen because not every frame in a compressed video actually has motion vectors, for example I-frames. It may also happen for some HEVC frame types where the decoder does not provide MV side-data.
So this message does not necessarily mean that the whole video cannot be read. In practice, such frames can simply be skipped or filled with zero motion vectors if needed.

Thank you for your reply. Your solution is very effective, but we still have one unresolved issue. That is, how should I decode the sampled frames? Because I see that the cv_reader API only has two parameters: video_path and with_residual.

Thanks for pointing this out. I think the confusion comes from the Python wrapper example in reader.py.

Currently the example wrapper only exposes the simple read_video() interface:

from . import api

def read_video(video_path: str, with_residual: bool = True):
    """
    Read motion vector and residual from encoded video file
    :param video_path: Path to H.264 video file
    :param with_residual: Whether to read residual from video
    """
    return api.read_video(
        video_path,
        int(not with_residual),  # without residual
    )

So from the Python side it looks like there are only two parameters.

However, the lower-level C++ API also provides a streaming callback interface called read_video_cb, which supports sampled-frame decoding via frame_ids.

You can refer to how we use this API in our preprocessing pipeline. For example, in:

llava_next/Compressed_Video_Reader/tool/offline_precompute_llava_codec_assets.py

we use read_video_cb together with frame_ids to fetch motion vectors and residuals for specific frames.

A simplified example from that file looks like this:

cv_api.read_video_cb(
    str(video_path),
    _cb,
    int(without_residual),
    int(max_frames),
    frame_ids,
    int(seek_to_frame),
    int(max_frames),
)

In this pattern, frame_ids controls which frames are emitted to the callback, and _cb processes each decoded frame.

In summary:
• read_video() is a simple wrapper that sequentially reads frames.
• read_video_cb() is the more flexible interface and supports frame sampling through frame_ids.

We will add a clearer Python wrapper for this in a later update.

[hanzifan]

Thank you for your reply. I have implemented the code as you suggested, but I found that the speed improvement brought by the sampling method I used is rather limited. Then I tried using multithreading and multiprocessing to process the video, but it raised the error: no AV_FRAME_DATA_MOTION_VECTORS side-data, which does not occur in the single-threaded version.

[YunyaoYan]

Thanks for the update.

This issue is usually not caused by sampling itself. More often, it is related to how the decoder is used under concurrency.

A few notes:

1. no AV_FRAME_DATA_MOTION_VECTORS side-data does not necessarily mean decoding failed. It means that the frame was decoded, but no motion-vector side-data was exported for that frame.

2. In HEVC, not every frame is guaranteed to have MV side-data. For example, some I-frames or some frame types may legitimately have no motion vectors.

3. If this warning only appears in the multithreaded / multiprocessing version, the most likely reason is that concurrency is exposing more edge cases, or that decoder states are not fully isolated between workers.

For concurrent usage, each worker should:

• open the video independently
• create its own decoder context independently
• avoid sharing FFmpeg state, frame buffers, or callback state across threads/processes

In practice, multiprocessing is usually safer than multithreading for this callback-based decoding path.

So this warning should generally be treated as a missing-MV frame rather than a fatal decoding error. Such frames can be skipped or filled with zero motion vectors if needed.