Copy of video after transpose has corrupted luminance data

[siddharthTyagi]

Hi,
I am trying to make a copy of a video after rotating it using FfmpegFrameFilter "transpose". Somehow, the copy has corrupted luminance data in the copy. I am unable to find a fix for it so far. Any help appreciated.

Regards

Following is the metadata of the input before rotation.
This is a recording from an android device (Pixel3) using front camera of the default camera app.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input-1622418765229.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.29.100 Duration: 00:00:05.25, start: 0.000000, bitrate: 8454 kb/s Stream #0:0(eng): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1280x720, 8353 kb/s, 28.38 fps, 60 tbr, 15360 tbn, 30720 tbc (default) Metadata: rotate : 270 handler_name : VideoHandle Side data: displaymatrix: rotation of 90.00 degrees Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default) Metadata: handler_name : SoundHandler

This is the metadata of the rotated video after using filters - vflip, transpose=cclock
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'rotated.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.45.100 Duration: 00:00:05.25, start: 0.000000, bitrate: 8355 kb/s Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 720x1280, 8254 kb/s, 28.38 fps, 60 tbr, 15360 tbn, 30720 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default) Metadata: handler_name : SoundHandler

This is the screenshot/frame of the video after rotating:

This is the metadata of the video after making a copy:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.45.100 Duration: 00:00:05.25, start: 0.000000, bitrate: 8583 kb/s Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 720x1280, 8482 kb/s, 28.38 fps, 60 tbr, 15360 tbn, 30720 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 94 kb/s (default) Metadata: handler_name : SoundHandler

This is the image/frame of the copied file:

If I use ffmpeg via commandline everything works. But unformtunately I don't have an option to do so at present. But I'm looking at that as a workaround.

Regards

I am using 'org.bytedeco', name: 'javacv-platform', version: '1.5.5'
H264 codec
YUV420P pixel format
the issue is replicated on android platform version as well.

Following is the code that reproduces for me:

`

import org.bytedeco.ffmpeg.global.avcodec;
import org.bytedeco.ffmpeg.global.avutil;
import org.bytedeco.javacv.*;

import java.io.File;
import java.util.Hashtable;
import java.util.Map;

import static org.bytedeco.ffmpeg.global.avcodec.AV_CODEC_ID_H264;
import static org.bytedeco.ffmpeg.global.avutil.AV_LOG_TRACE;
import static org.bytedeco.ffmpeg.global.avutil.av_log_set_level;

public class Main {

public static final String VIDEO_FORMAT_MP4 = "mp4";
public static int PIX_FMT_GRABBER = avutil.AV_PIX_FMT_YUV420P;
public static int PIX_FMT_RECORDER = avutil.AV_PIX_FMT_YUV420P;
static final Map<String, String> RECORDER_FFMPEG_OPTIONS = new Hashtable<>();
public final static CamcorderProfile profile = CamcorderProfile.QUALITY_720P;
static final int PIXEL_FORMAT_RGB = avutil.AV_PIX_FMT_RGB8;

public static FFmpegFrameRecorderWrapper createFrameRecorder(String out, FFmpegFrameGrabber frameGrabber,
                                                             String outputFormat) throws FFmpegFrameRecorder.Exception, FFmpegFrameFilter.Exception {
    return createFrameRecorder(out, frameGrabber, outputFormat, profile);
}

public static FFmpegFrameRecorderWrapper createFrameRecorder(String out, FFmpegFrameGrabber frameGrabber,
                                                             String outputFormat, CamcorderProfile profile)
        throws FFmpegFrameRecorder.Exception, FFmpegFrameFilter.Exception {

    // Output with same Input details
    FFmpegFrameRecorderWrapper frameRecorder = new FFmpegFrameRecorderWrapper(out, frameGrabber, outputFormat, profile);
    return frameRecorder;
}

public static FFmpegFrameGrabber createFrameGrabber(String url, String inputFormat) throws FFmpegFrameGrabber.Exception {
    FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url);
    frameGrabber.setCloseInputStream(true);
    frameGrabber.setFormat(inputFormat);
    frameGrabber.setPixelFormat(PIX_FMT_GRABBER);
    frameGrabber.start();
    return frameGrabber;
}

public static void main(String[] args) {
    try {
        FFmpegLogCallback.set();
        av_log_set_level(AV_LOG_TRACE);
        String input = rotate(args[0]);
        String out = copyAsMp4(input, VIDEO_FORMAT_MP4);
        System.out.println(out);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

private static String rotate(String input) throws FFmpegFrameGrabber.Exception, FFmpegFrameFilter.Exception, FFmpegFrameRecorder.Exception {
    FFmpegFrameGrabber grabber = null;
    String output = "/home/siddharth/Videos/rotated.mp4";
    if (new File(output).exists()) {
        new File(output).delete();
    }
    try {
        grabber = createFrameGrabber(input, VIDEO_FORMAT_MP4);
        // Input details
        FFmpegFrameRecorderWrapper recorder = null;
        // Using a Filter to add a WaterMark
        try {
            recorder = createFrameRecorder(output, grabber, VIDEO_FORMAT_MP4);
            Frame frame;
            FFmpegFrameFilter rotationFilter = null;
            try {
                String rot = grabber.getVideoMetadata().get("rotate");
                if (rot != null) {
                    int rotation = Integer.parseInt(rot);
                    rotation = (rotation + 360) % 360;
                    switch (rotation) {
                        case 90: {
                            rotationFilter = new FFmpegFrameFilter("transpose=clock", grabber.getImageWidth(),
                                    grabber.getImageHeight());
                        }
                        break;
                        case 180:
                            rotationFilter = new FFmpegFrameFilter("rotate=PI", grabber.getImageWidth(),
                                    grabber.getImageHeight());
                            break;
                        case 270: {
                            rotationFilter = new FFmpegFrameFilter(
                                    "[in]vflip [flipped];[flipped]transpose=cclock [out]",
                                    grabber.getImageWidth(),
                                    grabber.getImageHeight());
                        }
                        break;
                        default:
                            throw new FFmpegFrameFilter.Exception("Unsupported rotation!" + rotation);
                    }
                    rotationFilter.setPixelFormat(PIX_FMT_GRABBER);
                    rotationFilter.start();
                }
                while ((frame = grabber.grab()) != null) {
                    Long fts = grabber.getTimestamp();
                    recorder.setTimestamp(fts);
                    if (rotationFilter != null) {
                        rotationFilter.push(frame, grabber.getPixelFormat());
                        Frame filtered = null;
                        while ((filtered = rotationFilter.pull()) != null) {
                            recorder.record(filtered, rotationFilter.getPixelFormat());
                        }
                    } else {
                        recorder.record(frame, grabber.getPixelFormat());
                    }
                }
            } finally {
                if (rotationFilter != null) {
                    rotationFilter.stop();
                    rotationFilter.release();
                }
            }
        } finally {
            if (recorder != null) {
                // Stops the recorder
                recorder.stop();
                recorder.release();
            }
        }
    } finally {
        if (grabber != null) {
            grabber.stop();
            grabber.release();
        }
    }
    return output;
}

private static String copyAsMp4(String input, String format) throws Exception {
    String output = "/home/siddharth/Videos/out.mp4";
    if (new File(output).exists()) {
        new File(output).delete();
    }
    copyUsingJavacv(input, format, output);
    return output;
}

public static void copyUsingJavacv(String input, String format,
                                   String output) throws Exception {
    FFmpegFrameGrabber grabber = null;
    try {
        grabber = createFrameGrabber(input, format);
        // Output with same Input details
        FFmpegFrameRecorderWrapper recorder = null;
        try {
            // final output is MP4
            recorder = createFrameRecorder(output, grabber,
                    VIDEO_FORMAT_MP4);
            Frame frame;
            while ((frame = grabber.grab()) != null) {
                recorder.setTimestamp(grabber.getTimestamp());
                recorder.record(frame, grabber.getPixelFormat());
            }
        } finally {
            if (recorder != null) {
                // Stops the recorder
                recorder.stop();
                recorder.release();
            }
        }
    } finally {
        if (grabber != null) {
            grabber.stop();
            grabber.release();
        }
    }
}

public static class FFmpegFrameRecorderWrapper extends FFmpegFrameRecorder {
    private final CamcorderProfile profile;

    public FFmpegFrameRecorderWrapper(String out, FFmpegFrameGrabber grabber,
                                      String outputFormat, CamcorderProfile profile)
            throws FFmpegFrameRecorder.Exception, FFmpegFrameFilter.Exception {
        // we rotate video in all cases to be portrait
        super(out, profile.videoFrameHeight, profile.videoFrameWidth, grabber.getAudioChannels());
        this.profile = profile;
        configureRecorder(grabber, outputFormat);
        // Does de read / write
        start();
    }

    private void configureRecorder(FFmpegFrameGrabber frameGrabber, String outputFormat) throws FFmpegFrameRecorder.Exception, FFmpegFrameFilter.Exception {
        setCloseOutputStream(true);
        // Round down the FrameRate as passing a floating point does not work
        int frameRate = (int) (profile == null ? frameGrabber.getFrameRate() : profile.videoFrameRate);
        setFrameRate(frameRate);
        // Matching input with output
        setSampleRate(profile == null ? frameGrabber.getSampleRate() : profile.audioSampleRate);
        setAudioBitrate(profile == null ? frameGrabber.getAudioBitrate() : profile.audioBitRate);
        setVideoBitrate(profile == null ? frameGrabber.getVideoBitrate() : profile.videoBitRate);
        // Compress with YUV420P / H264 / AAC generating a MP4
        setFormat(outputFormat);
        setPixelFormat(PIX_FMT_RECORDER);
        setVideoCodec(AV_CODEC_ID_H264);
        setAudioCodec(avcodec.AV_CODEC_ID_AAC);
        setOptions(RECORDER_FFMPEG_OPTIONS);

        // grabber metadata
        // do not copy metadata instead apply filter on input to process frames as per metadata
        // recorder outputs to CDN which may strip metadata
        // frameRecorder.setVideoMetadata(frameGrabber.getVideoMetadata());

        // recorder config
        //        recorder.setInterleaved(true);
        // decrease "startup" latency in FFMPEG (see:
        // https://trac.ffmpeg.org/wiki/StreamingGuide)
        //        recorder.setVideoOption("tune", "zerolatency");
        // Constant Rate Factor (see: https://trac.ffmpeg.org/wiki/Encode/H.264)
        setVideoOption("crf", "18");
        // We don't want variable bitrate audio
        setAudioOption("crf", "0");
        // Highest quality
        setAudioQuality(0);
    }

}


public static class CamcorderProfile {
    public static final CamcorderProfile QUALITY_720P = new CamcorderProfile();
    public int videoFrameWidth = 1280;
    public int videoFrameHeight = 720;
    public int videoFrameRate = 60;
    public int audioSampleRate = 48000;
    public int audioBitRate = 96000;
    public int videoBitRate = 18000000;
}

}

`

[saudet]

You'll need to call FFmpegFrameRecorder.record() with the right pixel format. FFmpegFrameFilter.getPixelFormat() might not be equal to PIX_FMT_RECORDER.

[saudet]

BTW, we can easily use the ffmpeg program from Java if this does everything you need:
http://bytedeco.org/javacpp-presets/ffmpeg/apidocs/org/bytedeco/ffmpeg/ffmpeg.html

[saudet]

Do the same with FFmpegFrameFilter.push() and FFmpegFrameGrabber.getPixelFormat() to be sure.

[siddharthTyagi]

Updated the logic for filter.push and recorder.record (while copying) to use grabber pixel format. But the issue is still there.
Using ffmpeg directly is not the option that is available to me. I need to run the code inside an android app. I am including the native objects from the build inside as per the android packaging specifications. But the binary executables are stripped off while packaging.
The path to ffmpeg as in javacv helloworld example is giving me null. (proguard exclusion is set for javacv)
And I can only use ffmpeg to rotate the video. The copying has to be done using javacv as per the requirements.

[saudet]

static final int PIXEL_FORMAT_RGB = avutil.AV_PIX_FMT_RGB8;

This doesn't look right. You sure what you want isn't AV_PIX_FMT_RGB24?

[saudet]

Given the lack of feedback, I'm assuming that this was the bug, so I'll mark this as the answer.