tc358743 - incomplete first frame

[JakubVanek]

Describe the bug
First frame received by V4L2 applications from the tc358743 HDMI-to-CSI chip is partially corrupted. It looks as if the application starts receiving the frame as soon as it requests it, but this means that the first frame will be incomplete (the top part will be missing). This causes problems when recording short video clips, as it's not possible to skip the first frame easily in some frameworks.

EDIT: for GStreamer workaround, see #4058 (comment)

To reproduce

• install components of GStreamer 1.0
• enable dtoverlay=tc358743 in /boot/config.txt to hand the control of the CSI2 peripheral and the HDMI-to-CSI bridge to the Linux kernel
• shutdown Pi
• connect Auvidea B101 to Raspberry using the CSI2 interface
• boot Pi
• send an EDID binary to the HDMI bridge: v4l2-ctl -d /dev/video0 --set-edid file=/etc/edid.bin,format=raw
• feed a signal to the HDMI input of the bridge
• sync timings between HDMI and CSI: v4l2-ctl --set-dv-bt-timings query
• run

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=2 ! jpegenc ! multifilesink location=test%d.jpg

test0.jpg will contain the first, corrupted frame, test1.jpg will contain the second, valid frame.

Expected behaviour

Only complete frame data is sent to the application. - test0.jpg should be the same as test1.jpg.

Actual behaviour

First frame (test0.jpg):

Second frame (test1.jpg):

System
Pastebin link to raspinfo output: https://pastebin.com/fb8gHFJT

[6by9]

AFAIK that's just the way the tc358743 works. It starts sending data over the CSI2 link as soon as it is requested to, even if it is half way through receiving a frame in over the HDMI.

There is the g_skip_frames function within the V4L2 sensor subdev API (https://elixir.bootlin.com/linux/latest/source/include/media/v4l2-subdev.h#L493), but the tc358743 driver doesn't declare that any frames need to be skipped. Seeing as it's a mainline driver that should be fixed there. (If it is then I would look at adding the other half of the functionality to the unicam driver).

[6by9]

And green is just 0,0,0 in YUV space, so the buffer has been memset to 0 by some part of the initialisation.

Memory says that there is a way to get GStreamer to drop the first N frames in a capture, but I don't recall the syntax at present.

[JakubVanek]

Thank you for your quick reply. I tried to find something on Google regarding frame skipping using gst-launch-1.0, but I haven't found a simple solution yet. It seems that creating a small custom GStreamer element for this or driving the valve element from a wrapper program would fix the problem, but both approaches would make the solution a bit more complex.

Should I submit a patch for g_skip_frames to the latest mainline Linux release? Judging by how other sensor drivers seem to handle this, the change should be relatively straightforward (adding v4l2_subdev_sensor_ops to the current tc358743 ops).

[JakubVanek]

I have a candidate for the tc358743 patch here: JakubVanek@ca70569 for rpi kernel / JakubVanek@dace8aa for mainline. However, I'm not sure how to test this. I tried to build the 5.4 RPi kernel and it runs successfully on Pi4, but indeed, the unicam driver doesn't know how to skip frames yet. Would it be acceptable to send the patch as-is?

[6by9]

First step would be to ask the maintainer (Mats Randgaard) and linux-media whether they also see the same behaviour on their hardware. I don't believe it i is anything funny on the Pi CSI2 receiver, but possibly.

The CSI2 receiver does have a write pointer register which could be used to flag short frames as errors, but that makes life more awkward for any weird (non-standard) formats that are deliberately variable length (eg JPEG or H264). It does rely on the client throwing away returned buffers marked as errors, which isn't guaranteed.
It's also a slightly tricky one as it is the write pointer for what has actually hit memory, but the end of frame interrupt is triggered at the start of the receiver pipeline, so checking it in the interrupt handler will imply a slightly short buffer.

A quick query on the GStreamer mailing list to query whether there is a way to skip the first N frames may give you a quicker answer. It feels like there should be a way, but it may not be available via gst-launch.

[JakubVanek]

Thank you for the pointers, I will try to visit the mailing list in the coming days if I fail to solve this using alternative approaches.

I haven't experienced problems with truncated frames yet except for the first frame. Wouldn't it be possible to write the first few frames into a dummy buffer? However, other kernel drivers querying g_skip_frames() seem to configure the CSI2 hardware directly with the number of frames to skip, so this software approach may not be valid. I'm also not sure how to handle metadata buffers with this.

The quickest path to having a workaround may indeed be hacking something together with GStreamer. I will try to create a wrapper script for the pipeline - that should go relatively well, as I already had to use gst-python somewhere else.

[6by9]

I've just read the TC358743 datasheet. It has a section entitled "CSI-2 Tx One Frame Operation" which implies that after enabling CSI2-TX and HDMI-RX the chip waits for the VSYNC.

Now the trickier bit is that presumably the HDMI-RX is probably already enabled as you want the system to be able to detect the incoming video format. It'll take a bit more reading to check whether actually it is the enabling of the CSI-2 block that makes it wait for the VSYNC, or have you got part of the data being written into the internal FIFO ahead of time. The datasheet may not even go into that sort of detail.

[JakubVanek]

The following workaround seems to work with GStreamer. It turns out that GStreamer pad probes are quite powerful and can drop buffers on their own. (docs)

GStreamer wrapper script in Python

#!/usr/bin/env python3
import signal
import sys

import gi

gi.require_version('Gst', '1.0')
gi.require_version('GLib', '2.0')
from gi.repository import Gst, GLib


def main():
    """ GStreamer pipeline that can drop first N frames. """

    # setup external libs
    Gst.init(None)
    mainloop = GLib.MainLoop()

    # setup pipeline
    pipeline = make_pipeline()
    bus: Gst.Bus = pipeline.get_bus()
    bus.add_signal_watch()
    bus.connect("message", message_callback, mainloop)

    # run
    pipeline.set_state(Gst.State.PLAYING)

    # handle signals
    def on_signal(_sig, _stack):
        print("received signal, exiting early")
        pipeline.send_event(Gst.Event.new_eos())

    signal.signal(signal.SIGINT, on_signal)
    signal.signal(signal.SIGTERM, on_signal)
    mainloop.run()
    # final cleanup
    pipeline.set_state(Gst.State.NULL)


def make_pipeline():
    # top-level element
    pipeline: Gst.Pipeline = Gst.Pipeline.new()

    # insert your pipeline here
    # JPEGs are a trivial example that don't really need this, but more advanced video encoding might benefit from this,
    dataflow: Gst.Bin = Gst.parse_bin_from_description(f"""
        v4l2src device=/dev/video0 num-buffers=3 name=input
        ! jpegenc
        ! multifilesink location=image%d.jpg
    """, True)
    pipeline.add(dataflow)

    # register a pad probe that will drop first N frames
    src: Gst.Element = dataflow.get_by_name("input")
    srcpad: Gst.Pad = src.get_static_pad("src")
    srcpad.add_probe(Gst.PadProbeType.BUFFER, gen_frameskip(1))
    return pipeline


def gen_frameskip(frameskip):
    remaining = frameskip

    def handler(_pad: Gst.Pad, _info: Gst.PadProbeInfo):
        nonlocal remaining

        if remaining == 0:
            return Gst.PadProbeReturn.REMOVE
        else:
            remaining -= 1
            return Gst.PadProbeReturn.DROP

    return handler


def message_callback(_bus: Gst.Bus, msg: Gst.Message, mainloop: GLib.MainLoop):
    if msg.type == Gst.MessageType.EOS:
        mainloop.quit()
    elif msg.type == Gst.MessageType.ERROR:
        err, debug = msg.parse_error()
        print(f"gst error: {err}: {debug}", file=sys.stderr)
        mainloop.quit()
    return True


if __name__ == '__main__':
    main()

It could still be interesting to know why only part of the first frame arrives though.