Older code now fails with h264_v4l2m2m, 'capture' startup seems not to happen.

[Fredrum]

Hello!

I was just returning to a previous project of mine to do some fixes and noticed that the h264 is not decoding any longer.
Usually when something like this happens its because Iv'e touched some code and messed something up. I have spent a fair amount of time doing tests and comparisons and can't see what could have happened.

I am using Rpi4b 64bit Bullseye
(I stupidly updated the OS before starting all this)
Linux raspberrypi 5.15.76-v8+ #1597 SMP PREEMPT Fri Nov 4 12:16:41 GMT 2022 aarch64 GNU/Linux

The program is decoding streaming hevc and h264 video in realtime as it arrives over the network.
Hevc/h265 is working great still. But h264 have stopped decoding.
h264 works on a Ubuntu SD card that I had from a while back. 5 months old or so.
When I copy that source code over to the new RPiOS card and re-compile (without errors) it doesn't work.
On Ubuntu I am running your test/4.3.2/rpi_main version.
On the RPiOS card I have tried both with the default distributions of avcodec etc, and my own build test/4.3.2/rpi_main version that has been working before.
VLC works fine.
As does hello_drmprime playing back a h264 video.

UPDATE:
I just put all my latest code onto that Ubuntu card and h264 works there.
I just can't figure out why it would no longer work on a completely fresh RPiOS card.
End Update

UPDATE2:
I thought maybe it was to do with some new v4l2 kernel thing so found an older RPiOS image from March 2022 and tried with that. But No change in result.
End Update2

I am noticing that when I start the decode the AV Debug text is for me missing a section about 'capture' that is present when I run on Ubuntu and also hello_drmprime. (see attached image)

Does anyone know what the missing 'capture' section might indicate? From 'Dq event 5' and below.

Cheers!

[Fredrum]

I just found that if I rollback the rpi-ffmpeg code to this commit,
2097651

both h264 and h265 works again.
Maybe I can find a newer commit that also works but at least there's a hint here for anyone else in the same situation.

While looking around I also found these recent mentions,
https://trac.ffmpeg.org/ticket/10060

blakeblackshear/frigate#3780

Cheers!

[jc-kynesim]

Have you tried test/4.3.5/rpi_main? 4.3.2 is only (at best) marginally maintained.

[Fredrum]

I think I did try that one too but just to be sure I just now tried it again,

git clone --branch  test/4.3.5/rpi_main https://github.com/jc-kynesim/rpi-ffmpeg.git
cd rpi-ffmpeg

export CPPFLAGS="-I/usr/include/libdrm"
./configure --enable-shared --disable-static --enable-sand --enable-v4l2-request --enable-libdrm --enable-libudev --enable-opengl --enable-epoxy --enable-vout-egl  --enable-vout-drm
make -j3
sudo make install

But unfortunately this didn't change the problem situation.

Recall that I said in my OP that also distro default avcodec etc also behaved the same way for me using latest upgraded RPiOS.

I'll spend more time after work trying to narrow down the commit. Hopefully I can build quicker than doing a full full build as that takes ~15-20 mins.

[jc-kynesim]

I'm going to guess that my "improved" buffer fullness guessing interacts badly with whatever you are doing (it really does improve life for vlc & Kodi).
Given you can clearly run with debug output would you put a debug trace somewhere I can get at please.

[Fredrum]

Sure!
Though I have never 'added a debug trace' before. :)
I'll google it but any small further elaboration of what you'd expect me to do?

[jc-kynesim]

Sorry - I initially thought that your screen-shot was from your app. av_log_set_level(AV_LOG_TRACE); added near the start of your app should do the trick. If you need to go via some sort of custom logging then av_log_set_callback(...) is what you need to set the callback.

[Fredrum]

The screenshot was from the output of my program. (but on the working old Ubuntu environment) I currently have
av_log_set_level(AV_LOG_DEBUG);
just before my ffmpeg init code.
I'll change it to TRACE after work and see what I get.

[jc-kynesim]

I'm not promising anything, but with trace-grade debug I should at least know what I don't know!

[Fredrum]

Hi so here's the output of a run of HoT test/4.3.5/rpi_main, https://pastebin.com/fP9KdfPK

I also did a few test builds at various commits points of the same branch and found that,

Sep29 2022 - 3392699 - FAILS
Sep14 2022 - e6ce03a - WORKS!

[jc-kynesim]

OK... Thats interesting. It is almost certainly is the avcC "fix" that is breaking stuff. (The trace is typical of stream without a good SPS/PPS.) I don't suppose that your app can be simulated by an input file and a command line?
Do you have anything in avctx->extradata at init time & do you update it later? That change will prioritize whatever it sees at init time over whatever might exist at the time the first packet appears - this might plausibly be a mistake.

[jc-kynesim]

I've pushed a new test/4.3.5/rpi_main which fixes an "obvious" bug decoding a stream which doesn't use extradata and has SPS/PPS in the 1st pkt only. Might well fix your problem

[Fredrum]

Oh I started writing the below now see your new message about a new version! :)
Nice one thank you so much I will try it tonight!

I can't control the incoming video stream its from a Playstation console. With that in mind do you think there's anything I could do to make it work with the default RPiOS versions of avcodec? So that I could avoing needing custom ffmpeg builds?

---

I'll take a look in the evening about the avctx->extradata. My program manually initalizes ffmpeg. It does not use the incoming video data srteam for that, it uses just known info to let avcodec initialize, you can read the source here https://github.com/Fredrum/chiaki/blob/0c8f2b730a55f930522cc602bb1afbe3ccfd2057/rpi/src/io.cpp#L445

I could pipe the video data buffers out to a file and try inspect or play that, if you think that would help?

With some hackery I might be able to make it read a regular mp4 file instead of streamed data but I think that would take me a bit longer to figure out.

[jc-kynesim]

If this fix works for you I'll push it to Pi pretty much immediately so I guess it would be in the distribution in a week or two.

If you are feeling strong / helpful I'd appreciate it if you could try the dev/4.3.5/v4l2m2m_pace_2 branch as well. I've played with the poll timeouts to try to work around some ffmpeg stall issues when run from the command-line, but I'm more than a bit worried that I may break other streams / applications in the process. (If test/4.3.5/rpi_main doesn't work for you don't bother with this - it won't help.)

[Fredrum]

Ah Yes!! That works now!! Awesome! :D

Both test/4.3.5/rpi_main and dev/4.3.5/v4l2m2m_pace_2 built and installed as in the earlier post higher up.
Both work great with h264 and hevc.

Fantastic thank you very much for looking into this and making it work for my case! :)

Fred

[jc-kynesim]

I suspect its not just your case - thats a cock-up that will probably affect multiple people but managed to avoid my testing.

[Fredrum]

I must be impossible to move forward with a hyper complex piece of software like this and never break anything. There's sooo many different types of uses and situations and you can't possibly test them all!
Thank you so much again for your help I hope you're getting paid well for all this work! Video is such a key part of Raspberries!