Deprioritize HEVC-over-FLV formats

[Lesmiscore]

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Looks like BiliBili is offering HEVC-over-FLV formats, which isn't in standards and even not supported by ffmpeg. (How do browsers play? There's a library to demux for them)

Although they could have been skipped, I think they should be available for download for anyone interested in.

ref. https://trac.ffmpeg.org/ticket/6389
ref. #5821

... wait, should we implement some scheme to deprioritize such formats (i.e. impossible or out-of-spec combination of container/codecs) globally? @pukkandan

Template

Before submitting a pull request make sure you have:

• At least skimmed through contributing guidelines including yt-dlp coding conventions
• Searched the bugtracker for similar pull requests
• Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

#5821

What is the purpose of your pull request?

• Fix or improvement to an extractor (Make sure to add/update tests)
• New extractor (Piracy websites will not be accepted)
• Core bug fix/improvement
• New feature (It is strongly recommended to open an issue first)

[pukkandan]

should we implement some scheme to deprioritize such formats (i.e. impossible or out-of-spec combination of container/codecs) for all the formats?

You mean in core code? We could add this logic here

yt-dlp/yt_dlp/utils.py

Lines 6307 to 6309 in c733555

	# if format.get('preference') is None and format.get('ext') in ('f4f', 'f4m'): # Not supported?
	# format['preference'] = -1000

[Lesmiscore]

@pukkandan Maybe close this too and turn this PR to add the logic there?

[pukkandan]

Sounds good

[Vangelis66]

HEVC-over-FLV formats, which isn't in standards and even not supported by ffmpeg.

FWIW, there's (open-source?) code in GitHub (from Oct 2020) which implements exactly that (HEVC inside FLV container):
https://github.com/numberwolf/FFmpeg-QuQi-H265-FLV-RTMP

Another dev patched a slightly newer ffmpeg version from Nov 2020, see here; hopefully, current ffmpeg 5.x.x could be patched (and compiled?) in a similar fashion 😜 ...

Source: SO

Later addition:
More recent patches & compilation guide here 😄 ...

[Lesmiscore]

https://trac.ffmpeg.org/ticket/6389#comment:1
They don't seem to support it in master, although it doesn't stop us from patching ffmpeg to allow HEVC-over-FLV

[Lesmiscore]

yt-dlp/FFmpeg-Builds@6561554

[nihil-admirari]

TLDR:

1. A patch was added to FFmpeg-Builds.

2. GMP site's certificate has expired, and the owners are aware of it:

   YES, THE CERTIFICATE IS EXPIRED. Sorry about that. We're working on the renewal process.

   Builds are currently failing. I've managed to build a patched FFmpeg before the certificate expired, but it's only for Linux.

3. It is completely untested, since I don't have premium access to BiliBili, and I only get

   > yt-dlp https://www.bilibili.com/video/BV1Dx411n7Pg?p=32
   ...
   [BiliBili] Format(s) 1080P 高清, 360P 流畅, 480P 清晰, 720P 高清 are missing; you have to login or become premium member to download them
   

4. Support for VP8, VP9, AV1, and OPUS was omitted because:
   a. It wasn't requested.
   b. It causes too many questions.

5. yt-dlp should probably warn users and remux HEVC-over-FLC upon encountering, to prevent playback issues.

---

Gory details:

Another dev patched a slightly newer ffmpeg version from Nov 2020, see here; hopefully, current ffmpeg 5.x.x could be patched (and compiled?) in a similar fashion ...

Source: SO

Later addition:
More recent patches & compilation guide here ...

There is no shortage of such patches out there; one was even submitted to FFmpeg. The problem is that these patches are all slightly different, and I'm not knowledgeable of codec internals to sort out the differences.

1. The patch from FFmpeg mailing list assigns codec ID of 10 to HEVC, 11 to VP8, and 12 to VP9:

   FLV_CODECID_H265    = 10,
   FLV_CODECID_VP8     = 11,
   FLV_CODECID_VP9     = 12,

   @runner365 uses a totally different assignment (@xiaoxial patches match @runner365's HEVC and AV1):

   FLV_CODECID_HEVC    = 12,
   FLV_CODECID_AV1     = 13,
   FLV_CODECID_VP8     = 14,
   FLV_CODECID_VP9     = 15,

   I've selected the value of 12 since that's what mpegts.js expects, but please be aware that someone out there streams VP9 over FLV using the codec ID of 12.

2. FFmpeg mailing list patch parses AVSTREAM HEADERS:

   case FLV_CODECID_H265:
       par->codec_id = AV_CODEC_ID_H265;
       vstream->internal->need_parsing = AVSTREAM_PARSE_HEADERS;
       ret = 3;     // not 4, reading packet type will consume one byte
       break;

   @runner365 and @xiaoxial do not:

   case FLV_CODECID_HEVC:
        par->codec_id = AV_CODEC_ID_HEVC;
        vstreami->need_parsing = AVSTREAM_PARSE_NONE;
        ret = 3;     // not 4, reading packet type will consume one byte
        break;

   (AV_CODEC_ID_H265 and AV_CODEC_ID_HEVC are synonyms).

   @runner365 repeats the same block for VP8 and VP9, but mailing list patch does not. For VP8 and VP9, one byte is not consumed and need_parsing isn't touched at all:

    case FLV_CODECID_VP8:
        par->codec_id = AV_CODEC_ID_VP8;
        break;
    case FLV_CODECID_VP9:
        par->codec_id = AV_CODEC_ID_VP9;
        break;

   I can't help but think that @runner365 simply copy-pasted the same block three times. That's why I've chosen to parse AVSTREAM HEADERS, but I may be wrong. Since the code is untested, a switch back to AVSTREAM_PARSE_NONE may end up being necessary.

3. OPUS supports only 48 kHz sample rates, which, unfortunately, is not reflected in @runner365's flvenc.c

   else if (par->codec_id == AV_CODEC_ID_OPUS) {
       return FLV_CODECID_OPUS | FLV_SAMPLERATE_44100HZ |
              FLV_SAMPLESSIZE_16BIT | FLV_STEREO;
   }

4. @xiaoxial takes care to write a different HEVC end-of-stream:

   if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4) {
       put_avc_eos_tag(pb, sc->last_ts);
   } else if (par->codec_id == AV_CODEC_ID_HEVC) {
       put_hevc_eos_tag(pb, sc->last_ts);
   }

   @runner365 does it neither for HEVC nor for VP8/9.

5. @xiaoxial, however, writes codec header to every frame: bug report on CDN/FFmpeg_patches/0001-Add-SVT-HEVC-FLV-support-on-FFmpeg.patch VCDP/CDN#1.

6. @runner365 disallows missing PTS, @xiaoxial does not:

   if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4
       || par->codec_id == AV_CODEC_ID_H265 || par->codec_id == AV_CODEC_ID_VP8
       || par->codec_id == AV_CODEC_ID_VP9) {
         if (pkt->pts == AV_NOPTS_VALUE) {
             av_log(s, AV_LOG_ERROR, "Packet is missing PTS\n");
             return AVERROR(EINVAL);

   The check, however, was originally added to prevent overflow in AVC NALU:

   else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4
            || par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_VP8
            || par->codec_id == AV_CODEC_ID_VP9) {
       avio_w8(pb, 1); // AVC NALU
       avio_wb24(pb, pkt->pts - pkt->dts);
   }

   If I'm not mistaken, NALU stands for Network Abstraction Layer unit, which

   ... is a part of the H.264/AVC and HEVC video coding standards.

   Its applicability to VP8/VP9 is not at all obvious.

[pukkandan]

• yt-dlp should probably warn users and remux HEVC-over-FLC upon encountering, to prevent playback issues.

Remuxing without being requested may not be a good idea. I think we should just leave the format deprioritized. Opinion, @Lesmiscore ?

[Lesmiscore]

Remuxing without being requested may not be a good idea. I think we should just leave the format deprioritized. Opinion, @Lesmiscore ?

Agreed since we can't expect upstream to apply the patch

[nihil-admirari]

I've disabled SSL for GMP. Builds are operational once again.

Having no access to real world HEVC-over-FLV, i've downloaded an example from OSSRS, and recoded it as per instructions:

ffmpeg -re -i source.200kbps.768x320.flv -acodec copy -vcodec libx265 -f flv test.flv

Patched FFmpeg is capable of probing and playing the resulting file:

Input #0, flv, from 'test.flv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.34.102
  Duration: 00:03:30.69, start: 0.040000, bitrate: 273 kb/s
  Stream #0:0: Video: hevc (Main), yuv420p(tv), 768x320 [SAR 1:1 DAR 12:5], 25 fps, 25 tbr, 1k tbn
  Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 30 kb/s

Non-patched FFmpeg throws:

[flv @ 0x7f612c000c80] Video codec (c) is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[flv @ 0x7f612c000c80] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[flv @ 0x7f612c000c80] Could not find codec parameters for stream 0 (Video: none ([12][0][0][0] / 0x000C), none): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, flv, from 'test.flv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.34.102
  Duration: 00:03:30.69, start: 0.000000, bitrate: 273 kb/s
  Stream #0:0: Video: none ([12][0][0][0] / 0x000C), none, 25 fps, 25 tbr, 1k tbn
  Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 30 kb/s
No decoder could be found for codec none

Nonetheless, testing on real-world examples would be greatly appreciated.

Remuxing without being requested may not be a good idea.

At least a warning should be issued, when this format is selected, along the lines of “selected format is non-standard and unsupported by most player. Consider using --remux-video.”

[pukkandan]

Remuxing without being requested may not be a good idea.

At least a warning should be issued, when this format is selected, along the lines of “selected format is non-standard and unsupported by most player. Consider using --remux-video.”

We could add a note to format_note

[Lesmiscore]

Nonetheless, testing on real-world examples would be greatly appreciated.

Looks okay to me (-f source-1 --downloader ffmpeg https://live.bilibili.com/6)

edit: I think I found an unrelated bug Fixed. Hope we have a new release soon

$ ytdl  -f source-1 --downloader ffmpeg https://live.bilibili.com/6 -O is_live=
is_live = "NA"