[nebula] change to new API URL, fix up tests

[rohieb]

Fixes #7017

Nebula changed to a new API URL. Original patch was by @Lamieur on issue #7017, I've fixed up the tests, however I'm still getting errors that I cannot explain, because the .mp4 I'm getting when calling yt-dlp on the URL has a reasonable size and plays just fine.
(Redacted some of the URLs because Nebula needs a username and password.)

$ python test/test_download.py TestDownload.test_Nebula
[debug] Loaded 1823 extractors
[Nebula] Logging in to Nebula with supplied credentials
[Nebula] Authorizing to Nebula
[Nebula] Extracting URL: https://nebula.tv/videos/that-time-disney-remade-beauty-and-the-beast
[Nebula] that-time-disney-remade-beauty-and-the-beast: Fetching video meta data
[Nebula] that-time-disney-remade-beauty-and-the-beast: Fetching video stream info
[Nebula] that-time-disney-remade-beauty-and-the-beast: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] 84ed544d-4afd-4723-8cd5-2b95261f0abf: Downloading 1 format(s): 2508
[info] Writing video metadata as JSON to: test_Nebula_84ed544d-4afd-4723-8cd5-2b95261f0abf.info.json
[debug] Invoking hlsnative downloader on "https://media-production.nebula.app/[…some long ID…]/nebula-content/videos/lindsayellis/that-time-disney-remade-beauty-and-the-beast/video_upload:5603accc-90e2-49ea-b39f-be4fbe6e4831/video/avc1/1/media.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 369
[download] Destination: test_Nebula_84ed544d-4afd-4723-8cd5-2b95261f0abf.mp4
[download] 100% of    756.00B in 00:00:00 at 7.71KiB/s
F
======================================================================
FAIL: test_Nebula (__main__.TestDownload):
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rohieb/projects/yt-dlp/test/test_download.py", line 228, in test_template
    assertGreaterEqual(
  File "/home/rohieb/projects/yt-dlp/test/helper.py", line 293, in assertGreaterEqual
    self.assertTrue(got >= expected, msg)
AssertionError: False is not true : Expected test_Nebula_84ed544d-4afd-4723-8cd5-2b95261f0abf.mp4 to be at least 9.77KiB, but it's only 756.00B 

----------------------------------------------------------------------
Ran 1 test in 4.961s

FAILED (failures=1)

Description of your pull request and other information

Template

Before submitting a pull request make sure you have:

• At least skimmed through contributing guidelines including yt-dlp coding conventions
• Searched the bugtracker for similar pull requests
• Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)
• I am not the original author of this code and I would like @Lamieur to release his three-line patch under Unlicense

What is the purpose of your pull request?

• Fix or improvement to an extractor (Make sure to add/update tests)
• New extractor (Piracy websites will not be accepted)
• Core bug fix/improvement
• New feature (It is strongly recommended to open an issue first)

Copilot Summary

🤖 Generated by Copilot at 36ce55c

Summary

🌐🔢📚

Update the Nebula extractor to support the latest API and content changes. Fix the test cases for nebula.py.

Oh, we're the crew of the NebulaIE
And we sail the web for videos to see
We've changed our domain and our ID
So haul away and update the test cases with me

Walkthrough

• Update Nebula API domain to content.api.nebula.app (link, link)
• Use id field instead of zype_id field for video ID (link, link, link, link, link)
• Update test case metadata to match new Nebula content structure (link)

[delroth]

Tested this patch applied on top of latest stable (2023.03.04), Python 3.10, NixOS. Confirmed that nebula.tv downloads work again.

[bashonly]

It looks like the zype_id (old video ID value) is still available for some videos in the _fetch_video_metadata response.
For example, https://nebula.tv/videos/tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines (JSON truncated down to the relevant parts):

{
  "id": "video_episode:e389af9d-1dab-44f2-8788-ee24deb7ff0d",
  "type": "video_episode",
  "slug": "tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines",
  "title": "Did the US Really Blow Up the NordStream Pipelines?",
  "zype_id": "63f64c74366fcd00017c1513"
}

We could try to (non-fatally) extract zype_id and return it in the info_dict as

'_old_archive_ids': [make_archive_id(NebulaIE, zype_id)],

[rohieb]

@bashonly Like that?

[bashonly]

please avoid force-pushing in the future

[rohieb]

please avoid force-pushing in the future

Hmm okay, different projects seem to have different workflows in that regard. But thanks for fixing the issues, @bashonly!

[matejdro]

Tested out the PR and it appears that downloading just the audio is buggy.

Following command:

yt-dlp --netrc -f "ba" "https://nebula.tv/videos/tldrnewseu-why-has-the-rightwing-afd-become-so-popular-in-germany"

Will download the audio fine, but it will keep it named as .m3u8 which is wrong (it should be .m4a or whatever audio format video has).

Not sure if this is out of scope, just wanted to mention it. I can create a new issue if desired.

[bashonly]

@matejdro the master m3u8 for that video does not have any codec information for the audio track, and there is no default extension set in the arguments for _extract_m3u8_formats_and_subtitles

I think this is the best we could do, which will result in an .mp4 download instead of .m3u8:

     def _fetch_video_formats(self, slug):
        stream_info = self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/stream/',
                                             video_id=slug,
                                             auth_type='bearer',
                                             note='Fetching video stream info')
         manifest_url = stream_info['manifest']
-        return self._extract_m3u8_formats_and_subtitles(manifest_url, slug)
+        return self._extract_m3u8_formats_and_subtitles(manifest_url, slug, 'mp4')