[Nebula] Extractor failed to obtain "id"

[hutzisgone]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm reporting that a supported site is broken
• I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

UK

Provide a description that is worded well enough to be understood

I can still download older videos from Nebula, but recent videos from https://nebula.tv/videos/nothingbuttech-i-cloned-my-voice-to-expose-ai-drake on give an 'extractor failed to obtain "id"' error.

Please let me know if you require further information/login details.

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', '-f', 'bestvideo[height<=1440]+bestaudio/best[height<=1440]', '-o', 'H:/Videos/%(title)s-%(height)s.%(ext)s', '--embed-metadata', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.tv/videos/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back']
[debug] Encodings: locale cp932, fs utf-8, pref cp932, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (pip)
[debug] Python 3.10.0 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1l  24 Aug 2021)
[debug] exe versions: ffmpeg 5.1-essentials_build-www.gyan.dev (setts), ffprobe 5.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[Nebula] Logging in to Nebula with supplied credentials
[Nebula] Authorizing to Nebula
[Nebula] Extracting URL: https://nebula.tv/videos/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Fetching video meta data
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Fetching video stream info
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Downloading m3u8 information
ERROR: [Nebula] Extractor failed to obtain "id"; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
Traceback (most recent call last):
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1518, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1615, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1674, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2495, in process_video_result
    raise ExtractorError('Extractor failed to obtain "id"', ie=info_dict['extractor'])
yt_dlp.utils.ExtractorError: [Nebula] Extractor failed to obtain "id"; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

[gamer191]

If my understanding of nebula.py is correct (and it likely isn't), the issue is that that video's zype_id (fetched from https://content.watchnebula.com/video/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back/) is null

[pukkandan]

cc @hheimbuerger

Or, if they are unavailable, you can share account detail with me:
Contact: discord (pukkandan#4207) / reddit (/u/pukkandan) / email (pukkandan.ytdlp@gmail.com)

[Lamieur]

First problem is that there's a new API:

• current extractor uses https://content.watchnebula.com/video/...

• but the website started using https://content.api.nebula.app/video/...

And the old link still works, but actually returns 404 for videos posted in the last 24h.

The new API's reply is different, but the only important part is still there (manifest/m3u8).

Second problem: there's no zype_id for the main video. You can pull it from another API call, but I don't think it's necessary (it's not actually the real video id, it was used just to please yt-dlp).

If yt-dlp insists of having real ids, there actually are real video ids in Nebula (looks like "video_episode:8c2f6d8e-29ea-450f-bc16-51d896a2f59a" and is used in the API, like https://content.api.nebula.app/video_episodes/video_episode:8c2f6d8e-29ea-450f-bc16-51d896a2f59a/ - we might need to start using that API eventually). I'm removing the "video_episode:" part from the id in the patch, because it was getting mangled into "：" (also question marks in the titles are mangled into "？") when actually downloading.

Trivial patch working for now:

--- nebula.py@2023.03.04	2023-03-06 01:00:00.000000000 +0100
+++ nebula.py	2023-05-09 19:02:14.063160007 +0200
@@ -65,7 +65,7 @@
         return response['token']
 
     def _fetch_video_formats(self, slug):
-        stream_info = self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/stream/',
+        stream_info = self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/stream/',
                                             video_id=slug,
                                             auth_type='bearer',
                                             note='Fetching video stream info')
@@ -77,7 +77,7 @@
         channel_slug = episode['channel_slug']
         channel_title = episode['channel_title']
         return {
-            'id': episode['zype_id'],
+            'id': episode['id'].replace('video_episode:',''),
             'display_id': episode['slug'],
             'formats': fmts,
             'subtitles': subs,
@@ -186,7 +186,7 @@
     ]
 
     def _fetch_video_metadata(self, slug):
-        return self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/',
+        return self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/',
                                      video_id=slug,
                                      auth_type='bearer',
                                      note='Fetching video meta data')

[pukkandan]

Second problem: there's no zype_id for the main video. You can pull it from another API call, but I don't think it's necessary (it's not actually the real video id, it was used just to please yt-dlp).

The original nebula extractor was using Zype and the id was kept for backward compatibility of archive. But if an additional request is needed just to fetch it, we can change it.

@Lamieur Will you be fixing tests and opening a PR?

[Lamieur]

I have no idea how to run the tests, python test/test_download.py TestDownload.test_Nebula doesn't read my .netrc and fails on login, never getting to pulling video info. Someone smarter needs to step in.

As usual, I won't be making a PR :)

I wonder though, is preserving the current id helpful to people? For example, if they're archiving videos automatically I'm asking because it's still possible to grab that id with a different API call, which for my needs is a waste of time, so I just changed it to the Nebula UUID for myself (it's ugly, but already there in the json), but if preserving video ids is valuable in some use cases, perhaps my 3-line patch is not that good :)

[CarlFink]

Just dropping by to mention that I'm seeing the same issue here in the USA (unsurprisingly).

[CarlFink]

I'm not a Python dev, so forgive me if I'm missing something, please. @Lamieur, could you actually make that a patch?

[pukkandan]

python test/test_download.py TestDownload.test_Nebula doesn't read my .netrc and fails on login, never getting to pulling video info.

yt-dlp/CONTRIBUTING.md

Line 237 in ddae337

**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your username and password in it:

I wonder though, is preserving the current id helpful to people?

In theory, yes. But idk if anyone's actually using the feature with nebula. If additional API call is needed, imo it's best to not implement it rn and wait for someone to request support.

[pukkandan]

@CarlFink #7017 (comment)

[CarlFink]

@CarlFink #7017 (comment)

Not sure of the significance. If you want my Nebula account detes, no problem.

[CarlFink]

I don't know how useful this might be to you, but I'm able to download the streams manually.

First, locate two network streams named media.m3u8 in the browser developer console network tab. One represents the video and the other one, the audio. Then, download them both using VLC's Open Network Stream >> Convert action, and multiplex them using ffmpeg.

This is very annoying to do manually, but I'm pretty sure I can turn it into a script using VLC's command-line mode, or replacing VLC with ffmpeg again. Extracting the m3u8 URLs might be harder to automate. The thing I don't know how to do is extract the urls of the m3u8's from the page. Would I have to run an invisible browser instance to start the streams?

EDIT:
ffmpeg -i "first.m3u8" -i "second.m3u8" -codec copy TitleOfVideo.mp4

[rohieb]

@Lamieur thanks for your patch, I've fixed up the tests and made a preliminary PR, but I've still got some issues on the tests. See #7156

[CarlFink]

Any idea when the patch will be committed? I switched over to the nightly build, but it still won't work with Nebula.

Barring that, can anyone give me a hint on how to make ffmpeg download no more than a certain resolution of the video? It's defaulting to the highest res, which makes the downloads pointlessly huge and very, very slow. As is its well-known Achilles' heel, ffmpeg has nearly unreadable documentation.

Thanks.

[rohieb]

No idea what is the hold up for the patch, but for the size issue, you should be able to use the -F/--list-format option with yt-dlp to get a list of formats for the video, then put one of those formats behind the -f/--format option of yt-dlp.

[CarlFink]

No idea what is the hold up for the patch, but for the size issue, you should be able to use the -F/--list-format option with yt-dlp to get a list of formats for the video, then put one of those formats behind the -f/--format option of yt-dlp.

I hate to harass you, but ... "yt-dlp -F" reports

[generic] media: Downloading webpage [generic] media: Downloading m3u8 information [info] Available formats for media: ID EXT RESOLUTION │ PROTO │ VCODEC ACODEC ─────────────────────────────────────────── 0 mp4 unknown │ m3u8 │ unknown unknown

[bashonly]

@CarlFink
Here is a pre-release build with the Nebula API patch:
https://github.com/bashonly/yt-dlp/releases/tag/2023.06.09.211149

[CarlFink]

@CarlFink Here is a pre-release build with the Nebula API patch: https://github.com/bashonly/yt-dlp/releases/tag/2023.06.09.211149

Thanks. It'll be interesting, because I think Nebula changed something else. The ffmpeg command I entered above stopped working today.

[CarlFink]

I'm happy to report that it not only works perfectly, it seems to download an order of magnitude faster than the last release version that worked with Nebula. (I have no data, that's just my subjective impression.) I did not use aria2c.

Thanks go to the entire yt-dlp team, to @Lamieur, to @rohieb, and to @bashonly.

[OracoloJanus]

It still fails and now it says "ERROR: 'NoneType' object has no attribute 'format'"

[bashonly]

@OracoloJanus that's due to an unrelated bug/regression in version 2023.06.21
you need to run yt-dlp --update-to nightly or install master branch with pip