Site support request: watchnebula.com

[opsoyo]

Checklist

• I'm reporting a new site support request
• I've verified that I'm running youtube-dl version 2019.05.20
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that none of provided URLs violate any copyrights
• I've searched the bugtracker for similar site support requests including closed ones

Example URLs

• Video page: https://watchnebula.com/videos/wendover-how-hong-kong-changed-countries
• Video embed: https://player.zype.com/embed/5ce5603089382f54d38d0127.html?autoplay=false&access_token=[REDACTED]
• Channel page: https://watchnebula.com/wendover

Description

M3U file provided by single video example above: https://manifest.zype.com/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2NzbS56eXBlLmNvbS81YzE4MmQwNjY0OWYwZjEzNGEwMDE3MDMvNWNlNTYwMzA4OTM4MmY1NGQzOGQwMTI3LzVjZTU2MGUxMWM0Zjk2N2ZiMzdkODExYS81NDViZDZjYTY5NzAyZDA1YjkwMTAwMDAvMTlhYzA5MzEtNTA5OS00YzRlLTg0OTYtY2QwYzVlNTJhNDEyLm0zdTgiLCJwYXJhbXMiOnsiNnBMS01RM3kiOiJBM0x6aVEzaSIsIkYzOGIxZEY0IjoiNWNlZmM3ZTU1OTc3OGI1N2UxODMxMmM2In0sImV4cCI6MTU1OTIyODk1NX0.xMg1JJJMEBV1mAnlrYB4xP8rvGTGeN-5_DICu489jR8

M3U file for link above:
m3u.txt

Video page source code:
video_page_source.txt

Video page HTML DOM:
video_page_dom.txt

Channel page source code:
channel_page_source.txt

Channel page HTML DOM:
channel_page_dom.txt

[kristofferR]

Yeah, this would be sweet.

[opsoyo]

@remitamine Did you reopen from the comment by @kristofferR or do you need some testing done? Let me know. I'll try running whatever you need tonight.

[remitamine]

this issue has been closed by mistake instead of #18566.
the only change that is relevant to this issue is the support(in the next version) for Zype embed URLs with access_token, similar to the ones used by Nebula.

[danfiscus]

Is there a secure way I can share my account login with a developer willing to work on this?

[hheimbuerger]

FYI, I'm working on a Nebula extractor. (CC @danfiscus) It's at the proof-of-concept stage right now, but it successfully downloaded the three videos I threw at it so far.

I'm actually new to Nebula, subscribed just yesterday. Am I right to assume that there isn't a single public video (for testing)? They're all behind the subscription wall, correct?

This is also my first extractor. Can someone advise me how to set a cookie jar when testing with the test_download.py script? (Right now, I'm supplying my own cookie via self._set_cookie() in the extractor, but of course I don't want to push that.)

As @remitamine has already hinted at, it's just extracting some meta data and then uses the url_transparent type to defer to the Zype extractor.

[opsoyo]

AFAIK there is no public access to Nebula, at least not over the past two years.

Unfortunately, I don't know enough about YouTube-DL to help with the cookie situation.

And I can test your script whenever you're ready. Thanks for the work!

[hheimbuerger]

Here is my proof-of-concept implementation of Nebula support. Feedback very welcome, there's a good number of open questions — mostly with how to write a best practice extractor, but some are around Nebula as well.
And of course it would be great to test with as many videos as possible, to identify edge cases I haven't thought of. (Please send me links to videos that fail to download!)

I'm a bit frustrated that I cannot really figure out how Nebula determines the creator or uploader (often it's rather the series). There's other metadata missing, but that's the one that I personally wish was there.

I will submit a PR in the next days, but I'm not sure if I have the ability to do a full submission, with testing builds, Python 2.6 and 2.7, etc.

[hheimbuerger]

I did figure out a way to determine the channel name. I found the corresponding React code the frontend uses, and it's a bit absurd, but I'm now replicating the same approach in the extractor.

let channel;
if (video && video.categories && video.categories.length) {
    const channelTitle = video.categories.map((category) => (category.value[0]))
                                         .filter((title) => (!!title))[0];
    channel = getChannelByTitle(state, { title: channelTitle });
}

Basically, it finds the first (truthy) value in the category list and that's assumed to be the channel name. And then the channel details (e.g. the URL) are looked up by title (!) (not by any kind of ID) via an additional API call (that part I won't implement for now).

@opsoyo @danfiscus For an edge case test, are you aware of any 'free floating' videos on Nebula? I.e. can a video exist without a channel?

[opsoyo]

I haven't seen any videos without a channel. I think it's fair to assume no videos currently exist (publicly) without a channel.

As for the channel name, thanks for digging around. My original inspection didn't discover any signs of a well-formatted or organized JSON, so hacky sites will be treated in hacky ways.

[danfiscus]

I don't know of any channel-less videos either, but I can take a look later this evening and edit this comment.
Looks like you've made a lot of progress btw, thanks for your hard work!

[hheimbuerger]

I do want to implement channel support — that is, the ability to point youtube-dl at a channel, as opposed to video, and have it download all videos listed there — and I also roughly know how to do it, but I decided to go forward and try to get this accepted into the mainline first. Baby steps and all.

[InVItroVeritas]

So, is there any hope to this?

[hheimbuerger]

@InVItroVeritas The implementation is complete (for basic videos, not yet entire channels). We're now waiting for the project maintainers to review, suggest changes and hopefully accept the contribution. Note that there's currently 637 other open pull requests on this project, so this might take a while!

If you don't want to wait for that, and you have some basic experience with running Python applications, you're welcome to download my branch directly and test it. Please provide feedback!

[s0py]

@InVItroVeritas The implementation is complete (for basic videos, not yet entire channels). We're now waiting for the project maintainers to review, suggest changes and hopefully accept the contribution. Note that there's currently 637 other open pull requests on this project, so this might take a while!

If you don't want to wait for that, and you have some basic experience with running Python applications, you're welcome to download my branch directly and test it. Please provide feedback!

Hello! thank you for making this branch!
For anyone who is having some trouble installing:
(I am on ubuntu)
you first download the zip from hheimbeurger and extract the folder inside to somewhere nice. Then navigate there in the terminal and run the following command:
"sudo python3 setup.py install" and that should install his branch!

[wasmitnetzen]

If you are familiar with pip, you can also just run pip install git+ssh://git@github.com/hheimbuerger/youtube-dl.git@add-nebula-support.

[Xan-Kun]

Is this planned to go to master or only as an extra branch?

[hheimbuerger]

Is this planned to go to master or only as an extra branch?

Not sure who you're asking. It's not on me to decide, but the maintainers of youtube-dl.
As I mentioned before, there's hundreds of other requests for inclusion (pull requests), so I guess the chances of this to be officially released are pretty slim.

[Xan-Kun]

Thats sad to hear, as I really like nebula and think it's a great project/experiment.