Add a way to register custom (external) extractors.

[ravishi]

I wonder if this was not requested before.

I think that there should be a way to easily register custom extractors into youtube-dl.

It could be done through cli arguments or (even better) setuptools entry-points.

[yasoob]

You can! Just make a custom extractor and import it into init.py file
in the extractor folder :)

On Tue, Aug 6, 2013 at 12:40 AM, Dirley Rodrigues
notifications@github.comwrote:

I wonder if this was not requested before.

I think that there should be a way to easily register custom extractors
into youtube-dl.

It could be done through cli arguments or (even better) setuptools
entry-points.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1185
.

[phihag]

I'm not sure what this would be useful for. Why wouldn't you just publish your extractor? While youtube-dl is licensed under Unlicense (basically Public Domain), I don't see any reason why we should encourage forking, which this would amount to. Of course, you could argue that we're the bottleneck, but new extractors are typically the fastest changes accepted.

[ravishi]

@phihag,

It would be useful to allow third-party Python packages to register their own custom extractors.

I think that it would, on the contrary, discourage forking, since people would be able to just plug-in their own extractors.

Also, there are complex and specific extractors, like the ones that require the user to request for API keys and stuff from the service provider. I don't know your policy for extractor inclusion, but seems to me that higly specific or complex extractors are good to avoid.

[phihag]

Our policy for extractor inclusion is simple: If it is extracting audio or video, we'll include it. API keys can just go into the configuration if they need to stay secret. I'd like all users of youtube-dl benefit from an extractor.

[ravishi]

But how about complex extractors?

I have an extractor that depends on pyAMF and does some pretty nasty stuff with keys, secrets and salts. pyAMF must be compiled and depends on some amf libraries.

That would increase the complexity of youtube-dl (and it's installation process). But I could release it as a separate package and register an entry-point, so it would integrate into youtube-dl.

Anyone interested in this specific extractor could assume the trouble of compiling and setting it up, while the great majority of youtube-dl users wouldn't be bothered.

If it ever gets widely used, or if by some reason the complexity comes to be removed, it can be later merged into youtube-dl either by the original author or by another contributor.

[phihag]

@ravishi You're right, that is indeed a sensible reason for having an interface for external extractors. Are you willing to release your code under Unlicense? If so, can you give me a link to a repository?

[ravishi]

I don't have the code yet. I have some unreleased tools that download songs from some paid services that i'm willing to integrate into youtube-dl.

Since I know that this could be useful for you I can try to come up with something to show you.

[yasoob]

Yes @ravishi , it would be really helpful for us. Every little contribution counts. If you face any difficulty while making it compatible with youtube-dl then feel free to open another issue.

[ravishi]

I updated the PR at #1198, could you guys check it out?

[phihag]

Reconsidering, I do not think that this is a good idea. We regularly manage to go from suggested extractor to release with the new code within a day, and nearly always do within a week. Therefore, I do not think that turnaround time is a problem.

I fear this change would enable a wild proliferation of extractor code (and extractors-for-pay and the like). We are already seeing wildly grown tutorials which advise users to pass in crazy options. Imagine how much worse the situation would be if users started using locally-patched versions of youtube-dl! Inevitably, while the patches may have been useful at a specific date in a specific situation, mainline youtube-dl may add support in the meantime, or change its interfaces.

Furthermore, I do not want to ease it for developers to write code that will never be integrated into mainline youtube-dl - while we have an extremely lenient license, encouraging people to not contribute to the project is against the very spirit of open source software development.

rdio-dl seems to be the poster child for this PR. Looking at the code, the only reason for not plain integrating it into mainline youtube-dl is the lack of a license. I do not want to enable developers to write plugins for youtube-dl under different (potentially proprietary) licenses.

I apologize for the delay, but I sadly have to close this issue as wontfix. It's easy enough to create a private branch of youtube-dl with a private extractors. The vast majority of extractors would be accepted just fine into youtube-dl, where we can ensure that our users get a consistent experience and where we can update the code if helper functions change.

[ravishi]

rdio-dl used to depend on an external library (PyAMF) to do its work. That
was library that had C code in it and wasn't very portable. Only in a
recent update it was possible to stream through http, and thus extract
songs without the need of the pyamf library. Its the same thing with
Spotify (which can be extracted trough libspotify). These services also
need highly specific session and cookie keeping in order to properly work.
How can we support that if not through external extractors, given that the
goal of youtube-dl seems to be easily installed (one bundle) and portable?

I'm open to discussion here, I just don't want to be constrained to pure
Python standard library to build all extractors. YoutubeDL has an awesome
infrastructure for developing this kind of extractors, but it's hard to
reuse that without any kind hooking system.

Em qua, 17 de dez de 2014 21:00, Philipp Hagemeister <
notifications@github.com> escreveu:

Closed #1185 #1185.

—
Reply to this email directly or view it on GitHub
#1185 (comment).

[phihag]

Well, if an external library is truly necessary, than we can just make the import in the relevant method (and fail gracefully if the library is not there). We already do that for xattrs, libc extensions for terminal title setting, and probably others.