Add support for third-party extractors #1198

Closed
wants to merge 3 commits into
from

Projects

None yet

4 participants

@ravishi
ravishi commented Aug 6, 2013

In response to issue #1185.

Other packages can register new extractors through the youtube_dl.extractors setuptools entry-point.

For example:

setup(
    # ...
    entry_points={
        'youtube_dl.extractors': [
            'myextractor = mypackage.extractor.MyExtractorIE',
        ],
    },
)

Once the package AND youtube-dl are installed, the registered extractor will work just like an ordinary extractor.

One issue that I didn't take into account while coding this is the order of the extractors. The extractors will be registered in the order defined by the pkg_resources algorithm, which is uknown to me.

@ravishi
ravishi commented Aug 6, 2013

There is the draft implementation. Please critique it.

I'll prepare some documentation on this feature and also an example implementation. I'll submit them as soon as I finish them.

@jaimeMF
Collaborator
jaimeMF commented Aug 22, 2013

Just one comment: is there some reason to not add the external extractors to _ALL_CLASSES, instead of adding them in gen_extractors?

@jaimeMF
Collaborator
jaimeMF commented Aug 22, 2013

Another thing, the pkg_resources module is not part of the standard library, you should add something like this to make sure it works in all python installations:

try:
    import pkg_resources
    def pkg_iter_entry_points(*args, **kargs):
        return pkg_resources.iter_entry_points(*args, **kargs)
except ImportError:
    def pkg_iter_entry_points(*args, **kargs):
        return []
@ravishi
ravishi commented Sep 27, 2013

@jaimeMF,

I didn't add them to _ALL_CLASSES in order to avoid calling a plugin system at import time.

Also, I didn't quite understand the purpose of _ALL_CLASSES. It's not part of any kind of public API or something, right? Seems to me that it only exists as a convenient way to organize that specific piece of code. Am I getting anything wrong here?

@jaimeMF
Collaborator
jaimeMF commented Sep 30, 2013

It makes senses.

About the order problem you commented, it shouldn't matter as long as the _VALID_URL fields don't match the same urls, actually our extractors are ordered according to globals().items().

It looks good to me, @phihag what's your opinion?, and where should we document it for developers?.

@ravishi
ravishi commented Sep 30, 2013

One thing that I couldn't figure yet is how to make this work with youtube-dl binaries. Specially on Windows. I don't know py2exe enough to tell if it's possible to use this kind of entry points.

@jaimeMF
Collaborator
jaimeMF commented Oct 1, 2013

The "binaries" should work in all Unix systems (it's just the source code in a zip file with a shebang), because it uses the same python executable you run, for example I've tested it inside a virtualenv where I've installed an InfoExtractor and it works.
Windows is another story, the youtube-dl.exe comes with its own interpreter, so I don't think it can use the entry points.

@ravishi
ravishi commented Oct 3, 2013

Cool. Maybe we could create a plugin system or something like that. Like a directory where the users could simply extract the extractors into. youtube-dl then would scan that directory, gather and load the extractors. That could work for windows.

@FiloSottile
Collaborator

I think that the use case for this is mainly other python packages offering an extractor, not really self-contained plugins (if it's self contained, merge it!)

So, the current implementation is good :) can you add some documentation and maybe rebase without whitespace changes while I test that it does not break the binaries?

@ravishi
ravishi commented Sep 4, 2014

Ok, guys, I'm nobody's poet but I tried to write some documentation on this feature. Is that enough for it to get merged?

@jaimeMF? @FiloSottile? Someone?

ravishi added some commits Aug 6, 2013
@ravishi ravishi Add support for third-party extractors
Other packages can register new extractors through the `youtube_dl.extractors`
setuptools entry-point:

    # in mypackage/setup.py
    setup(
        ...
        entry_points={
            'youtube_dl.extractors': [
                'myextractor = mypackage.extractor.MyExtractorIE',
            ],
        },
    )
8f2216b
@ravishi ravishi Make sure things still work without pkg_resources 7afa5c4
@ravishi ravishi Explain setuptools entry points in the README b4ea83d
@phihag
Collaborator
phihag commented Dec 17, 2014

Please refer to the discussion in #1185 as to why I do not think that this is a good idea, and actively want to limit the number of third-party extractors. Now, if there truly is an extractor that cannot be integrated into youtube-dl, but still has to be distributed on a wide scale, I may change my mind. As it stands, I have to apologize for the delay and sadly reject this pull request.

@phihag phihag closed this Dec 17, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment