Future of Mopidy-YouTube (was: API Compliance Audit)

[dz0ny]

I received mail (which sadly went to SPAM folder)
in order to continue use we would need to go through https://services.google.com/fb/forms/ytapicommercializationapplication/. However mopidy-client is violating TOS for API access, strictly https://developers.google.com/youtube/terms/required-minimum-functionality which means we must show video window and ads in order to use API.

Furthermore, since API-key is linked to my primary account, it seems my Google account is now under review. One of the punishments for violating TOS is the suspension of the account, so a lot of fun.

Google also informed me that API key has been invalidated so it can no longer be used. My suggestion here is to discontinue mopidy-youtube since we cannot guarantee (even if we do OAuth authentication and they provision their own key) that users won't have their account under suspension/review.

[interru]

pafy doesn't use the API / youtube-dl also doesn't. So the only thing we need to replace is the search. We could webscrape the search results or create an API complaint standalone project with it's own open API which we secretly exploit. :D

[madc]

NewPipe also does not use the API. Not sure, if Google is OK with that tough..

NewPipe does not use any Google framework libraries, or the YouTube API. It only parses the website in order to gain the information it needs. Therefore this app can be used on devices without Google Services installed. Also, you don't need a YouTube account to use NewPipe, and it's FLOSS.

[seth--]

I've been working on migrating from pafy to youtube-dl, you can see my progress at https://github.com/seth--/mopidy-youtube/tree/youtube-dl

This is based on #64 , which still used the api for searching. This PR changes that and scrapes /results instead.

Instead of hiting the url for searching and then hiting each result for details, I just take everything from /results.
Bitrate is not available, but doing 1 request instead of 16 seems worth it to me.

Lookup results are cached. I don't know why, but mopidy was calling lookup() twice when pressing play

I've added lots of logging

While this PR shouldn't be merged without unit tests working, it would be useful if somebody tried it while I fix that. This would make mopidy-youtube work again and close #92, #41 and related issues

TODO:
Unit tests
Only gets results from the first page. Ideally, it should keep getting new ones when you scroll. At least, it should be configurable
Support for youtube playlists (how? maybe get all the videos in the playlist and add them to the result?)
Pasting a url into the search box doesn't work. Should work for single videos and playlists
(I get that some of these issues come from working on hadogenes/mopidy-youtube instead of mopidy/mopidy-youtube, but the code is already there and will be easy to fix them. This also explains the merge conflicts)

Possible issues:
Album is set to "Youtube". mopidy-soundcloud uses "Soundcloud", so this seems ok
Artist is set to the uploader name. There is code to link it to the channel, but it performs more requests, doesn't work properly and may not be useful
Album images are not returned on search. It doesn't seem to affect musicbox_webclient or iris.

Let me know if this works for you and if the performance is acceptable

[kingosticks]

Do you think scraping the search results is going to be robust enough? Do you have any metrics for comparing the performance I.e. How much slower is this?

[kingosticks]

And related to the first question, have you looked at one of the other existing scrapers like https://github.com/aviaryan/python-gsearch ?

[seth--]

Do you think scraping the search results is going to be robust enough?

It uses a regex, so it may break when youtube gets a redesign. Luckily, that doesn't happen often. I've tried to make the regex not too specific, allowing it to keep working if there are small changes in the html. Anyway, fixing the regex takes very little time, I can just do it when needed

Do you have any metrics for comparing the performance I.e. How much slower is this?

Compared to what? Search doesn't work on develop, there isn't anything to measure

I asked for people to give this a try because I'm having around 40% packet loss at home and it's impossible for me to be really test the performance, but it seems to be acceptable for such a bad connection

And related to the first question, have you looked at one of the other existing scrapers like https://github.com/aviaryan/python-gsearch ?

That seems to be for searching google and my patch is for searching youtube, I don't see the relationship

[kingosticks]

Sorry, i was searching earlier, that Google search project came up and I somehow didn't twig that it wasn't what I was looking for! Please ignore.

In regards to performance, you could register for your own youtube API key which can be substituted for the revoked one.

[PureTryOut]

@seth-- I tried out your fork, and at least using the search provided by Iris, I get no results no matter what I search for. Are you sure it still works?

[laurensV]

My suggestions is to not discontinue modipy-youtube, but lets try to obey the TOS. Maybe we can play the video with ads in the webclient whenever its playing a youtube video? And instead of linking it to one account a possibility for people to do OAuth authentication and provision their own key

[jluttine]

Would it make sense to use API for searching and then web scraping for playing the audio? If I understood correctly, this would have the benefit that searching works easily and we wouldn't need to play ads nor violate TOS for playing the audio.

[stekern]

@jluttine As far as I can tell, that's the original behavior. The Google API was only used to fetch results, and the results were then passed to pafy which finally passed a streaming url to GStreamer. The actual streaming happened outside of the Google APIs.

I've forked @seth--'s work and done some testing for a couple of hours. It's working quite decently. Some quick notes:

1. Adding tracks to the tracklist can be relatively slow.
2. When a track starts, it takes a couple of seconds before playback actually starts. However, if I start a track and quickly perform a seek to another location in the track, the playback starts almost immediately.
3. Search is pretty quick. I've implemented support for fetching multiple pages of results at a time here https://github.com/stekern/mopidy-youtube/tree/youtube-dl.
4. Tracks sometimes abruptly end when seeking. Seems to happen with .mp4 streams. .webm streams seem to work fine.

[gilgongo]

Hi - can I assume this issue relates to the fact that I can't run mopidy-youtube on Ubuntu 18.04?

INFO     Disabled extension youtube: Dependency The 'pafy>=0.3.35' distribution was not found and is required by the application not found

$dpkg --list | grep pafy
ii  python3-pafy                          0.5.2-2                             all          Download videos and retrieve metadata from YouTube

[jodal]

If in doubt it something has the same root cause, please open a new issue.

The Debian package of pafy only packages the Python 3 version. To use Mopidy-Youtube you must install pafy from PyPI using python2.7's pip.

I'm on mobile now, but will try to remember to review and update any install docs for this over the next few days.

[Neronus]

I'm wondering - why not go via youtube_dl, instead of querying the API directly.

IIUC, you can use youtube_dl.YoutubeDL().extract_info('ytsearch10:elvis') to download meta-information for the top 10 tracks with query 'elvis'.

[natumbri]

@Neronus Interesting idea. The reason for using the API is that it is much faster (probably because of the number and nature of http requests required to get the information without the API).

I wonder if using youtube_dl would be better than the web scraping approach that I've used in the stuff I've been doing on mopidy-youtube..?

Have you had a look at my fork? It is working pretty well at the moment...

[Neronus]

Yepp, youtube_dl is a tad slow, especially if it has to scrape detailed information for each result (say, because we want to know a track's length).

OTOH, using the API means the use of an API key. I'm worried that Google decides that we're doing something unjustified. I'd rather not lose access to the rest of my Google associated products because of streaming Music from YouTube :D

[natumbri]

@Neronus you should check out my version over my fork (which I've submitted as a pull request here).

The Google API is optional. If you want to use an API key, you can (for speed and reliability), but if you don't want to (because, for example, you're worried that it might be outside the Google terms of service), you don't have to.

Best of both worlds.

[Neronus]

Thanks - I tried it without setting a key. In that case, search results return empty. With youtube_dl it's indeed close to being unbearably slow.

…

On Sun, Nov 3, 2019 at 11:10 PM natumbri ***@***.***> wrote: @Neronus <https://github.com/Neronus> you should check out my version over my fork <http://natumbri/mopidy-youtube> (which I've submitted as a pull request here <#m_563131535534046772_115>). The Google API is *optional*. If you want to use an API key, you can (for speed and reliability), but if you don't want to (because, for example, you're worried that it might be outside the Google terms of service), you don't have to. Best of both worlds. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#92>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQ5TQX7CSQHKP42T73JULQR5DV3ANCNFSM4EU2BRBA> .

[natumbri]

@Neronus did you try my fork, with no API key? It works for me just fine, and others seem to be having success.

Although it is a bit slower than the API version, in my experience it is not unbearable, and it certainly should return some results.

[NathanealV]

So we can't use mopidy youtube? Is there any other alternative?

[natumbri]

@NathanealV My version of mopidy-youtube works for me. Maybe try that and see if it works for you, too? I'd be interested to hear how you go and to help you get it going. Once you've seen it in action, you can decide if it is good enough.

[NathanealV]

@natumbri I am downloading it now but what commands do I use?

sudo pip install https://github.com/natumbri/mopidy-youtube/archive/develop.zip

I have just used this. Do I need to execute the API one as well?
Also can I use this with Mopidy Iris?

[PureTryOut]

@NathanealV My version of mopidy-youtube works for me. Maybe try that and see if it works for you, too? I'd be interested to hear how you go and to help you get it going. Once you've seen it in action, you can decide if it is good enough.

Is there no chance of your changes getting merged back to this original source? I do not like running random forks.

[NathanealV]

I tried using your service without the API but it just causes my server to show disconnected in settings and it won't reconnect until the service restarts. Any suggestions?

[natumbri]

@PureTryOut it doesn't seem likely that my changes will get merged back into the source. I've submitted a pull request and offered to become maintainer, but no feedback on that from the devs.

I suspect the devs are flat out on other things (Python 3, maybe...) and don't have the time or motivation to review such a fundamental change to such a marginal plug-in.

And - when it comes to writing code - I really don't know what I'm doing, so the code probably needs lots of revision and I'm probably not really a good choice for maintainer - I'd probably create more work for everyone!

[natumbri]

@NathanealV that doesn't sound great. Maybe open an issue over in my fork. When you do, make sure you describe your setup, including version numbers for mopidy and youtube-dl, your mopidy.conf and maybe post some logs. And we can go from there. At this stage your description is a bit too general to even guess at what might be wrong.

[natumbri]

@NathanealV thanks for opening the issue over on my fork, and glad to hear you were able to get it working by updating youtube_dl and fixing a permissions issue!

[kingosticks]

HI @natumbri,
Your PRs are appreciated but they are difficult to review because of their size/scope, and because none of the Core Dev team currently use the extension. I was trying to encourage them to be kept smaller but I see how that's hard to do with such a fundamental change, and how it also slows you down, which is no fun. Our development documentation were refreshed as part of the latest release and there's useful info in there regarding contributing and code style etc.

As you guessed, we have been busy with Mopidy 3 but I'm sorry we have not been more communicative. Especially considering you've clearly spent a lot of time on your fork. You have pretty much fulfilled the criteria for become project maintainer and you've demonstrated your commitment by remaining active here and helping users. Thank you.

But Mopidy-YouTube must be ported to Python 3 for it to have a future, would you be able to do that to your fork? We can offer some guidance with this if you need, and it shouldn't be too bad a job. If so, you are the best person to take on the maintainer role and we will transfer the project to your GitHub account. That is, assuming you are still interested?

[natumbri]

@kingosticks no problems at all! I absolutely recognise that you've all been very busy getting mopidy 3 out - a huge job.

And between the dramatic changes and my inexperience with Python (programming, generally) and with git, reviewing my PRs is obviously not straightforward. (And that's without even mentioning the fundamentally questionable footing on which the extension sits, vis-a-vis the YouTube terms of service....)

I do hope I can port it to python3 - I quite like using it: there is a huge variety of music content (particularly live and rare stuff) on YouTube that isn't really available elsewhere. I've started moving to p3 and I don't think it will take too much work.

The process of porting will hopefully help me to better understand how the extension works and maybe iron out some existing bugs (without introducing too many new ones). Hopefully I can also learn how to use Python, git and the testing and CI tools.

But today I'm watching the cricket - go Aussie
🇦🇺🏏🇳🇿🍺

Cheers
Nik

[kingosticks]

So I finally got around to trying to transfer the repository to your account but it turns out you can only do that if the destination account (@natumbri) doesn't already have a repository with the same name. So you need to delete your GitHub fork first (but obviously keep your local copy with all your changes!).

[natumbri]

Oh great! Rather than deleting, I renamed my GitHub fork - will that work? If so, you can go ahead and transfer, and I'll delete my fork , once everything is tidied up.

[jodal]

Tried again now. New error: "You can only transfer a repository from an organization to yourself at this time"

Thus, I've given @natumbri owner access to the repo, so they can try transferring it themselves.

[natumbri]

This seems to have worked!

[jodal]

Great! Could you please remove the repo's webhook that posts updates to Zulip? :-)

[jodal]

Also, let me know what your PyPI username is, so I can give you access to upload new releases.

[natumbri]

I've removed the webhook and have set up a PyPI account (natumbri).

Cheers!

[jodal]

I've now given you permission to push releases to PyPI :-)

[natumbri]

Excellent!