Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is get_streaming_data still working? #196

Closed
KoljaWindeler opened this issue May 17, 2021 · 43 comments
Closed

Is get_streaming_data still working? #196

KoljaWindeler opened this issue May 17, 2021 · 43 comments
Labels
yt-update A server-side change caused this issue

Comments

@KoljaWindeler
Copy link

KoljaWindeler commented May 17, 2021

Hi, this is one of those "some user have a problem but not all" things.

Original issue:
KoljaWindeler/ytube_music_player#97
I can't reproduce the issue myself but quite a few user suddenly lost the ability to listen to music.

My code would call get_streaming_data but (according to the logs) would only get an empty response {}.

As of now I use a backup method for those users but wonder why get_streaming_data isn't working anymore for so many users.

This is the interesting part:
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] No adaptiveFormat and no formats found
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] get_streaming_data(SQyu9vfW9Bk)
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] {}

https://github.com/KoljaWindeler/ytube_music_player/blob/dc06c8b1c37c6cc0fc01769f7d9854f11a4a4f26/custom_components/ytube_music_player/media_player.py#L1118

@sigma67
Copy link
Owner

sigma67 commented May 17, 2021

It might be related to the x-goog-visitor-id. Can you have the troubled users try again with the latest version? Perhaps that already fixed the issues.

@KoljaWindeler
Copy link
Author

hah, interesting .. now I have the same issue with 17.0 ... what can i do to get you the needed infos?

@sigma67
Copy link
Owner

sigma67 commented May 18, 2021

Thanks for being persistent, it seems the flow has changed on the YouTube side. There is now a YouTube Music specific endpoint, it no longer uses the YouTube get_video_info.

The new endpoint is https://music.youtube.com/youtubei/v1/player. Besides streamingData it also returns some other things like videoDetails, playerConfig, storyboards and an attestation, to prevent bots.

I propose we replace the existing call with that one, but keep the method name.

@sigma67 sigma67 added the yt-update A server-side change caused this issue label May 18, 2021
@KoljaWindeler
Copy link
Author

I can totally adopt as the current result is empty anyway ;)

@sigma67
Copy link
Owner

sigma67 commented May 18, 2021

I pushed a simple implementation, feel free to try and provide feedback :)

The method is now get_song only which also provides the streaming data.

@KoljaWindeler
Copy link
Author

KoljaWindeler commented May 18, 2021

hmm ok .. i was able to install the latest version from github and test the get_song interface. It look fine but I can't decode a link from these data anymore .. so the format or the way how the data was coded seems to have changed .. damn ..

Edit: ok this is really a problem for me .. i used to load your api and the pytube lib (https://pytube.io/en/latest/).
Your code provided the streaming data (ciphered_signature + url) and the pytube.cipher.get_signature() methode decoded the ciphered_signature. Joining them together resulted in the direct streaming link that I'd forward to the various players. It turns out that those links aren't valid any more. So I'd assume that the cipher for YouTube.music and YouTube now differ. Bummer.
Any idea how I could get the direct streaming link now?

@KoljaWindeler
Copy link
Author

ok this is really a problem for me .. i used to load your api and the pytube lib (https://pytube.io/en/latest/).
Your code provided the streaming data (ciphered_signature + url) and the pytube.cipher.get_signature() methode decoded the ciphered_signature. Joining them together resulted in the direct streaming link that I'd forward to the various players. It turns out that those links aren't valid any more. So I'd assume that the cipher for YouTube.music and YouTube now differ. Bummer.
Any idea how I could get the direct streaming link now?

https://github.com/KoljaWindeler/ytube_music_player/blob/8aea65d2c5a7ed0ef22677428ca001ce927578ca/custom_components/ytube_music_player/media_player.py#L1164

@KoljaWindeler
Copy link
Author

some more info... the cipher of youtube and youtube music are indeed identical ... so even if I point Pytube to the Youtube music js i get the same decoding schema ... and the same decoded signature ... and the same error ... so it feels like something is of with the data that get_song returns .. any clue?

@sigma67
Copy link
Owner

sigma67 commented May 18, 2021

get_song is currently simply passing on the response that is being returned by the player endpoint without modification. You can verify easily by using the cipher in the response data of the player request in your browser instead. As mentioned above, have you tried running urllib.parse.unquote on the cipher before passing it to the other library?

@KoljaWindeler
Copy link
Author

Hi, yep I unquote the url. The s parameter isn't quoted or at least wouldn't change (tested ;))

@KoljaWindeler
Copy link
Author

KoljaWindeler commented May 19, 2021

good morning, I got some news .. there is definitely some off :D
I've searched for 'player' in the network tab and found the strreamingData. I've copied those data to a simple script and decoded the url without issues (horray)

after that grabbed the data for the same videoId via ytmusicapi and failed. so something is happening between browser and ytmusicapi. The issue is that the url is almost completely of .. so it's hard to tell where the issue is ..

here is the script (I guess both links won't work for you as they are mapped to my IP but the way should be fairly clear)
https://pastebin.com/FDqXu6W7

what can I do to help?

Edit1:
the format of the URL look the 'same' .. at least both (webbrowser vs ytmusicapi) have the same fields / format .. sure different server, different Id, different timestamp but the webbrowser link changes as well everytime I refresh (and all of them work)
same for the signature .. about the same length, same 'coding' ..

edit2:
not sure if this is relevant but i get two player responses in the browser .. selecting the first will give me the streaming data to the video I want .. the second might be some pre-buffering? anyway .. both provide decodable streaming data

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Thanks for the script, I tried it and it worked fine for me. No errors and both URLs get printed at the end. Can you point out where you're getting errors with the script?

The POST data of the two in-browser requests differs only with regard to the isPrefetch flag, which is true for the first one. So I think we can ignore that.

Other things that are missing in the ytmusicapi request POST data are the following (disregarding the context key, which shouldn't matter as it's used in all requests):

{ 
  "cpn": "gPcYTcHt2gharuC3",
  "playbackContext": {
    "contentPlaybackContext": {
      "autoCaptionsDefaultOn": false,
      "autonavState": "STATE_OFF",
      "html5Preference": "HTML5_PREF_WANTS",
      "lactMilliseconds": "3021",
      "liveContext": {
        "startWalltime": "0"
      },
      "mdxContext": {},
      "referer": "https://music.youtube.com/search?q=oasis+wonderwall",
      "signatureTimestamp": 18765,
      "vis": 1
    }
  }
}

CPN is a nonce computed clientside for debug purposes (Client Playback Nonce). Not sure if the playbackContext is required, but seems not as your script was working for me.

@KoljaWindeler
Copy link
Author

So both urls printed at the end work for you? The non_browser url Always results in a error 403 for me

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Yes. There might be something wrong with your headers as 403 is returned if you're not authorized. Can you access your library with those headers?

@KoljaWindeler
Copy link
Author

Library works 100% correct

@KoljaWindeler
Copy link
Author

Would it help if I send you my header file via email?

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

You could try doing another setup run, as the setup procedure has changed in the latest release. If that doesn't work you can send it to ytmusicapi@gmail.com

@KoljaWindeler
Copy link
Author

still didn't work .. email is on the way .. lets see if that works for you

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

No errors at all using your code and headers. Not sure what I should be looking for. I literally copy pasted the file in your email and ran it. I only changed the filepath to a file in the local directory (not /tmp/). Full output below:

[{'browseId': 'MPREb_OU8Rzgen6yg', 'title': 'Augsburger Puppenkiste - Urmel aus dem Eis', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/J0z6ybtB_c4sNCwjjrf-lgBQIW35Wx0EUCogiwNoPWlMytORdhKqnMTJfnvr0xRz_Z9IXaBH457yOnW4=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/J0z6ybtB_c4sNCwjjrf-lgBQIW35Wx0EUCogiwNoPWlMytORdhKqnMTJfnvr0xRz_Z9IXaBH457yOnW4=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Augsburger Puppenkiste', 'id': 'UC635W9l46-eIncasLducfEQ'}]}, {'browseId': 'MPREb_6wNjSd7Niz8', 'title': 'Die Super-Hunde (Das Special)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/mEFbJJQnSxDredoaybBQqpLnxpl4QsFaQDKDYoeosfij18iU3HWZMJkgAx99HxmXWoeGY6V31pTZIALe=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/mEFbJJQnSxDredoaybBQqpLnxpl4QsFaQDKDYoeosfij18iU3HWZMJkgAx99HxmXWoeGY6V31pTZIALe=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2020', 'artists': [{'name': 'PAW Patrol', 'id': 'UC6LfFqHnWV8iF94n54jwYGw'}]}, {'browseId': 'MPREb_vaNGs9bAg27', 'title': 'Folge 48: Der Piratenschatz', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/fcSkFZEL08rlI3Wzf14a498YP2hVtHcl8XMUknWxXXAeyeXkl2kHPwdMNRpGHf4xHxF_dRtuL4P79rj-=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/fcSkFZEL08rlI3Wzf14a498YP2hVtHcl8XMUknWxXXAeyeXkl2kHPwdMNRpGHf4xHxF_dRtuL4P79rj-=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2020', 'artists': [{'name': 'PAW Patrol', 'id': 'UC6LfFqHnWV8iF94n54jwYGw'}]}, {'browseId': 'MPREb_CxBREHcOAvV', 'title': 'Folge 118: Ein Wal braucht Hilfe', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/C6cB1qKjce3iNh1-jQQ2qA4cOGpxQb4XOwO2Mm-BiBtk2ouoKeeBEgxFLnlZk5hPU2bXvmeNcdDJl1AF=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/C6cB1qKjce3iNh1-jQQ2qA4cOGpxQb4XOwO2Mm-BiBtk2ouoKeeBEgxFLnlZk5hPU2bXvmeNcdDJl1AF=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'EP', 'year': '2021', 'artists': [{'name': 'PAW Patrol', 'id': 'UC6LfFqHnWV8iF94n54jwYGw'}]}, {'browseId': 'MPREb_TEWBxwfV8f1', 'title': 'Folge 122: Ein königliches Konzert', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/nKdunQv1xGBnyjE_Sj7zQcSOEgLL-qbEsVzh0hnoVc-6OZHl6bikEIMY5UFgxy2hh3PmraKoo8fEvDea=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/nKdunQv1xGBnyjE_Sj7zQcSOEgLL-qbEsVzh0hnoVc-6OZHl6bikEIMY5UFgxy2hh3PmraKoo8fEvDea=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'EP', 'year': '2021', 'artists': [{'name': 'PAW Patrol', 'id': 'UC6LfFqHnWV8iF94n54jwYGw'}]}, {'browseId': 'MPREb_4R414j4ecyK', 'title': 'Folge 24: Die Hundeschau', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/lblJFflx_K3pG1qgAtf7Vydz9HfClul9L_F_Dv3y0NUC_6Ijg_gVDHY1otrQPorstOL5KVOwaAu4Uvrb=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/lblJFflx_K3pG1qgAtf7Vydz9HfClul9L_F_Dv3y0NUC_6Ijg_gVDHY1otrQPorstOL5KVOwaAu4Uvrb=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2020', 'artists': [{'name': 'PAW Patrol', 'id': 'UC6LfFqHnWV8iF94n54jwYGw'}]}, {'browseId': 'MPREb_cSMQFcJAK7s', 'title': 'Alles verschenkt!, Alles Winter!, Alles gebacken!, Alles taut!', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/TfRUOTStq-_IrQNfFi6bZRqpm1hXZudajKrPT6y0vcCEtX6CDjE3gYsOz_ikx-WqjAuafjLT0UjqLpY=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/TfRUOTStq-_IrQNfFi6bZRqpm1hXZudajKrPT6y0vcCEtX6CDjE3gYsOz_ikx-WqjAuafjLT0UjqLpY=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_qBw9U332jao', 'title': 'Alles vermurkst!, Alles geheim!, Alles saust um die Wette!', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/EXQF65fkVN7bfmxtrkH2DMEjsgxmPIj2v6uBNFOb7mCouv_5caquRwItqkpoHjNcLu7KD98ShFGWkCYanA=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/EXQF65fkVN7bfmxtrkH2DMEjsgxmPIj2v6uBNFOb7mCouv_5caquRwItqkpoHjNcLu7KD98ShFGWkCYanA=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_u3YrzKyyqPB', 'title': 'Alles Freunde!, Alles wieder gut! (Oder: Wie der kleine Rabe zu seinem Namen kommt)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/0f3_R3YbtWtTzst_Y9AwIzqV3A7x1EcPLcuaQHgx69Z3JeAN4p-Vcgeq_rkvY0zKco8kgNwN-gJCjxby=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/0f3_R3YbtWtTzst_Y9AwIzqV3A7x1EcPLcuaQHgx69Z3JeAN4p-Vcgeq_rkvY0zKco8kgNwN-gJCjxby=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_EkLVURyOaMs', 'title': 'Alles erlaubt?, Alles Urlaub! (Oder: Socke will auch verreisen)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/eVUAhMA3Xo2HCARQr4HbDhkGYWTEXb2okJKOvs_TywY4-wP10-UIzcNJpxGguS3iKFMQImvYDfwfog_I=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/eVUAhMA3Xo2HCARQr4HbDhkGYWTEXb2okJKOvs_TywY4-wP10-UIzcNJpxGguS3iKFMQImvYDfwfog_I=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_YkNZ59PGyIr', 'title': 'Socke aus dem All, Der Hypnotiseur, Streithähne', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/Bc29YecItPdABtEWJ3XpAi1KgGsfoiKK7sKmqjPO46AZp_hbeE4M4S2P-sVJI5GqGkjKWjcdN8roxtoapg=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/Bc29YecItPdABtEWJ3XpAi1KgGsfoiKK7sKmqjPO46AZp_hbeE4M4S2P-sVJI5GqGkjKWjcdN8roxtoapg=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Der kleine Rabe Socke', 'artists': [{'name': 'Jan Strathmann', 'id': 'UChCfCbHp68znIXMEixX7zsQ'}]}, {'browseId': 'MPREb_0oe8tZsYxFf', 'title': 'Die Mutprobe, Ein echter Krimi, Der geteilte Wald', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/sOtviylb30Sg99xtLju8bz6hpARDXQFcEi-lTDMjj3QRhGlFULZbVFZAvLBrjm26D4gpiO2OzhFUrkS2=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/sOtviylb30Sg99xtLju8bz6hpARDXQFcEi-lTDMjj3QRhGlFULZbVFZAvLBrjm26D4gpiO2OzhFUrkS2=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Der kleine Rabe Socke', 'artists': [{'name': 'Jan Strathmann', 'id': 'UChCfCbHp68znIXMEixX7zsQ'}]}, {'browseId': 'MPREb_uD88gnJRLN4', 'title': 'Alles mutig!, Alles für dich!, Alles getröstet!', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/Zj6B0cCmU7fkwyWpmnAeISFqGuThZulSXEs_mAjcl42uo_bpafRiaQZ7P5vdXgHAfnKDaa8j-GPgXrE=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/Zj6B0cCmU7fkwyWpmnAeISFqGuThZulSXEs_mAjcl42uo_bpafRiaQZ7P5vdXgHAfnKDaa8j-GPgXrE=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_koF1MsWFxjb', 'title': 'Alles Frühling!: Alles Freunde!, Alles wächst!, Alles gefärbt! (Drei Geschichten vom kleinen Raben Socke)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/P4GMfQKOZYMp0x5wTatLcroyVuT6kSuS1eVo6XDoCVUAtQd-DrfyiNuCB5TWCXCOzjqqKLpEreiy2bnl1g=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/P4GMfQKOZYMp0x5wTatLcroyVuT6kSuS1eVo6XDoCVUAtQd-DrfyiNuCB5TWCXCOzjqqKLpEreiy2bnl1g=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': 'Annet Rudolph', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_AOOtClRKUZR', 'title': 'Der kleine Rabe Socke - Hörspiel zum Film', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/F4kR34FizTdRnLwZY7wqYpwiex2NpHCoPIeUvhf6laS9jlvg93GChJMC7lhy1gBGPzXPP3SZZLjra-OPvA=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/F4kR34FizTdRnLwZY7wqYpwiex2NpHCoPIeUvhf6laS9jlvg93GChJMC7lhy1gBGPzXPP3SZZLjra-OPvA=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Der kleine Rabe Socke', 'id': 'UCM5_A-zhU-OJFMZOf5gv-ew'}]}, {'browseId': 'MPREb_hu5FRQNIOav', 'title': 'Folgen 104 - 108: Wettlauf gegen die Zeit', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/qooPWgtMkolsSeGkRBbEipukXmBJq_1amrMLWUSOrbUmn_AnMf4AXa9IE77Qsm43k5Qh7h8UrmTnbu_1=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/qooPWgtMkolsSeGkRBbEipukXmBJq_1amrMLWUSOrbUmn_AnMf4AXa9IE77Qsm43k5Qh7h8UrmTnbu_1=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_OKClsYnH8Fq', 'title': 'Folgen 119 - 123: Der Tag des Pinguins', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/VYCqmxFAP9sPTnHwqsawaST1WBC_zt6ceDZSNkd_QKk16HaWH0_Zctl-aKhKaZlSOVzVRQh3pKDBOtsu=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/VYCqmxFAP9sPTnHwqsawaST1WBC_zt6ceDZSNkd_QKk16HaWH0_Zctl-aKhKaZlSOVzVRQh3pKDBOtsu=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_o9nL4EQho3G', 'title': 'Folgen 109 - 113: Sams Geburtstag', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/6BITretZ7HoQ8LsboGl5KVNYK1lcVpcFyaqOFYg3icmshH7SIGUc2qLtg205UhWjfmfYTZP79cAKGXO-=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/6BITretZ7HoQ8LsboGl5KVNYK1lcVpcFyaqOFYg3icmshH7SIGUc2qLtg205UhWjfmfYTZP79cAKGXO-=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_PmnQwiQQuXe', 'title': 'Folgen 114 - 118: Tierische Rettung', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/OKJD6uKrcD3HJjN_mY76MCtRAqWPuhrbShd879K3gagP7HcG7NJGpPhNO_A5W_Kf2US8AEoe8u_ow6wf=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/OKJD6uKrcD3HJjN_mY76MCtRAqWPuhrbShd879K3gagP7HcG7NJGpPhNO_A5W_Kf2US8AEoe8u_ow6wf=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_BfmgAjML3Zw', 'title': 'Folgen 124 - 128: Der Schatz von Pontypandy Pete', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/zFT6Izx_jNAUxxDwqAmyopuURAehtA1v75sFNOdoDmctBq5viaAv3o2EIhupDxE-DYH20p5Dn6rhswUZ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/zFT6Izx_jNAUxxDwqAmyopuURAehtA1v75sFNOdoDmctBq5viaAv3o2EIhupDxE-DYH20p5Dn6rhswUZ=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2020', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_V9RBwfsOz3e', 'title': 'Achtung Außerirdische (Das Original-Hörspiel zum Film)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/uYF1A9I0FwsUpSpv6lnnmrltbEGfxzYH5XT0jr7h5t35Hk-Nou4m-Ll4HloEj-6NRNjI9xt4BVC0oS8=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/uYF1A9I0FwsUpSpv6lnnmrltbEGfxzYH5XT0jr7h5t35Hk-Nou4m-Ll4HloEj-6NRNjI9xt4BVC0oS8=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2017', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_oYAOqxEFEvc', 'title': 'Plötzlich Filmheld! (Das Original-Hörspiel zum Film)', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/ESyQyn8W-BfY4nNYpFXbfuKXyZZNIAiOUH904lL01_P_wyfQhwszkDJED--dc9PDYIGFTGmEx1q7gBLr=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/ESyQyn8W-BfY4nNYpFXbfuKXyZZNIAiOUH904lL01_P_wyfQhwszkDJED--dc9PDYIGFTGmEx1q7gBLr=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2019', 'artists': [{'name': 'Feuerwehrmann Sam', 'id': 'UCPJoR__XyzZOXsKVGpnYugw'}]}, {'browseId': 'MPREb_VJh1uGFxfCd', 'title': "Rock'n'Roll Realschule (Unplugged)", 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/ovzVGZxgipuhWfewaNvFp9IazXGgn9S7QRXne2KMwVDwU7twhoGKlt9CiAWZA_YZczoPsT2XeDhApE1f=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/ovzVGZxgipuhWfewaNvFp9IazXGgn9S7QRXne2KMwVDwU7twhoGKlt9CiAWZA_YZczoPsT2XeDhApE1f=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2002', 'artists': [{'name': 'Die Ärzte', 'id': 'UCsJsxj9pPP8sZuaRiaNAavQ'}]}, {'browseId': 'MPREb_yO6z8Jf9VNx', 'title': 'The Take Over', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/4Ap6OdsV3v8Fpl_1msLMnOP2sWwJvYwinI96EeluGbK5Sj6n3cdXqnsSCQXFG0flwuGTergXwnJVZoOJ8Q=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/4Ap6OdsV3v8Fpl_1msLMnOP2sWwJvYwinI96EeluGbK5Sj6n3cdXqnsSCQXFG0flwuGTergXwnJVZoOJ8Q=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2009', 'artists': [{'name': 'Zion I', 'id': 'UC3x4CptwiIGhsr5BjVaO0AQ'}]}, {'browseId': 'MPREb_4pL8gzRtw1p', 'title': 'Revival', 'thumbnails': [{'url': 'https://lh3.googleusercontent.com/b-euORTq-_MSzc3SyI0QOjLqnCBJbWuMmH1YlyJKAVvJfOpPqv3bFH-KSDoQwV9e8Xey1X2NRo9xZ2U=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/b-euORTq-_MSzc3SyI0QOjLqnCBJbWuMmH1YlyJKAVvJfOpPqv3bFH-KSDoQwV9e8Xey1X2NRo9xZ2U=w544-h544-l90-rj', 'width': 544, 'height': 544}], 'type': 'Album', 'year': '2017', 'artists': [{'name': 'Eminem', 'id': 'UCedvOgsKFzcK3hA5taf3KoQ'}]}]
found HQ
the url was:
https://r4---sn-i5h7lned.googlevideo.com/videoplayback?expire=1621429315&ei=47ekYPOcGN2m1gLsho84&ip=31.18.169.141&id=o-ANvweDc0vyjDgCGLc8xb8EKKTLJFUYZt8DhTb01lrtaq&itag=141&source=youtube&requiressl=yes&mh=1B&mm=31%2C26&mn=sn-i5h7lned%2Csn-5goeen76&ms=au%2Conr&mv=m&mvi=4&pl=24&ctier=A&pfa=5&gcr=de&initcwndbps=2113750&hightc=yes&vprv=1&mime=audio%2Fmp4&ns=_E7L8g55NsPD7YAR1Mjz9MoF&gir=yes&clen=9046483&dur=281.017&lmt=1565939075624456&mt=1621407409&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB_REMIX&txp=1311222&n=IE-k8D-v7_gbamsyF&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cpfa%2Cgcr%2Chightc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgftYt58gK6qraAj7MWCOJ4lnenaZDYFReNuDZT7spAAMCIQDvyAmUN1EtQXXk-1E3CpES2k7HaSnRafSjnE9c7qNm7A%3D%3D
and is now
https://r4---sn-h0jelne7.googlevideo.com/videoplayback?expire=1621446399&ei=n_qkYPLVHpi67gOsxL7YAQ&ip=2a02%3A810d%3A8cbf%3Aa817%3Aa073%3A3f0%3Aa0d5%3Ab6d0&id=o-ANXOPi100AneNzldzNtpF0RIPZBdxGBQLlzzDWg2TGnv&itag=141&source=youtube&requiressl=yes&mh=1B&mm=31%2C26&mn=sn-h0jelne7%2Csn-4g5ednsz&ms=au%2Conr&mv=m&mvi=4&pl=40&ctier=A&pfa=5&gcr=de&initcwndbps=1853750&hightc=yes&vprv=1&mime=audio%2Fmp4&ns=HYHZQIh_tbNzN_hbYc6z6TMF&gir=yes&clen=9046483&dur=281.017&lmt=1565939075624456&mt=1621424455&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB_REMIX&txp=1311222&n=CBPO_XUtgq_xWUqzh&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cpfa%2Cgcr%2Chightc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAINbpfk6NpzKrPDOwzOQ0D7njA5avEfNBOkxbNsx0kpOAiEAmE2NCdjiJ_BbFEJVFl-oGjVpKtbe9kRn2k4aiB95b1E%3D

the siganature was:
R=wJChK6Rf0xKB6IA6R8oUt0gdjO6Unlo-QJtYqK5bwxBDQICIK5QRy6QEgjt8q5AQ97C=ftPR2ZDA8896K1v4kroTpMgIQRw8JQ0qOe
and is now
ypppOq0QJ8wRgIhAMW1J5NbHjpBqAUMO7zO9fc7A-KitCATBhi0KNj7g34XAiEAlWF622QPQjAjlzPZXHTB9mHBoaihkMa1eHe0kWMoniY=Y=
good link:
https://r4---sn-i5h7lned.googlevideo.com/videoplayback?expire=1621429315&ei=47ekYPOcGN2m1gLsho84&ip=31.18.169.141&id=o-ANvweDc0vyjDgCGLc8xb8EKKTLJFUYZt8DhTb01lrtaq&itag=141&source=youtube&requiressl=yes&mh=1B&mm=31%2C26&mn=sn-i5h7lned%2Csn-5goeen76&ms=au%2Conr&mv=m&mvi=4&pl=24&ctier=A&pfa=5&gcr=de&initcwndbps=2113750&hightc=yes&vprv=1&mime=audio%2Fmp4&ns=_E7L8g55NsPD7YAR1Mjz9MoF&gir=yes&clen=9046483&dur=281.017&lmt=1565939075624456&mt=1621407409&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB_REMIX&txp=1311222&n=IE-k8D-v7_gbamsyF&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cpfa%2Cgcr%2Chightc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgftYt58gK6qraAj7MWCOJ4lnenaZDYFReNuDZT7spAAMCIQDvyAmUN1EtQXXk-1E3CpES2k7HaSnRafSjnE9c7qNm7A%3D%3D&sig=0QJ8wRQIgMpTork4v1K6988ADZ2RPtf=C79QA5q8tjgEQ6yRQ5KICIQDBxwb5KqYtJQ-olnU6Ojdg0tUo8R6AI6BKx0fR6KhCJw

bad link:
https://r4---sn-h0jelne7.googlevideo.com/videoplayback?expire=1621446399&ei=n_qkYPLVHpi67gOsxL7YAQ&ip=2a02%3A810d%3A8cbf%3Aa817%3Aa073%3A3f0%3Aa0d5%3Ab6d0&id=o-ANXOPi100AneNzldzNtpF0RIPZBdxGBQLlzzDWg2TGnv&itag=141&source=youtube&requiressl=yes&mh=1B&mm=31%2C26&mn=sn-h0jelne7%2Csn-4g5ednsz&ms=au%2Conr&mv=m&mvi=4&pl=40&ctier=A&pfa=5&gcr=de&initcwndbps=1853750&hightc=yes&vprv=1&mime=audio%2Fmp4&ns=HYHZQIh_tbNzN_hbYc6z6TMF&gir=yes&clen=9046483&dur=281.017&lmt=1565939075624456&mt=1621424455&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB_REMIX&txp=1311222&n=CBPO_XUtgq_xWUqzh&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cpfa%2Cgcr%2Chightc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAINbpfk6NpzKrPDOwzOQ0D7njA5avEfNBOkxbNsx0kpOAiEAmE2NCdjiJ_BbFEJVFl-oGjVpKtbe9kRn2k4aiB95b1E%3D&sig=YinoMWk0eHe1aMkhiaoBHm9BTHXZPzljAjQPQ226FWlAEiAX43g7jNK0ihBTACtiK-A7cf9Oz7OMUAqBpjHbN5J1WMAhIgRw8JQ0qOpp

Process finished with exit code 0

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Oh I think there's a misunderstanding. You're doing another step by calling the googlevideo.com URL and getting 403 on that?

Besides id and signature differences the only thing that stands out to me is that the URLs have different IPs. The browser result is an IPv4 IP and the ytmusicapi result is an IPv6 ip. Strangely enough, the IPv4 IP is a foreign IP with ping around 40ms, while the IPv6 IP is my very own IP. What do those IPs look like for you?

@KoljaWindeler
Copy link
Author

KoljaWindeler commented May 19, 2021

Exactly .. I open the resulting googlemusic url .. one leads to .. well the MP4 the other to a 403.
The 31.18. ... Is my ip

Edit: I guess none of the urls are working for you right? The browser based link is connected to my ip and the other one won't work due to the bug. But you should be able to fetch a signatureCiphered from the website which would then generate a valid link for a given videoId. At least that's how I came up with the data

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Sorry I got a bit confused in your script there...

It seems it's indeed related to the missing POST data I included above. Will commit a fix shortly

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Unfortunately I'm not sure what the signatureTimestamp is based on, but it was always 18766 for me, even when not signed in.

@KoljaWindeler
Copy link
Author

So the additional information fixes the issue for you? In other words:
Can you open the generated link and will it play music for you? Will test right after dinner!!

@sigma67
Copy link
Owner

sigma67 commented May 19, 2021

Yes, I verified that now :)

@KoljaWindeler
Copy link
Author

here as well .. great. could you publish a new version so we can roll this out to all others?
Thanks a thousand times! great work!

@sigma67
Copy link
Owner

sigma67 commented May 20, 2021

Closed in 8c4b857

@sigma67 sigma67 closed this as completed May 20, 2021
@simu
Copy link

simu commented May 21, 2021

Playback (I'm using ytmusicapi via mopidy_ytmusic) broke again for me today (song URLs result in 403), and inspecting the requests on the web app showed that signatureTimestamp has changed to 18767 for me now. Adjusting the timestamp in the code fixes playback.

@KoljaWindeler
Copy link
Author

Same here

@sigma67
Copy link
Owner

sigma67 commented May 21, 2021

Incrementing by one after a day seems very suspicious. Turns out it is derived from the UNIX datestamp! The datestamp is the number of days since 01-01-1970, which is currently 18768. If you decrement that by one you get the signatureTimestamp 18767 that's currently being used.

Strangely enough, about 43 hours ago (time of my commit), the timestamp was 18766, so that would imply that the timestamp had just incremented in the 5 hours before that time. That would mean that the increment time is not UTC midnight, as it would be if it was derived from the UNIX timestamp. Perhaps we can monitor in the coming hours when the signatureTimestamp changes, so we can avoid similar issues in the future.

@simu
Copy link

simu commented May 21, 2021

Turns out it is derived from the UNIX datestamp! The datestamp is the number of days since 01-01-1970, which is currently 18768. If you decrement that by one you get the signatureTimestamp 18767 that's currently being used.

I figured this out in the mean time as well, but I haven't been able to pinpoint the increment time yet. I'm currently running a patched version of get_song() which uses signatureTimestamp = (date.today() - date.fromtimestamp(0)).days - 1 (date is from datetime import date).

@sigma67 sigma67 reopened this May 21, 2021
@KoljaWindeler
Copy link
Author

KoljaWindeler commented May 21, 2021

UTC15:55 still 18767
UTC16:34 still 18767

@impliedchaos
Copy link
Contributor

I'm not sure how viable of a strategy it is to do it based on time. The signatureTimestamp is set in the base.js version. This doesn't change at the same time everyday. My application tries to fetch the base.js url every 15 minutes:

2021-05-18 19:22:34,717 player URL /s/player/08244190/player_ias.vflset/en_US/base.js
2021-05-18 19:37:36,302 player URL /s/player/ae091644/player_ias.vflset/en_US/base.js

2021-05-19 07:08:48,142 player URL /s/player/ae091644/player_ias.vflset/en_US/base.js
2021-05-19 07:23:49,798 player URL /s/player/fba90263/player_ias.vflset/en_US/base.js

2021-05-20 19:14:43,799 player URL /s/player/fba90263/player_ias.vflset/en_US/base.js
2021-05-20 19:29:45,302 player URL /s/player/3d0175c7/player_ias.vflset/en_US/base.js

(times are UTC+4)

@sigma67
Copy link
Owner

sigma67 commented May 21, 2021

Seems it isn't a fixed interval but instead depends on when the server script gets updated.

The signatureTimestamp value is hardcoded (probably dynamically generated by the server) in the base.js player script retrieved from music.youtube.com. Looking at other apps like YoutubeExplode they get the signatureTimestamp by parsing the player script retrieved from the watch page. That's at least two additional requests though, one to get the page with the URL of the player script, another to get the player script, from which you can then retrieve the signatureTimestamp.

Another alternative could be to try the datestamp - 1 method by default and fall back to datestamp if it fails. Then we'd need to verify the correctness of the URL though, by checking for 403 when calling the response URL. Or make signatureTimestamp an optional parameter to get_song.

This is under the assumption that the server will only accept the signatureTimestamp in the most current player script, which seems to be the case from what we've seen.

Edit: @impliedchaos seems we commented at the same time but have a similar assessment of the situation 👍

@KoljaWindeler
Copy link
Author

Well .. grabbing the signatureTimestamp wouldn't be necessary every single time. One could grab it, set it, decode the url, check the header and refetch if the header returns a 403 ...

@impliedchaos
Copy link
Contributor

Or make signatureTimestamp an optional parameter to get_song.

This would be my personal preference. And if the parameter isn't passed, then use the datestamp - 1 logic.

@KoljaWindeler
Copy link
Author

KoljaWindeler commented May 21, 2021

jep, something like this https://pastebin.com/sxbwa8Ee just with 37 instead of 38
edit: just because it's quite fancy .. during testing I get most of the time 18767 but I've already seen 18768 twice ..

sigma67 pushed a commit that referenced this issue May 21, 2021
@sigma67
Copy link
Owner

sigma67 commented May 21, 2021

I've added a quick fix along the lines of what was suggested above. I'd gladly accept a PR for a get_signatureTimestamp helper method since you guys seem to know more about that part than me

sigma67 pushed a commit that referenced this issue May 21, 2021
@KoljaWindeler
Copy link
Author

the current change totally works for me .. just tested .. i guess adding the lookup into the library wouldn't really help because it would heavily slow down the function .. would you consider to release this change?

@sigma67
Copy link
Owner

sigma67 commented May 22, 2021

Published a new release.

i guess adding the lookup into the library wouldn't really help because it would heavily slow down the function

I was thinking more of making a separate method that can be called to get the signatureTimestamp, so others don't have to reimplement that part

@KoljaWindeler
Copy link
Author

Fair enough. Will push a PR based on my stuff tomorrow

@KoljaWindeler
Copy link
Author

works just perfect here .. I guess we can close this once more
Thanks a lot guys!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
yt-update A server-side change caused this issue
Projects
None yet
Development

No branches or pull requests

4 participants