[youtube] Fix throttling by decrypting n-sig #1437

pukkandan · 2021-10-27T03:46:34Z

Overhauls jsinterp to handle youtube nsig decryption
Fixes: ytdl-org/youtube-dl#29326

TODO:

Check fallbacks and ensure all failures are non-fatal
Add tests
Refactor/cleanup

shoxie007 · 2021-10-27T06:03:15Z

Amazing. Will this be the final and explicit solution to the throttling issue, rather than a mere workaround like the android-client trick?

coletdjnz · 2021-10-27T06:06:38Z

Amazing. Will this be the final and explicit solution to the throttling issue, rather than a mere workaround like the android-client trick?

Yep! 😉

coletdjnz

Tested it works on the other js-based clients so far (web_embedded, mweb, web_music, web_creator)

I think for the time-being we should keep the android client, but as a secondary client by default (i.e. player_client=web,android). But we'll need to rework the skipping formats that already exist code to account for that.

yt_dlp/extractor/youtube.py

coletdjnz · 2021-10-27T05:26:22Z

yt_dlp/extractor/youtube.py

-                    fmt.get('qualityLabel') or quality.replace('audio_quality_', '')))),
+                    fmt.get('qualityLabel') or quality.replace('audio_quality_', ''),
+                    ' (throttled)' if not n_sig else None))),
+                'source_preference': -10 if not n_sig else -1,


Maybe we could look into overwriting an existing format with same itag if the source preference is higher (or something similar)?

e.g. we use player_client=web,android. If we fail to decrypt nsig for web, the throttled web formats have lower preference than the equivalent formats on android.

~~It shouldn't be too hard to add~~ (actually, it is a bit tricky), but if we are keeping the android fallback, why not keep android,web iself?

Hmm yeah if it's going to be too much trouble than it's worth probably best to just keep android,web

yt_dlp/extractor/youtube.py

pukkandan · 2021-10-27T08:49:55Z

yt_dlp/extractor/youtube.py

+            self.to_screen(f'Extracted nsig function from {player_id}:\n{func_code[1]}\n')
+
+        return lambda s: jsi.extract_function_from_code(*func_code)([s])
+
    def _extract_signature_timestamp(self, video_id, player_url, ytcfg=None, fatal=False):


@coletdjnz Do you know why this isnt cached to disk like the signature?

Like how the nsig is caching the code to the disk, whereas the sig is caching some characters?

I'm not actually sure. Unless I'm reading this wrong (it should really have been documented), it seems like they cache some sort of character mapping/offset to the disk?

yt-dlp/yt_dlp/extractor/youtube.py

Lines 1743 to 1755 in b7b186e

if cache_spec is not None:

return lambda s: ''.join(s[i] for i in cache_spec)

if self._load_player(video_id, player_url):

code = self._code_cache[player_id]

res = self._parse_sig_js(code)

test_string = ''.join(map(compat_chr, range(len(example_sig))))

cache_res = res(test_string)

cache_spec = [ord(c) for c in cache_res]

self._downloader.cache.store('youtube-sigfuncs', func_id, cache_spec)

return res

I don't understand it either. Hence why for nsig, I went with storing the actual js function itself. For _extract_signature_timestamp I assume we can directly cache the sts itself.

_extract_signature_timestamp I assume we can directly cache the sts itself.

I never actually thought about caching that, but yeah we could do that as well. Though I think it would currently only help in the case where the webpage is not downloaded, but the iframe fallback is (usually happens when 429nd).

I had another look at it:
For the normal signature function, it seems to be reversing the encrypted signature, but 'shuffling' some of the characters with replacement (so using same characters and same length). Hence being able to cache a mapping of the character positions?

But with the nsig it's different - the decrypted nsig seems to have different characters and length than the encrypted nsig.

yt_dlp/extractor/youtube.py

* master: (28 commits) [ffmpeg] Detect libavformat version for `aac_adtstoasc` and print available features in verbose head Based on ytdl-org/youtube-dl#29581 [ffmpeg] Accurately detect presence of setts [ExtractAudio] Use `libfdk_aac` if available Closes yt-dlp#1502 Authored by: CrypticSignal [ffmpeg] Framework for feature detection Related: yt-dlp#1502, yt-dlp#1237, ytdl-org/youtube-dl#29581 [ExtractAudio] Rescale --audio-quality correctly Authored by: CrypticSignal, pukkandan [fragment] Fix progress display in fragmented downloads Closes yt-dlp#1517 [youtube] Remove unnecessary no-playlist warning [utils] Parse `vp09` as vp9 [jsinterp] Handle default in switch better [Instagram] Fix incorrect resolution (yt-dlp#1494) [vk] Fix login (yt-dlp#1495) [docs,cleanup] Improve docs and minor cleanup Closes yt-dlp#1387, yt-dlp#1404, yt-dlp#1408, yt-dlp#1485, yt-dlp#1415, yt-dlp#1450, yt-dlp#1492 [youtube] refactor itag processing [linkedin] Don't login multiple times [vk] Add subtitles (yt-dlp#1480) [Olympics] Fix extractor (yt-dlp#1483) [PlanetMarathi] Add extractor (yt-dlp#1484) [Instagram] Add login to playlist (yt-dlp#1488) [ceskatelevize] Fix extractor (yt-dlp#1489) [youtube] Fix throttling by decrypting n-sig (yt-dlp#1437) ...

Initial implementation

5b04c74

coletdjnz reviewed Oct 27, 2021

View reviewed changes

coletdjnz linked an issue Oct 27, 2021 that may be closed by this pull request

"This video is not available" on some youtube videos #1380

Closed

7 tasks

pukkandan commented Oct 27, 2021

View reviewed changes

Lesmiscore reviewed Oct 27, 2021

View reviewed changes

yt_dlp/extractor/youtube.py Show resolved Hide resolved

cleanup youtube

ec89afd

pukkandan force-pushed the nsig_decrypt branch from 5dee610 to ec89afd Compare October 27, 2021 12:49

cleanup jsinterp

1871799

pukkandan marked this pull request as ready for review October 29, 2021 16:55

pukkandan force-pushed the nsig_decrypt branch from 5b7ba21 to 6eb0a33 Compare October 30, 2021 15:28

pukkandan added 3 commits October 31, 2021 09:19

Add test and fix linter

0e01ccf

fix overextending function body

c7f62fb

jsinterp tests

88227c3

pukkandan force-pushed the nsig_decrypt branch from 6eb0a33 to 88227c3 Compare October 31, 2021 04:19

pukkandan merged commit 404f611 into yt-dlp:master Oct 31, 2021

This comment has been minimized.

Sign in to view

coletdjnz mentioned this pull request Oct 31, 2021

[YouTube] Randomly slow youtube download speed ytdl-org/youtube-dl#29326

Closed

Leseratte10 mentioned this pull request Oct 31, 2021

[Feature Request] add zspotify as spotify downloader! #1368

Closed

5 tasks

dirkf mentioned this pull request Nov 2, 2021

[JSInterp] Back-port JS interpreter upgrade from yt-dlp PR #1437 ytdl-org/youtube-dl#30188

Closed

11 tasks

itssidhere mentioned this pull request Nov 8, 2021

[BUG] YouTube throttling videos Hexer10/youtube_explode_dart#180

Closed

dirkf mentioned this pull request Dec 17, 2021

[YouTube] Unthrottle downloads by responding to the "n" parameter challenge ytdl-org/youtube-dl#30184

Closed

11 tasks

pukkandan deleted the nsig_decrypt branch January 29, 2022 20:16

pukkandan mentioned this pull request Mar 3, 2023

[youtube] Download speeds still throttled for some clients after range fix #6400

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[youtube] Fix throttling by decrypting n-sig #1437

[youtube] Fix throttling by decrypting n-sig #1437

pukkandan commented Oct 27, 2021 •

edited

shoxie007 commented Oct 27, 2021

coletdjnz commented Oct 27, 2021

coletdjnz left a comment •

edited

coletdjnz Oct 27, 2021

pukkandan Oct 27, 2021 •

edited

coletdjnz Oct 29, 2021

pukkandan Oct 27, 2021 •

edited

coletdjnz Oct 27, 2021 •

edited

pukkandan Oct 29, 2021

coletdjnz Oct 29, 2021

coletdjnz Oct 29, 2021 •

edited

This comment has been minimized.

	if cache_spec is not None:
	return lambda s: ''.join(s[i] for i in cache_spec)

	if self._load_player(video_id, player_url):
	code = self._code_cache[player_id]
	res = self._parse_sig_js(code)

	test_string = ''.join(map(compat_chr, range(len(example_sig))))
	cache_res = res(test_string)
	cache_spec = [ord(c) for c in cache_res]

	self._downloader.cache.store('youtube-sigfuncs', func_id, cache_spec)
	return res

[youtube] Fix throttling by decrypting n-sig #1437

[youtube] Fix throttling by decrypting n-sig #1437

Conversation

pukkandan commented Oct 27, 2021 • edited

shoxie007 commented Oct 27, 2021

coletdjnz commented Oct 27, 2021

coletdjnz left a comment • edited

Choose a reason for hiding this comment

coletdjnz Oct 27, 2021

Choose a reason for hiding this comment

pukkandan Oct 27, 2021 • edited

Choose a reason for hiding this comment

coletdjnz Oct 29, 2021

Choose a reason for hiding this comment

pukkandan Oct 27, 2021 • edited

Choose a reason for hiding this comment

coletdjnz Oct 27, 2021 • edited

Choose a reason for hiding this comment

pukkandan Oct 29, 2021

Choose a reason for hiding this comment

coletdjnz Oct 29, 2021

Choose a reason for hiding this comment

coletdjnz Oct 29, 2021 • edited

Choose a reason for hiding this comment

This comment has been minimized.

pukkandan commented Oct 27, 2021 •

edited

coletdjnz left a comment •

edited

pukkandan Oct 27, 2021 •

edited

pukkandan Oct 27, 2021 •

edited

coletdjnz Oct 27, 2021 •

edited

coletdjnz Oct 29, 2021 •

edited