--twofactor ignored under youtube:playlist and youtube:watchlater extractors

[effleurager]

youtube:playlist & youtube:watchlater do not process two factor authentication codes when presented with the follow-up dialogue to signing in.

[debug] System config: []
[debug] User config: [u'--add-metadata', u'-o', u'%(upload_date)s - %(title)s.%(ext)s', u'-c', u'-f', u'bestvideo[ext=mp4][width<=1920]+bestaudio[ext=m4a]', u'--embed-thumbnail', u'--embed-subs', u'--exec', u'IFS=$n && j={} && k=`expr length $j` && l=`expr $k - 15` && notify-send "YouTube DL" --icon /usr/local/bin/icon.png "`expr substr $j 12 $l`"']
[debug] Command-line args: [u'https://www.youtube.com/playlist?list=PLARhCYccw3tb0ewDzW4eE28DNc9V_oIZE', u'-n', u'--twofactor', u'780848', u'-v', u'--write-pages']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.07.21
[debug] Python version 2.7.9 - Linux-3.19.0-23-generic-x86_64-with-Ubuntu-15.04-vivid
[debug] exe versions: ffmpeg N-73160-g2c77ca4, ffprobe N-73160-g2c77ca4
[debug] Proxy map: {}
[youtube:playlist] Downloading login page
[youtube:playlist] Saving request to None_https_-_accounts.google.com_ServiceLogin.dump
[youtube:playlist] Logging in
[youtube:playlist] Saving request to None_https_-_accounts.google.com_ServiceLogin.dump
[youtube:playlist] PLARhCYccw3tb0ewDzW4eE28DNc9V_oIZE: Downloading webpage
[youtube:playlist] Saving request to PLARhCYccw3tb0ewDzW4eE28DNc9V_oIZE_https_-_www.youtube.com_playlistlist=PLARhCYccw3tb0ewDzW4eE28DNc9V_oIZE.dump
ERROR: The playlist doesn't exist or is private, use --username or --netrc to access it.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 656, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 275, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1452, in _real_extract
    return self._extract_playlist(playlist_id)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1389, in _extract_playlist
    expected=True)
ExtractorError: The playlist doesn't exist or is private, use --username or --netrc to access it.

[debug] System config: []
[debug] User config: [u'--add-metadata', u'-o', u'%(upload_date)s - %(title)s.%(ext)s', u'-c', u'-f', u'bestvideo[ext=mp4][width<=1920]+bestaudio[ext=m4a]', u'--embed-thumbnail', u'--embed-subs', u'--exec', u'IFS=$n && j={} && k=`expr length $j` && l=`expr $k - 15` && notify-send "YouTube DL" --icon /usr/local/bin/icon.png "`expr substr $j 12 $l`"']
[debug] Command-line args: [u':ytwatchlater', u'-v', u'-n', u'--twofactor', u'630575']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.07.21
[debug] Python version 2.7.9 - Linux-3.19.0-23-generic-x86_64-with-Ubuntu-15.04-vivid
[debug] exe versions: ffmpeg N-73160-g2c77ca4, ffprobe N-73160-g2c77ca4
[debug] Proxy map: {}
[youtube:watchlater] Downloading login page
[youtube:watchlater] Logging in
[youtube:watchlater] WL: Downloading webpage
ERROR: The playlist doesn't exist or is private, use --username or --netrc to access it.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 656, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 275, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1765, in _real_extract
    return self._extract_playlist('WL')
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1389, in _extract_playlist
    expected=True)
ExtractorError: The playlist doesn't exist or is private, use --username or --netrc to access it.

[reddraggone9]

I'm experiencing this problem as well. I temporarily turned off two-factor authentication and :ytwatchlater worked fine. After a little poking around youtube_dl/extractor/youtube.py, it looks like the page structure has indeed changed. I'm not really sure what I'm doing, but here's the diff including the changes I made. Sorry for the mess.

diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
index facd837..f07b4de 100644
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -46,7 +46,7 @@ from ..utils import (
 class YoutubeBaseInfoExtractor(InfoExtractor):
     """Provide base functions for Youtube extractors"""
     _LOGIN_URL = 'https://accounts.google.com/ServiceLogin'
-    _TWOFACTOR_URL = 'https://accounts.google.com/SecondFactor'
+    _TWOFACTOR_URL = 'https://accounts.google.com/signin/challenge'
     _NETRC_MACHINE = 'youtube'
     # If True it will raise an error if no login info is provided
     _LOGIN_REQUIRED = False
@@ -128,7 +128,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
         # Two-Factor
         # TODO add SMS and phone call support - these require making a request and then prompting the user

-        if re.search(r'(?i)<form[^>]* id="gaia_secondfactorform"', login_results) is not None:
+        if re.search(r'(?i)<form[^>]* id="challenge"', login_results) is not None:
             tfa_code = self._get_tfa_info()

             if tfa_code is None:
@@ -136,31 +136,27 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 self._downloader.report_warning('(Note that only TOTP (Google Authenticator App) codes work at this time.)')
                 return False

-            # Unlike the first login form, secTok and timeStmp are both required for the TFA form
+            def find_value(id):
+                match = re.search(r'id="%s"\s+value="(.+)">' % id, login_results, re.M | re.U)
+                if match is None:
+                    self._downloader.report_warning('Failed to get %s - did the page structure change?' % id)
+                return match.group(1)

-            match = re.search(r'id="secTok"\n\s+value=\'(.+)\'/>', login_results, re.M | re.U)
-            if match is None:
-                self._downloader.report_warning('Failed to get secTok - did the page structure change?')
-            secTok = match.group(1)
-            match = re.search(r'id="timeStmp"\n\s+value=\'(.+)\'/>', login_results, re.M | re.U)
-            if match is None:
-                self._downloader.report_warning('Failed to get timeStmp - did the page structure change?')
-            timeStmp = match.group(1)
+            challengeId = find_value("challengeId")
+            challengeType = find_value("challengeType")
+            gxf = find_value("gxf")

             tfa_form_strs = {
                 'continue': 'https://www.youtube.com/signin?action_handle_signin=true&feature=sign_in_button&hl=en_US&nomobiletemp=1',
-                'smsToken': '',
-                'smsUserPin': tfa_code,
-                'smsVerifyPin': 'Verify',
-
-                'PersistentCookie': 'yes',
-                'checkConnection': '',
                 'checkedDomains': 'youtube',
-                'pstMsg': '1',
-                'secTok': secTok,
-                'timeStmp': timeStmp,
+                'pstMsg': '0',
                 'service': 'youtube',
                 'hl': 'en_US',
+                'challengeId': challengeId,
+                'challengeType': challengeType, # This doesn't appear to change
+                'gxf': gxf,
+                'Pin': tfa_code,
+                'TrustDevice': 'on',
             }
             tfa_form = dict((k.encode('utf-8'), v.encode('utf-8')) for k, v in tfa_form_strs.items())
             tfa_data = compat_urllib_parse.urlencode(tfa_form).encode('ascii')
@@ -173,7 +169,8 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
             if tfa_results is False:
                 return False

-            if re.search(r'(?i)<form[^>]* id="gaia_secondfactorform"', tfa_results) is not None:
+            # TODO What is the correct value for these?
+            if re.search(r'(?i)<form[^>]* id="challenge"', tfa_results) is not None:
                 self._downloader.report_warning('Two-factor code expired. Please try again, or use a one-use backup code instead.')
                 return False
             if re.search(r'(?i)<form[^>]* id="gaia_loginform"', tfa_results) is not None:

[reddraggone9]

To be clear, that diff is not a fix, just where I got to so far by saving the received page and using Firefox's network monitor to see what happened when the form was manually submitted. Somebody with more of a clue may or may not find it useful.

[ironiridis]

Is this issue still unresolved? It could probably be closed.