Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some valid cookies cause youtube-dl to break #24300

Closed
EhsanCh opened this issue Mar 9, 2020 · 3 comments
Closed

Some valid cookies cause youtube-dl to break #24300

EhsanCh opened this issue Mar 9, 2020 · 3 comments
Labels

Comments

@EhsanCh
Copy link

@EhsanCh EhsanCh commented Mar 9, 2020

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dl version 2020.03.08
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

[debug] System config: [u'-4', u'--cookies', u'/home/youtube-cookie.txt']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.instagram.com/p/B9g4ilHJOv7/']
/usr/lib64/python2.7/_MozillaCookieJar.py:109: UserWarning: cookielib bug!
Traceback (most recent call last):
  File "/usr/lib64/python2.7/_MozillaCookieJar.py", line 71, in _really_load
    line.split("\t")
ValueError: too many values to unpack

  _warn_unhandled_exception()
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main
  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 441, in _real_main
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 419, in __init__
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2329, in _setup_opener
  File "/usr/bin/youtube-dl/youtube_dl/utils.py", line 2762, in load
  File "/usr/lib64/python2.7/_MozillaCookieJar.py", line 111, in _really_load
    (filename, line))
cookielib.LoadError: invalid Netscape format cookies file u'/home/youtube-cookie.txt': u'.instagram.com\tTRUE\t/\tTRUE\t0\turlgen\t"{\\"45.12.3.45\\":\t24940}:1jBJ5c:yMVdw1trErKtMZlJkGK3zfvFI4U"'

Description

According to RFC 6265 , white space allowed in cookie values with double quote:

4.1.1. Syntax

Informally, the Set-Cookie response header contains the header name
"Set-Cookie" followed by a ":" and a cookie. Each cookie begins with
a name-value-pair, followed by zero or more attribute-value pairs.
Servers SHOULD NOT send Set-Cookie headers that fail to conform to
the following grammar:
set-cookie-header = "Set-Cookie:" SP set-cookie-string
set-cookie-string = cookie-pair *( ";" SP cookie-av )
cookie-pair = cookie-name "=" cookie-value
cookie-name = token
cookie-value = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
; US-ASCII characters excluding CTLs,
; whitespace DQUOTE, comma, semicolon,
; and backslash

if you don't fix this issue, youtube-dl will break after downloading any link from Instagram or other sites that send cookies with space. it can be used by a mechanism to break youtube-dl.

NOTE: i re-created this issue because #24299 closed without acceptable response !!!
When downloading Instagram video, it sent a cookie like :
"{\"45.12.3.45\": 24940}:1jBJ5c:yMVdw1trErKtMZlJkGK3zfvFI4U"
and write a line in cookie file that cause cookie file to be invalid for next youtube-dl run.
Instagram does not send invalid cookie and its cookies are accepted in all major browsers like Chrome of Firefox.

@harryt04
Copy link

@harryt04 harryt04 commented Mar 9, 2020

I'm getting this same issue with youtube-dl version 2020.03.08

/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/cookiejar.py:2076: UserWarning: http.cookiejar bug!
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/cookiejar.py", line 2038, in _really_load
    line.split("\t")
ValueError: too many values to unpack (expected 7)

  _warn_unhandled_exception()
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/cookiejar.py", line 2038, in _really_load
    line.split("\t")
ValueError: too many values to unpack (expected 7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/__init__.py", line 441, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 419, in __init__
    self._setup_opener()
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2329, in _setup_opener
    self.cookiejar.load(ignore_discard=True, ignore_expires=True)
  File "/usr/local/lib/python3.7/site-packages/youtube_dl/utils.py", line 2762, in load
    self._really_load(cf, filename, ignore_discard, ignore_expires)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/cookiejar.py", line 2078, in _really_load
    (filename, line))
http.cookiejar.LoadError: invalid Netscape format cookies file './drew.txt': 'blytzpay.udemy.com\tFALSE\t/\tFALSE\t1628122050\tud_last_auth_information\t"{\\"backend\\":\t\\"udemy-auth\\"\\054\t\\"suggested_user_email\\":\t\\"drew.royster@blytzpay.com\\"\\054\t\\"suggested_user_name\\":\t\\"Drew\\"\\054\t\\"suggested_user_avatar\\":\t\\"https://i.udemycdn.com/user/50x50/anonymous_3.png\\"}:1izrBB:SRdOKBOmvPVX6TBhhA56N3gW62s"'
@harryt04
Copy link

@harryt04 harryt04 commented Mar 9, 2020

Downgrading to version 2020.03.06 resolves my issue. Although a proper fix moving forward would be nice.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 9, 2020

For those who have troubles reading english:

cookie-octet = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
; US-ASCII characters excluding CTLs,
; whitespace DQUOTE, comma, semicolon,
; and backslash

Space is %x20.

Next, comment clearly states:

; US-ASCII characters excluding CTLs,
; whitespace DQUOTE, comma, semicolon,
; and backslash

and with errata 5518 applied:

; US-ASCII characters excluding CTLs,
; whitespace, DQUOTE, comma, semicolon,
; and backslash

US-ASCII characters excluding CTLs, whitespace, DQUOTE, comma, semicolon, and backslash

excluding whitespace

@dstftw dstftw closed this Mar 9, 2020
@dstftw dstftw added the invalid label Mar 9, 2020
@ytdl-org ytdl-org locked and limited conversation to collaborators Mar 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.