download with cookie file issue

[ciboo]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.09.04.1. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2016.09.04.1

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl --no-cache-dir --cookies ~/Downloads/cookies.txt --print-traffic --dump-pages --verbose https://www.permaskills.net/opening-credits/ 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--no-cache-dir', u'--cookies', u'/Users/REDACTED_USER/Downloads/cookies.txt', u'--print-traffic', u'--dump-pages', u'--verbose', u'https://www.permaskills.net/opening-credits/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.04.1
[debug] Python version 2.7.10 - Darwin-15.6.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.1.3, ffprobe 3.1.3
[debug] Proxy map: {}
[generic] opening-credits: Requesting header
send: u'HEAD /opening-credits/ HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.permaskills.net\r\nCookie: aelia_cs_selected_currency=USD; _icl_current_language=en; wp_woocommerce_session_REDACTED_SESSION=0c0083348774aaaf2bf5d23f5476779e%7C%7C1473230243%7C%7C1473226643%7C%7Cc7751da811dbf96efaa7d7cdad749892; CookieInfo=accepted; _sp_id.5f60=d0ade15174846918.1472473356.21.1473160800.1473154299; _sp_ses.5f60=*\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Tue, 06 Sep 2016 11:21:40 GMT
header: Server: Apache
header: Set-Cookie: _icl_current_language=en; expires=Wed, 07-Sep-2016 11:21:40 GMT; path=/
header: Set-Cookie: aelia_cs_selected_currency=USD; expires=Wed, 07-Sep-2016 11:21:40 GMT; path=/
header: X-Pingback: https://www.permaskills.net/V1/xmlrpc.php
header: Expires: Wed, 11 Jan 1984 05:00:00 GMT
header: Cache-Control: no-cache, must-revalidate, max-age=0
header: Pragma: no-cache
header: Vary: Accept-Encoding,User-Agent
header: Content-Encoding: gzip
header: Content-Length: 20
header: Connection: close
header: Content-Type: text/html; charset=UTF-8
WARNING: Could not send HEAD request to https://www.permaskills.net/opening-credits/: HTTP Error 404: Not Found
[generic] opening-credits: Downloading webpage
send: u'GET /opening-credits/ HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: *\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.permaskills.net\r\nCookie: aelia_cs_selected_currency=USD; _icl_current_language=en; wp_woocommerce_session_REDACTED_SESSION=0c0083348774aaaf2bf5d23f5476779e%7C%7C1473230243%7C%7C1473226643%7C%7Cc7751da811dbf96efaa7d7cdad749892; CookieInfo=accepted; _sp_id.5f60=d0ade15174846918.1472473356.21.1473160800.1473154299; _sp_ses.5f60=*\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Tue, 06 Sep 2016 11:21:41 GMT
header: Server: Apache
header: Set-Cookie: _icl_current_language=en; expires=Wed, 07-Sep-2016 11:21:42 GMT; path=/
header: Set-Cookie: aelia_cs_selected_currency=USD; expires=Wed, 07-Sep-2016 11:21:42 GMT; path=/
header: X-Pingback: https://www.permaskills.net/V1/xmlrpc.php
header: Expires: Wed, 11 Jan 1984 05:00:00 GMT
header: Cache-Control: no-cache, must-revalidate, max-age=0
header: Pragma: no-cache
header: Vary: Accept-Encoding,User-Agent
header: Connection: close
header: Transfer-Encoding: chunked
header: Content-Type: text/html; charset=UTF-8
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 394, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1998, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 434, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 427, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 506, in __exit__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_MozillaCookieJar.py", line 118, in save
    f = open(filename, "w")
IOError: [Errno 13] Permission denied: u'/Users/REDACTED_USER/Downloads/cookies.txt'

---

Description of your issue, suggested solution and other information

Using the same cookies.txt file with both youtube-dl and curl I noticed that:

1. curl's request is successful (HTTP status 200 and page downloaded)
2. youtube-dl requests fails (HTTP status 404)
3. youtube-dl is not sending the cookie wordpress_logged_in_REDACTED_SESSION which is probably why the server is rejecting the request as if the client is not authenticated
4. The cookies.txt is read-only to prevent youtube-dl to overwrite the file (hence the last error youtube-dl is reporting)

Nothe that I've replaced some sensitive data with the REDACTED text.

Please let me know if you need more information.

Thank you!

cookies.txt contents:

# HTTP Cookie File
# To download this content click here, press return (Mac only) or copy and paste into a cookies.txt file.
# It can be then used by wget and other standard complaint tools.
# Example:  wget -x --load-cookies cookies.txt https://www.permaskills.net/opening-credits/
#
www.permaskills.net FALSE   /V1/wp-content/plugins  TRUE    0   wordpress_sec_REDACTED_SESSION  REDACTED%7C1473246938%7Choha9754IixUWiPSTU5T8xoGUVk6e0UBQtoIqfI00uj%7C148e38dbe6ac43021dd5fb443e501897e91db0a2cef86dfc36ae60ed7acfd33c
www.permaskills.net FALSE   /V1/wp-admin/   FALSE   1473167939  _icl_current_admin_language en
www.permaskills.net FALSE   /V1/wp-admin    TRUE    0   wordpress_sec_REDACTED_SESSION  REDACTED%7C1473246938%7Choha9754IixUWiPSTU5T8xoGUVk6e0UBQtoIqfI00uj%7C148e38dbe6ac43021dd5fb443e501897e91db0a2cef86dfc36ae60ed7acfd33c
www.permaskills.net FALSE   /V1/    TRUE    0   wordpress_logged_in_REDACTED_SESSION    REDACTED%7C1473246938%7Choha9754IixUWiPSTU5T8xoGUVk6e0UBQtoIqfI00uj%7C170e0688f71a9acd757b8cb4e06abc1f17c7c2316973ccc3464939b10ef6798e
www.permaskills.net FALSE   /   FALSE   1480441635  CookieInfo  accepted
www.permaskills.net FALSE   /   FALSE   1473230256  wp_woocommerce_session_REDACTED_SESSION 0c0083348774aaaf2bf5d23f5476779e%7C%7C1473230243%7C%7C1473226643%7C%7Cc7751da811dbf96efaa7d7cdad749892
www.permaskills.net FALSE   /   TRUE    0   wordpress_logged_in_REDACTED_SESSION    REDACTED%7C1473246938%7Choha9754IixUWiPSTU5T8xoGUVk6e0UBQtoIqfI00uj%7C170e0688f71a9acd757b8cb4e06abc1f17c7c2316973ccc3464939b10ef6798e
www.permaskills.net FALSE   /   FALSE   1473247199  _icl_current_language   en
www.permaskills.net FALSE   /   FALSE   1536232799  _sp_id.5f60 d0ade15174846918.1472473356.21.1473160800.1473154299
www.permaskills.net FALSE   /   FALSE   1473162599  _sp_ses.5f60    *
www.permaskills.net FALSE   /   FALSE   1473247202  aelia_cs_selected_currency  USD

curl output:

curl -v -b cookies.txt https://www.permaskills.net/opening-credits/
*   Trying 207.210.203.1...
* Connected to www.permaskills.net (207.210.203.1) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: www.permaskills.net
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA RootREDACTED
> GET /opening-credits/ HTTP/1.1
> Host: www.permaskills.net
> User-Agent: curl/7.43.0
> Accept: */*
> Cookie: CookieInfo=accepted; _icl_current_language=en; _sp_id.5f60=d0ade15174846918.1472473356.21.1473160800.1473154299; _sp_ses.5f60=*; aelia_cs_selected_currency=USD; wordpress_logged_in_REDACTED_SESSION=REDACTED%7C1473246938%7Choha9754IixUWiPSTU5T8xoGUVk6e0UBQtoIqfI00uj%7C170e0688f71a9acd757b8cb4e06abc1f17c7c2316973ccc3464939b10ef6798e; wp_woocommerce_session_REDACTED_SESSION=0c0083348774aaaf2bf5d23f5476779e%7C%7C1473230243%7C%7C1473226643%7C%7Cc7751da811dbf96efaa7d7cdad749892
> 
< HTTP/1.1 200 OK
< Date: Tue, 06 Sep 2016 11:20:59 GMT
< Server: Apache
* Replaced cookie _icl_current_language="en" for domain www.permaskills.net, path /, expire 1473247259
< Set-Cookie: _icl_current_language=en; expires=Wed, 07-Sep-2016 11:20:59 GMT; path=/
* Replaced cookie aelia_cs_selected_currency="USD" for domain www.permaskills.net, path /, expire 1473247259
< Set-Cookie: aelia_cs_selected_currency=USD; expires=Wed, 07-Sep-2016 11:20:59 GMT; path=/
< X-Pingback: https://www.permaskills.net/V1/xmlrpc.php
< Expires: Wed, 11 Jan 1984 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0
< Pragma: no-cache
< Link: <https://www.permaskills.net/?p=986>; rel=shortlink
< Vary: Accept-Encoding,User-Agent
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8

[dstftw]

wordpress_logged_in* cookies are discarded because they have expiry time = 0 and considered expired (0 < now).

[ciboo]

Is there a workaround for this?

[dstftw]

Set non-expired expiry time.

[ciboo]

It's working, thanks :)