SSLError when windows cert store contains bad certificates #1060

MDM-79 · 2021-09-23T01:38:21Z

In latest mpc-hc 1.9.16 (that just added support for yt-dlp.exe) it does not work with latest yt-dlp.exe 2021.09.02.
This is the error I get:

I tried even the x86 file, full or short YT address, file in MPC folder or not, name only, no mater if the path/file name is specified or not, or where the file is... - not once it worked.
I was pointed here by clsid2: clsid2/mpc-hc#1320
Hope you can help'

pukkandan · 2021-09-23T06:13:34Z

please run yt-dlp -Uv in a console and give me the complete output. Just the traceback is way too little info to go on

And in the future, dont ignore the issue template. It exists for a reason

MDM-79 · 2021-09-23T12:02:56Z

Sorry for the template, but I just couldn't figure what to put in it, since none fits my case.

Here is the result, and it is same as above (from the folder where I have the yt-dlp.exe file, i had to modify your command to: .\yt-dlp -Uv):

Traceback (most recent call last):
  File "yt_dlp\__main__.py", line 19, in <module>
  File "yt_dlp\__init__.py", line 776, in main
  File "yt_dlp\__init__.py", line 737, in _real_main
  File "yt_dlp\YoutubeDL.py", line 577, in __init__
  File "yt_dlp\YoutubeDL.py", line 3339, in _setup_opener
  File "yt_dlp\utils.py", line 2352, in make_HTTPS_handler
  File "ssl.py", line 750, in create_default_context
  File "ssl.py", line 574, in load_default_certs
  File "ssl.py", line 566, in _load_windows_store_certs
ssl.SSLError: not enough data: cadata does not contain a certificate (_ssl.c:4161)
[13540] Failed to execute script '__main__' due to unhandled exception!

I suppose it does not work probably because I do not have yt-dlp installed, i just placed it in the folder of MPC-HC, and expected it to work like the youtube-dl.exe did!

If it has to be installed (including Python!), then I am not so interested in using it that way, too complicated, but I can just try and test it for fun if i figure how (I see 4 ways for it, and for which download binary? And just guessing that most MPC-HC users wont go the extra trouble); I just want to watch video links in the player.

MDM-79 · 2021-09-23T12:19:42Z

Forgot to mention that I already have Python installed for the use of UBU Bios tool, but not that i know how to use it much further from a console than to update PLTable and install colorama...

pukkandan · 2021-09-23T12:20:20Z

I suppose it does not work probably because I do not have yt-dlp installed, i just placed it in the folder of MPC-HC, and expected it to work like the youtube-dl.exe did!

There is no installation needed

I'm not entirely sure what the issue is, but it looks like yt-dlp cant access the certificate store. Can you confirm whether youtube-dl still works on ur PC?

MDM-79 · 2021-09-23T12:48:01Z

Yes, youtube-dl works just fine!

I tried and successfully installed it (i think) using the python3 -m pip install --upgrade yt-dlp option, and still no go.
Yes, even the first window point to the the certificate store (if matters, I have SSL 3.0 and TLS 1.3 enabled...?).

But now I do have the console output:

[debug] Command-line config: ['-Uv']
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\yt-dlp.exe\__main__.py", line 7, in <module>
  File "C:\Program Files\Python39\lib\site-packages\yt_dlp\__init__.py", line 776, in main
    _real_main(argv)
  File "C:\Program Files\Python39\lib\site-packages\yt_dlp\__init__.py", line 737, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
  File "C:\Program Files\Python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 577, in __init__
    self._setup_opener()
  File "C:\Program Files\Python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 3339, in _setup_opener
    https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
  File "C:\Program Files\Python39\lib\site-packages\yt_dlp\utils.py", line 2352, in make_HTTPS_handler
    context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: not enough data: cadata does not contain a certificate (_ssl.c:4159)

pukkandan · 2021-09-23T12:57:32Z

In the original post, you were using the exe, correct?

Also, since I cant reproduce the issue myself, if I make a patch, would you be willing to test it out (installing with PIP)?

pukkandan · 2021-09-23T13:03:04Z

what is weird is that the relevent code has never been changed in yt-dlp https://github.com/yt-dlp/yt-dlp/blame/master/yt_dlp/utils.py#L2351-2373

MDM-79 · 2021-09-23T13:03:43Z

Correct, I used the .exe (tried the yt-dlp_x86.exe one too); and even after install tried with and without (without nothing happens in the player..).

Yes i can test it!
Just instruct me how since I already have it installed (I also if important, have run the Compile command too python3 -m pip install --upgrade pyinstaller mutagen pycryptodome websockets).

MDM-79 · 2021-09-23T13:06:32Z

what is weird is that the relevent code has never been changed in yt-dlp https://github.com/yt-dlp/yt-dlp/blame/d79323136fabc2cd72afc7c124e17797e32df514/youtube_dl/utils.py#L399-L421

Just tried again, it works just fine with youtube-dl (using the x64 player wersion - mpc-hc64.exe)...?

pukkandan · 2021-09-23T13:40:31Z

Can you try these things:

Install youtube-dl with pip install youtube-dl instead of using the exe and then run youtube-dl -v. If this works, we can confirm the issue is in fact with yt-dlp code and not caused due to some difference in python versions.
Check that python's SSL module is working correctly. For that,
a. Open python interpretter (run py or python if py doesnt work)
b. Copy paste the version that it shows so that I know exactly what version is running
c. Run these:

import ssl
len(ssl.enum_certificates('ROOT'))
len(ssl.enum_certificates('CA'))
ssl.create_default_context()

(continue to the next line even if one gives error)

PS: You should run exit() to get out of the python interpretter

➤ py
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> len(ssl.enum_certificates('ROOT'))
47
>>> len(ssl.enum_certificates('CA'))
9
>>> ssl.create_default_context()
<ssl.SSLContext object at 0x000001DA53A65F40>
>>> exit()

➤

MDM-79 · 2021-09-23T14:00:19Z

Here is the result (same as before):

C:\Windows\system32>pip install youtube-dl
Collecting youtube-dl
  Downloading youtube_dl-2021.6.6-py2.py3-none-any.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 2.2 MB/s
Installing collected packages: youtube-dl
Successfully installed youtube-dl-2021.6.6

C:\Windows\system32>youtube-dl -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\youtube-dl.exe\__main__.py", line 7, in <module>
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\__init__.py", line 475, in main
    _real_main(argv)
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\__init__.py", line 442, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 422, in __init__
    self._setup_opener()
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 2395, in _setup_opener
    https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\utils.py", line 2297, in make_HTTPS_handler
    context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: [ASN1] nested asn1 error (_ssl.c:4174)

For b. and c.:

C:\Windows\system32>py
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> len(ssl.enum_certificates('ROOT'))
68
>>> len(ssl.enum_certificates('CA'))
39
>>> ssl.create_default_context()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: not enough data: cadata does not contain a certificate (_ssl.c:4159)
>>> exit()

P.S.
If maters, I have TLS 1.3 in Windows 10 enabled (forced) system-wide with reg command:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v1.0]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v3.0]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v3.0]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

pukkandan · 2021-09-23T14:44:30Z

Here is the result (same as before):

This is very useful info. Since the pip version of youtube-dl is also affected, that means the issue is caused somewhere b/w python 3.4 (used by youtube-dl.exe) and python 3.7 (used by yt-dlp_x86.exe)

https://github.com/python/cpython/blob/ec7ffa4b5b262369f726a54e145e9c03eaeb4c1a/Lib/ssl.py#L825-L829 says

        # no explicit cafile, capath or cadata but the verify mode is
        # CERT_OPTIONAL or CERT_REQUIRED. Let's try to load default system
        # root CA certificates for the given purpose. This may fail silently.

But in your case, this is NOT failing silently. I should be able to make a workaround for this (by ignoring the certs), but I'm not sure if that could break other parts. Once I make a patch, I'll ping you so you can test

It is odd that I could not find any issue reports about this with a quick search. But, since I can't reproduce it on my machine, making a bug report with all necessary details to python devs is also impractical 😢

If maters, I have TLS 1.3 in Windows 10 enabled (forced) system-wide with reg command:

This could be the culprit, but the actual code that does the certificate verification is in C (which I am not good at), so I cant really figure out the underlying issue. All I can do with a potential patch is to ignore that error and proceed without loading the certificates

MDM-79 · 2021-09-23T18:21:08Z

OK, have tried all I can think off, on my PC and Laptop:

I removed the TLS 1.3 forced system-wide - same thing
Fully uninstalled and deleted all traces of Python and reinstalled it (tried a short path too) - same
Tried Python 3.7.9 - same
Finally tried Python 3.4.3, and here I do have a different result:

C:\Windows\system32>pip install youtube-dl
Collecting youtube-dl
  Downloading https://files.pythonhosted.org/packages/a4/43/1f586e49e68f8b41c4be416302bf96ddd5040b0e744b5902d51063795eb9/youtube_dl-2021.6.6-py2.py3-none-any.whl (1.9MB)
    100% |################################| 1.9MB 311kB/s
Installing collected packages: youtube-dl

Successfully installed youtube-dl-2021.6.6

C:\Windows\system32>youtube-dl -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.3 (CPython) - Windows-8-6.2.9200
[debug] exe versions: none
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

C:\Windows\system32>py
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> len(ssl.enum_certificates('ROOT'))
68
>>> len(ssl.enum_certificates('CA'))
40
>>> ssl.create_default_context()
<ssl.SSLContext object at 0x000000000269F8E8>
>>> exit()

Don't know if this can help you...
And sure, make a patch and ping me, so I can test it!

MDM-79 · 2021-09-27T18:36:34Z

Still not had time to make a patch? Or stumbled on to something, if I can help in any way...
BTW latest yt-dlp.exe fails too.

pukkandan · 2021-09-27T19:29:21Z

Yeah, been busy with other stuff

Could you test this branch? https://github.com/pukkandan/yt-dlp-dev/tree/ssl-test

Test both with default options and with --no-check-certificates. If the program does load up correctly, also try downloading a random video to make sure network requests work correctly

You can install the branch with pip if you have git in your system

pip install git+https://github.com/pukkandan/yt-dlp-dev/tree/ssl-test

otherwise, just clone the branch and run yt-dlp.cmd similar to how you would normally run the exe

MDM-79 · 2021-09-28T12:12:22Z

It seams I am not such an advanced user to know how to do this, or if did it correctly...

Namely cmd form the cloned zip download does not work, and I do not know what to do with --no-check-certificates, here the results

But running pyinst.py file did work, and an .exe file (in "dist" folder) was produced, which I tested in MPC-HC with same failure...
After I try from the cloned and installed folder I have this

Have installed git (Git-2.33.0.2-64-bit.exe) but the pip install git+https://github.com/pukkandan/yt-dlp-dev/tree/ssl-test gives me this

If this gives you any insight, you are a miracle worker! 🥇

pukkandan · 2021-09-28T12:17:09Z

Sorry, it should be --no-check-certificate, no "s" at the end. Otherwise, your first attempt is correct. Can you do that again with the corrected option name and let me know

pukkandan · 2021-09-28T12:30:01Z

The error message in your latest attempt, nested asn1 error, did give me enough info to atleast identify the issue. One or more certificates in your root store is damaged. See https://bugs.python.org/issue35665. Python devs have flagged this as "not a bug" and I somewhat agree. I think I can write a workaround to ignore the faulty certificates, but it would be better if you could check and remove the faulty certificates from the system altogether

Another thing that the linked issue points out is that python 3.6 is less strict about this issue. So it could be a potential workaround for you till we come up with a more permanent solution

MDM-79 · 2021-09-28T12:47:06Z

The second option with --no-check-certificate now works, and downloads the playable file!

I obviously have the SAME problem than the guy in your link since I am from Serbia too, and those certificates "MUPCA Root" are (unfortunately-badly executed) crucial police 🙄 ones to be able too read ID cards and use personal signing certificates, and they're are all valid...
So the option to remove the faulty certificates, is a no go to me (or anyone in Serbia using their ID card - individuals, companies and entrepreneurs like me)...

That means, if you could write a workaround, it would be great! 😁 (mainly because I only want to use the .exe in MPC-HC to view videos, not so much to download and use as separate program).

pukkandan · 2021-09-28T13:21:04Z

That means, if you could write a workaround, it would be great!

It is possible, but a bit tricky. I'd have to monkeypatch code from the python's core ssl library. Could you try reporting the issue in python tracker again (or just comment on the closed issue - not sure what their policies are on that) explaining that these certificates are valid?

Even once I patch it for ytdlp, you (and as I understand, all Serbians) will encounter the same issue whenever you run another python 3.7+ program that connects to the internet

MDM-79 · 2021-09-28T16:09:05Z

Have commented on the "closed", and have opened a new one let's see what they say too...

pedjas · 2021-09-28T18:34:51Z

This hurts number of Python applications, even those published by large players. Basically, any attempt to read any certificate (for example to load any https url) fails due to this issue.

And that is a bug of Python. If you check code that causes this issue you will notice bad code.

When some certificate is needed Python loops and tries to load each and every certificate installed instead of loading only certificate that is needed and skipping others.
No exception handling. Python just crashes instead of graciously handle (skip) loading "bad" certificate.

This issue can be easily solved with one simple if and one simple exception handler.

pukkandan · 2021-09-28T19:47:06Z

Please keep the discussion here to be yt-dlp specific and move python discussion to the relevent issues opened in their tracker

@MDM-79 I have updated the https://github.com/pukkandan/yt-dlp-dev/tree/ssl-test branch. Could you do another test run?

MDM-79 · 2021-09-28T20:28:17Z

Sorry for the off-topic!

I can confirm, now it works! 😁 Even without the --no-check-certificate option!

pukkandan · 2021-09-28T20:31:54Z

sweet. I was concerned about performance. But doesn't seem like that's an issue

But it doesn't seem like you have just one bad certificate, but quite a lot of them?

I'll clean up the code and ask you to test one more time. Then I can commit it

MDM-79 · 2021-09-28T20:37:51Z

Works fast.

You are correct, a lot of them...
Just look at my post above you will see just how many MUP certificates are needed! (most certainly all messed up, and i missed some in the screenshot) 🤣

Great! 😉

P.S.
I have an answer, but am not sure they understood us...

MDM-79 · 2021-09-28T21:32:07Z

Tested from latest https://github.com/pukkandan/yt-dlp-dev/tree/ssl-test and it works, also no visible warnings anymore either!

* Remove old compat code * Load certificates only when not using nocheckcertificate * Load each certificate individually Closes #1060 Related bugs.python.org/issue35665, bugs.python.org/issue4531

MDM-79 · 2021-09-28T21:40:06Z

Works no problem now. Finally will have a working yt-dlp.exe for MPC-HC!
Thank you!

As for python and/or openssl will see if anything is done there...

* Remove old compat code * Load certificates only when not using nocheckcertificate * Load each certificate individually Closes yt-dlp#1060 Related bugs.python.org/issue35665, bugs.python.org/issue4531

pukkandan · 2022-04-20T13:22:42Z

python/cpython#91740

pukkandan added the incomplete Further information is needed label Sep 23, 2021

pukkandan added cant-reproduce The issue cannot be reliably reproduced and removed incomplete Further information is needed labels Sep 23, 2021

pukkandan added the external issue Issue with an external tool label Sep 28, 2021

pukkandan self-assigned this Sep 28, 2021

This comment has been minimized.

Sign in to view

pukkandan mentioned this issue Sep 28, 2021

Workaround for bug in ssl.SSLContext.load_default_certs #1118

Merged

MDM-79 mentioned this issue Sep 28, 2021

"MUPCA Root" Certificates - SSLError when windows cert store contains bad certificates openssl/openssl#16701

Open

pukkandan changed the title ~~yt-dlp fails with latest 1.9.16 mpc-hc~~ SSLError when windows cert store contains bad certificates Sep 28, 2021

pukkandan closed this as completed in #1118 Sep 28, 2021

SSLError when windows cert store contains bad certificates #1060

SSLError when windows cert store contains bad certificates #1060

Comments

MDM-79 commented Sep 23, 2021

pukkandan commented Sep 23, 2021

MDM-79 commented Sep 23, 2021 • edited by pukkandan

MDM-79 commented Sep 23, 2021 • edited

pukkandan commented Sep 23, 2021

MDM-79 commented Sep 23, 2021 • edited by pukkandan

pukkandan commented Sep 23, 2021

pukkandan commented Sep 23, 2021 • edited

MDM-79 commented Sep 23, 2021 • edited

MDM-79 commented Sep 23, 2021

pukkandan commented Sep 23, 2021 • edited

MDM-79 commented Sep 23, 2021 • edited

pukkandan commented Sep 23, 2021

MDM-79 commented Sep 23, 2021 • edited

MDM-79 commented Sep 27, 2021

pukkandan commented Sep 27, 2021

MDM-79 commented Sep 28, 2021

pukkandan commented Sep 28, 2021

pukkandan commented Sep 28, 2021

MDM-79 commented Sep 28, 2021 • edited

pukkandan commented Sep 28, 2021

This comment has been minimized.

MDM-79 commented Sep 28, 2021 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pedjas commented Sep 28, 2021 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pukkandan commented Sep 28, 2021

MDM-79 commented Sep 28, 2021 • edited

pukkandan commented Sep 28, 2021

MDM-79 commented Sep 28, 2021 • edited

MDM-79 commented Sep 28, 2021

MDM-79 commented Sep 28, 2021

pukkandan commented Apr 20, 2022

MDM-79 commented Sep 23, 2021 •

edited by pukkandan

MDM-79 commented Sep 23, 2021 •

edited

MDM-79 commented Sep 23, 2021 •

edited by pukkandan

pukkandan commented Sep 23, 2021 •

edited

MDM-79 commented Sep 23, 2021 •

edited

pukkandan commented Sep 23, 2021 •

edited

MDM-79 commented Sep 23, 2021 •

edited

MDM-79 commented Sep 23, 2021 •

edited

MDM-79 commented Sep 28, 2021 •

edited

MDM-79 commented Sep 28, 2021 •

edited

pedjas commented Sep 28, 2021 •

edited

MDM-79 commented Sep 28, 2021 •

edited

MDM-79 commented Sep 28, 2021 •

edited