New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify password for SSL client side certificate #1573

Open
botondus opened this Issue Sep 3, 2013 · 119 comments

Comments

Projects
None yet
@botondus

botondus commented Sep 3, 2013

As far as I know currently it's not possible to specify the password for the client side certificate you're using for authentication.
This is a bit of a problem because you typically always want to password protect your .pem file which contains the private key. openssl won't even let you create one without a password.

@botondus

This comment has been minimized.

Show comment
Hide comment
@botondus

botondus Sep 3, 2013

Something like:

requests.get('https://kennethreitz.com', cert='server.pem', cert_pw='my_password')

botondus commented Sep 3, 2013

Something like:

requests.get('https://kennethreitz.com', cert='server.pem', cert_pw='my_password')

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Sep 3, 2013

Member

Pretty sure you're supposed to use the cert param for that: cert=('server.pem', 'my_password')

Member

sigmavirus24 commented Sep 3, 2013

Pretty sure you're supposed to use the cert param for that: cert=('server.pem', 'my_password')

@t-8ch

This comment has been minimized.

Show comment
Hide comment
@t-8ch

t-8ch Sep 3, 2013

Contributor

@sigmavirus24
The tuple is for (certificate, key). Currently there is no support for encrypted keyfiles.
The stdlib only got support for those in version 3.3.

Contributor

t-8ch commented Sep 3, 2013

@sigmavirus24
The tuple is for (certificate, key). Currently there is no support for encrypted keyfiles.
The stdlib only got support for those in version 3.3.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Sep 3, 2013

Member

Heh, @t-8ch, you accidentally linked to a file on your local FS. ;) Correct link.

Member

Lukasa commented Sep 3, 2013

Heh, @t-8ch, you accidentally linked to a file on your local FS. ;) Correct link.

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Sep 4, 2013

Member

Quite right @t-8ch. This is why I should never answer issues from the bus. :/

Member

sigmavirus24 commented Sep 4, 2013

Quite right @t-8ch. This is why I should never answer issues from the bus. :/

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Sep 4, 2013

Member

So the current consensus is we don't support this. How much work is it likely to be to add support in non-3.3 versions of Python?

Member

Lukasa commented Sep 4, 2013

So the current consensus is we don't support this. How much work is it likely to be to add support in non-3.3 versions of Python?

@sdog869

This comment has been minimized.

Show comment
Hide comment
@sdog869

sdog869 Feb 18, 2014

How hard would it be to throw an error on this condition? I just ran into this silly problem and it took two hours to figure out, it would be nice if it would throw an error, it currently just sits there looping. Thanks for the awesome library!

sdog869 commented Feb 18, 2014

How hard would it be to throw an error on this condition? I just ran into this silly problem and it took two hours to figure out, it would be nice if it would throw an error, it currently just sits there looping. Thanks for the awesome library!

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Feb 18, 2014

Member

Wait, it sits where looping? Where in execution do we fail? Can you print the traceback from where we loop?

Member

Lukasa commented Feb 18, 2014

Wait, it sits where looping? Where in execution do we fail? Can you print the traceback from where we loop?

@sdog869

This comment has been minimized.

Show comment
Hide comment
@sdog869

sdog869 Feb 19, 2014

It seems to hang right here:

r = requests.get(url,
auth=headeroauth,
cert=self.cert_tuple,
headers=headers,
timeout=10,
verify=True)

I tried turning the timeout out up or down to no avail, but I imagine it knows well before the timeout it can't use the cert. Thanks!

sdog869 commented Feb 19, 2014

It seems to hang right here:

r = requests.get(url,
auth=headeroauth,
cert=self.cert_tuple,
headers=headers,
timeout=10,
verify=True)

I tried turning the timeout out up or down to no avail, but I imagine it knows well before the timeout it can't use the cert. Thanks!

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Feb 19, 2014

Member

Ah, sorry, I wasn't clear. I meant to let it hang and then kill it with Ctrl + C so that python throws a KeyboardInterrupt exception, then to see where we are in the traceback. I want to know where in Requests the execution halts.

Member

Lukasa commented Feb 19, 2014

Ah, sorry, I wasn't clear. I meant to let it hang and then kill it with Ctrl + C so that python throws a KeyboardInterrupt exception, then to see where we are in the traceback. I want to know where in Requests the execution halts.

@maxnoel

This comment has been minimized.

Show comment
Hide comment
@maxnoel

maxnoel Jul 30, 2014

What's happening (or at least what I've seen in many cases) is that OpenSSL, upon being given a password-protected certificate, will prompt the user for a password. It shows up in no logs (because the prompt is directly printed), and it doesn't time out because it's waiting for a user to press enter.

Needless to say, it's cubmersome, dangerous behavior when the code is running on a server (because it'll hang your worker with no option for recovery other than killing the process).

Is there a way to make requests raise an exception in that case instead of prompting for a password, or is that completely out of your control and in OpenSSL's hands?

maxnoel commented Jul 30, 2014

What's happening (or at least what I've seen in many cases) is that OpenSSL, upon being given a password-protected certificate, will prompt the user for a password. It shows up in no logs (because the prompt is directly printed), and it doesn't time out because it's waiting for a user to press enter.

Needless to say, it's cubmersome, dangerous behavior when the code is running on a server (because it'll hang your worker with no option for recovery other than killing the process).

Is there a way to make requests raise an exception in that case instead of prompting for a password, or is that completely out of your control and in OpenSSL's hands?

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Jul 30, 2014

Member

@maxnoel I'm pretty sure this is in OpenSSL's hands but if you can answer @Lukasa's question (the last comment on this issue) it would be very helpful in giving a definite answer regarding if there was anything we can do to help.

Member

sigmavirus24 commented Jul 30, 2014

@maxnoel I'm pretty sure this is in OpenSSL's hands but if you can answer @Lukasa's question (the last comment on this issue) it would be very helpful in giving a definite answer regarding if there was anything we can do to help.

@jjguy

This comment has been minimized.

Show comment
Hide comment
@jjguy

jjguy Apr 16, 2015

You can confirm OpenSSL is blocking on stdin for the passphrase from the interactive python prompt:

>>> r = requests.get("https://foo.example.com/api/user/bill", cert=("client.crt", "client.key"))
Enter PEM pass phrase:
>>>

If you're running from a backgrounded process, I assume OpenSSL will block waiting on that input.

jjguy commented Apr 16, 2015

You can confirm OpenSSL is blocking on stdin for the passphrase from the interactive python prompt:

>>> r = requests.get("https://foo.example.com/api/user/bill", cert=("client.crt", "client.key"))
Enter PEM pass phrase:
>>>

If you're running from a backgrounded process, I assume OpenSSL will block waiting on that input.

@maxnoel

This comment has been minimized.

Show comment
Hide comment
@maxnoel

maxnoel Apr 16, 2015

That's correct. Is there anything requests can do to prevent that from happening? Raising an exception when no password is given would be far more useful than prompting for stuff on stdin (especially in a non-interactive program).

maxnoel commented Apr 16, 2015

That's correct. Is there anything requests can do to prevent that from happening? Raising an exception when no password is given would be far more useful than prompting for stuff on stdin (especially in a non-interactive program).

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Apr 16, 2015

Member

I'm afraid that I don't know of any way. @reaperhulk?

Member

Lukasa commented Apr 16, 2015

I'm afraid that I don't know of any way. @reaperhulk?

@reaperhulk

This comment has been minimized.

Show comment
Hide comment
@reaperhulk

reaperhulk Apr 16, 2015

There are ways to stop OpenSSL from doing this, but I'm not sure if they're exposed by pyOpenSSL. Where does requests call pyopenssl to load the client cert? I can dig a bit.

reaperhulk commented Apr 16, 2015

There are ways to stop OpenSSL from doing this, but I'm not sure if they're exposed by pyOpenSSL. Where does requests call pyopenssl to load the client cert? I can dig a bit.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Apr 16, 2015

Member

@reaperhulk It's done from in urllib3, here.

Member

Lukasa commented Apr 16, 2015

@reaperhulk It's done from in urllib3, here.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Apr 16, 2015

Member

We also do something very similar for the stdlib, which will be a whole separate problem.

Member

Lukasa commented Apr 16, 2015

We also do something very similar for the stdlib, which will be a whole separate problem.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Apr 16, 2015

Member

So we can do this with PyOpenSSL using a patch like this. In the stdlib version, we need to use load_cert_chain with a password.

Member

Lukasa commented Apr 16, 2015

So we can do this with PyOpenSSL using a patch like this. In the stdlib version, we need to use load_cert_chain with a password.

@telam

This comment has been minimized.

Show comment
Hide comment
@telam

telam Oct 22, 2015

Has this problem been solved? I'm currently running into this while trying to connect to an Apache server.

telam commented Oct 22, 2015

Has this problem been solved? I'm currently running into this while trying to connect to an Apache server.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Oct 22, 2015

Member

It has not.

Member

Lukasa commented Oct 22, 2015

It has not.

@mikelupo

This comment has been minimized.

Show comment
Hide comment
@mikelupo

mikelupo Dec 18, 2015

What about PKCS#12 formatted (and encrypted) containers which could contain a client cert/key? Would this fall under the same feature request?

mikelupo commented Dec 18, 2015

What about PKCS#12 formatted (and encrypted) containers which could contain a client cert/key? Would this fall under the same feature request?

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa
Member

Lukasa commented Dec 18, 2015

@mikelupo Yup.

@Altynai

This comment has been minimized.

Show comment
Hide comment
@Altynai

Altynai Jan 8, 2016

@telam @mikelupo
I have the same problem and Googled a lot, finally, I solved it by using pycurl.
In my situation, I use openssl to convert my .pfx file to .pem file which contains both cert & key(encrypted with pass phrase), then invoke the following code.

import pycurl
import StringIO

b = StringIO.StringIO()
c = pycurl.Curl()
url = "https://example.com"
c.setopt(pycurl.URL, url)
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.setopt(pycurl.CAINFO, "/path/cacert.pem")
c.setopt(pycurl.SSLKEY, "/path/key_file.pem")
c.setopt(pycurl.SSLCERT, "/path/cert_file.pem")
c.setopt(pycurl.SSLKEYPASSWD, "your pass phrase")
c.perform()
c.close()
response_body = b.getvalue()

BTW, for security, it's better to not do hardcode for pass phrase

Altynai commented Jan 8, 2016

@telam @mikelupo
I have the same problem and Googled a lot, finally, I solved it by using pycurl.
In my situation, I use openssl to convert my .pfx file to .pem file which contains both cert & key(encrypted with pass phrase), then invoke the following code.

import pycurl
import StringIO

b = StringIO.StringIO()
c = pycurl.Curl()
url = "https://example.com"
c.setopt(pycurl.URL, url)
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.setopt(pycurl.CAINFO, "/path/cacert.pem")
c.setopt(pycurl.SSLKEY, "/path/key_file.pem")
c.setopt(pycurl.SSLCERT, "/path/cert_file.pem")
c.setopt(pycurl.SSLKEYPASSWD, "your pass phrase")
c.perform()
c.close()
response_body = b.getvalue()

BTW, for security, it's better to not do hardcode for pass phrase

@maxnoel

This comment has been minimized.

Show comment
Hide comment
@maxnoel

maxnoel Jan 15, 2016

Of course. That said, the problem isn't really that a pass phrase is required -- it's that OpenSSL makes your program hang while waiting for someone to type a passphrase in stdin, even in the case of a non-interactive, GUI or remote program.

When a passphrase is required and none is provided, an exception should be raised instead.

maxnoel commented Jan 15, 2016

Of course. That said, the problem isn't really that a pass phrase is required -- it's that OpenSSL makes your program hang while waiting for someone to type a passphrase in stdin, even in the case of a non-interactive, GUI or remote program.

When a passphrase is required and none is provided, an exception should be raised instead.

@FirefighterBlu3

This comment has been minimized.

Show comment
Hide comment
@FirefighterBlu3

FirefighterBlu3 Jan 16, 2016

if you use a default passphrase of '' for the key, openssl won't hang.
it'll return a bad password text. you can immediately alter your py flow
to then notify the user without that apparant stall

FirefighterBlu3 commented Jan 16, 2016

if you use a default passphrase of '' for the key, openssl won't hang.
it'll return a bad password text. you can immediately alter your py flow
to then notify the user without that apparant stall

@ldkingvivi

This comment has been minimized.

Show comment
Hide comment
@ldkingvivi

ldkingvivi Jan 22, 2016

any plan to add this feature

ldkingvivi commented Jan 22, 2016

any plan to add this feature

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jan 22, 2016

Member

We want to add it, but we have no schedule to add it at this time.

Member

Lukasa commented Jan 22, 2016

We want to add it, but we have no schedule to add it at this time.

@vinitkumar

This comment has been minimized.

Show comment
Hide comment
@vinitkumar

vinitkumar Feb 24, 2016

@botondus I think I found a simpler way to achieve this with request library. I am documenting this for other people who are facing the issue.

I assume that you have a .p12 certificate and a passphrase for the key.

Generate certificate and private key.

// Generate the certificate file.
openssl pkcs12 -in /path/to/p12cert -nokeys -out certificate.pem
// Generate private key with passpharse, First enter the password provided with the key and then an arbitrary PEM password //(say: 1234) 
openssl pkcs12 -in /path/to/p12cert -nocerts -out privkey.pem

Well, we are not done yet and we need to generate the key that doesn't require the PEM password every time it needs to talk to the server.

Generate key without passphrase.

// Running this command will prompt for the pem password(1234), on providing which we will obtain the plainkey.pem
openssl rsa -in privkey.pem -out plainkey.pem

Now, you will have certificate.pem and plainkey.pem, both of the files required to talk to the API using requests.

Here is an example request using these cert and keys.

import requests
url = 'https://exampleurl.com'
headers = {
            'header1': '1214141414',
            'header2': 'adad-1223-122'
          }
response = requests.get(url, headers=headers, cert=('~/certificate.pem', '~/plainkey.pem'), verify=True)
print response.json()

Hope this helps:

cc @kennethreitz @Lukasa @sigmavirus24

vinitkumar commented Feb 24, 2016

@botondus I think I found a simpler way to achieve this with request library. I am documenting this for other people who are facing the issue.

I assume that you have a .p12 certificate and a passphrase for the key.

Generate certificate and private key.

// Generate the certificate file.
openssl pkcs12 -in /path/to/p12cert -nokeys -out certificate.pem
// Generate private key with passpharse, First enter the password provided with the key and then an arbitrary PEM password //(say: 1234) 
openssl pkcs12 -in /path/to/p12cert -nocerts -out privkey.pem

Well, we are not done yet and we need to generate the key that doesn't require the PEM password every time it needs to talk to the server.

Generate key without passphrase.

// Running this command will prompt for the pem password(1234), on providing which we will obtain the plainkey.pem
openssl rsa -in privkey.pem -out plainkey.pem

Now, you will have certificate.pem and plainkey.pem, both of the files required to talk to the API using requests.

Here is an example request using these cert and keys.

import requests
url = 'https://exampleurl.com'
headers = {
            'header1': '1214141414',
            'header2': 'adad-1223-122'
          }
response = requests.get(url, headers=headers, cert=('~/certificate.pem', '~/plainkey.pem'), verify=True)
print response.json()

Hope this helps:

cc @kennethreitz @Lukasa @sigmavirus24

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Feb 24, 2016

Member

I have heard through the grapevine that Amazon does exactly this, internally.

Member

kennethreitz commented Feb 24, 2016

I have heard through the grapevine that Amazon does exactly this, internally.

@SethMichaelLarson

This comment has been minimized.

Show comment
Hide comment
@SethMichaelLarson

SethMichaelLarson Sep 1, 2017

@erikbern This is public information. You can achieve the same result by running the same command.

SethMichaelLarson commented Sep 1, 2017

@erikbern This is public information. You can achieve the same result by running the same command.

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Sep 1, 2017

Member

@SethMichaelLarson From @erikbern's GitHub profile "Chief Troll Officer". Perhaps they were just trolling?

Member

sigmavirus24 commented Sep 1, 2017

@SethMichaelLarson From @erikbern's GitHub profile "Chief Troll Officer". Perhaps they were just trolling?

@SethMichaelLarson

This comment has been minimized.

Show comment
Hide comment
@SethMichaelLarson

SethMichaelLarson Sep 1, 2017

@erikbern @sigmavirus24 Ah! I didn't know who I was speaking to. Proceed! 🙇

SethMichaelLarson commented Sep 1, 2017

@erikbern @sigmavirus24 Ah! I didn't know who I was speaking to. Proceed! 🙇

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 1, 2017

I cant see anything except the sha-1 certificate when I run from postman
May be i need to add this somehow to Pycharm

ghost commented Sep 1, 2017

I cant see anything except the sha-1 certificate when I run from postman
May be i need to add this somehow to Pycharm

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Sep 1, 2017

Member

If you literally browse to the website in Chrome that should be sufficient.

Member

Lukasa commented Sep 1, 2017

If you literally browse to the website in Chrome that should be sufficient.

@erikbern

This comment has been minimized.

Show comment
Hide comment
@erikbern

erikbern Sep 1, 2017

@SethMichaelLarson running what command? FYI the comment has been deleted now but there was an entire BEGIN CERTIFICATE blob here earlier... afaik you don't want to share that online

erikbern commented Sep 1, 2017

@SethMichaelLarson running what command? FYI the comment has been deleted now but there was an entire BEGIN CERTIFICATE blob here earlier... afaik you don't want to share that online

@ahnolds

This comment has been minimized.

Show comment
Hide comment
@ahnolds

ahnolds Sep 1, 2017

@erikbern That was just the public key for the cert...

ahnolds commented Sep 1, 2017

@erikbern That was just the public key for the cert...

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Sep 2, 2017

Member

Certificates are public data; they are transmitted in plaintext over the network on each connection attempt.

Member

Lukasa commented Sep 2, 2017

Certificates are public data; they are transmitted in plaintext over the network on each connection attempt.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 7, 2017

I went to the cert chain and only found a Sha-1 cert and the .Pem cert that I am using to hit the API

ghost commented Sep 7, 2017

I went to the cert chain and only found a Sha-1 cert and the .Pem cert that I am using to hit the API

@seghcder

This comment has been minimized.

Show comment
Hide comment
@seghcder

seghcder Sep 19, 2017

@AnoopPillai I got your example code from Sep 1 working without issue using a client-side pem file with password. It seems the host is using a regular cert. With @Lukasa thanks very much !

seghcder commented Sep 19, 2017

@AnoopPillai I got your example code from Sep 1 working without issue using a client-side pem file with password. It seems the host is using a regular cert. With @Lukasa thanks very much !

@mkane848

This comment has been minimized.

Show comment
Hide comment
@mkane848

mkane848 Oct 9, 2017

I'm unfortunately still having issues, even with the Temp File method. I can use the .pfx in Google Postman and have no issues authenticating (so I know my credentials work), but I'm still getting 401s with Python. Unfortunately the support guy from the company I'm dealing with hasn't been much help - does anyone have any suggestions for troubleshooting?

At this stage I'm genuinely unsure of where to even look for the problem since other people are reporting success with the Temp File method and I still haven't heard anything back from their Cert Management team.

Any advice would be much appreciated - please let me know if I can provide any additional information to make this easier.

Thanks :)

mkane848 commented Oct 9, 2017

I'm unfortunately still having issues, even with the Temp File method. I can use the .pfx in Google Postman and have no issues authenticating (so I know my credentials work), but I'm still getting 401s with Python. Unfortunately the support guy from the company I'm dealing with hasn't been much help - does anyone have any suggestions for troubleshooting?

At this stage I'm genuinely unsure of where to even look for the problem since other people are reporting success with the Temp File method and I still haven't heard anything back from their Cert Management team.

Any advice would be much appreciated - please let me know if I can provide any additional information to make this easier.

Thanks :)

@seghcder

This comment has been minimized.

Show comment
Hide comment
@seghcder

seghcder Oct 9, 2017

Just a suggestion, did you try converting PFX to PEM? Also, if the server is also using a username/password, you'll need to add that the get/post request using auth=(). I've been using the class DESAdapter(HTTPAdapter) approach above for several weeks now without issue, using a password protected PEM file.

seghcder commented Oct 9, 2017

Just a suggestion, did you try converting PFX to PEM? Also, if the server is also using a username/password, you'll need to add that the get/post request using auth=(). I've been using the class DESAdapter(HTTPAdapter) approach above for several weeks now without issue, using a password protected PEM file.

@mkane848

This comment has been minimized.

Show comment
Hide comment
@mkane848

mkane848 Oct 10, 2017

@ideasean Getting invalid credentials still. I should be pointing the load_cert_chain at a .pem file generated by the pfx_to_pem function written for the Temp File method, correct? It has the private key and the cert in it.

Since the .pfx works with Postman but it won't authenticate here, could that mean that something's going wrong in the conversion process?

mkane848 commented Oct 10, 2017

@ideasean Getting invalid credentials still. I should be pointing the load_cert_chain at a .pem file generated by the pfx_to_pem function written for the Temp File method, correct? It has the private key and the cert in it.

Since the .pfx works with Postman but it won't authenticate here, could that mean that something's going wrong in the conversion process?

@seghcder

This comment has been minimized.

Show comment
Hide comment
@seghcder

seghcder Oct 10, 2017

I did not use the temp file method. I used the DESAdapter approach pretty much as written in AnoopPillai's post on Sep1 above starting with -

I did try with that code change (code pasted below) and ended up with the same error that i got with the tempfile method.

I can't speak to the conversion process, but perhaps a good test is to try using the converted pem file with Postman?

Also note that I used the approach above because my pem file was encrypted / password protected, and Python requests currently does not support that. If your pem ends up being not password protected, then you should be able to use native requests per link (but then you will have an unprotected cert on your file system).

seghcder commented Oct 10, 2017

I did not use the temp file method. I used the DESAdapter approach pretty much as written in AnoopPillai's post on Sep1 above starting with -

I did try with that code change (code pasted below) and ended up with the same error that i got with the tempfile method.

I can't speak to the conversion process, but perhaps a good test is to try using the converted pem file with Postman?

Also note that I used the approach above because my pem file was encrypted / password protected, and Python requests currently does not support that. If your pem ends up being not password protected, then you should be able to use native requests per link (but then you will have an unprotected cert on your file system).

@mkane848

This comment has been minimized.

Show comment
Hide comment
@mkane848

mkane848 Oct 10, 2017

@ideasean I broke down the .pfx as per this method and got a .pem file with Bag Attributes and Certificate as well as a .pem file with Bag Attributes and an Encrypted Private Key.

Still getting invalid credentials, I guess I'll try putting the certs through on Postman and seeing if they work but I can't figure out why I'm apparently unable to unpack this .pfx properly

I also tried the openssl command openssl pkcs12 -in <my_pfx>.pfx -out certificate.cer -nodes, and it's still giving me a 401 error when I change to it like so: context.load_cert_chain('certificate.cer')

mkane848 commented Oct 10, 2017

@ideasean I broke down the .pfx as per this method and got a .pem file with Bag Attributes and Certificate as well as a .pem file with Bag Attributes and an Encrypted Private Key.

Still getting invalid credentials, I guess I'll try putting the certs through on Postman and seeing if they work but I can't figure out why I'm apparently unable to unpack this .pfx properly

I also tried the openssl command openssl pkcs12 -in <my_pfx>.pfx -out certificate.cer -nodes, and it's still giving me a 401 error when I change to it like so: context.load_cert_chain('certificate.cer')

@mkane848

This comment has been minimized.

Show comment
Hide comment
@mkane848

mkane848 Oct 10, 2017

I installed the above-mentioned .cer and Postman doesn't even ask to use it when I make the API call (unlike the popup when it asks to use the .pfx), not sure how else I can make it use that specific cert since there's no "Certificates" panel in the settings like the docs say there is.

mkane848 commented Oct 10, 2017

I installed the above-mentioned .cer and Postman doesn't even ask to use it when I make the API call (unlike the popup when it asks to use the .pfx), not sure how else I can make it use that specific cert since there's no "Certificates" panel in the settings like the docs say there is.

@seghcder

This comment has been minimized.

Show comment
Hide comment
@seghcder

seghcder Oct 10, 2017

You may be using the browser version of Postman, which doesn't include the cert panel, ssl validation disable etc. Try the full client to change certificate settings. You may want to continue this discussion on a different thread then, as we are a bit off topic.

seghcder commented Oct 10, 2017

You may be using the browser version of Postman, which doesn't include the cert panel, ssl validation disable etc. Try the full client to change certificate settings. You may want to continue this discussion on a different thread then, as we are a bit off topic.

@aiguofer

This comment has been minimized.

Show comment
Hide comment
@aiguofer

aiguofer Oct 12, 2017

@mkane848 saw your original comment where you were getting a ValueError: String expected. You might want to check pyca/pyopenssl#701 and urllib3/urllib3#1275.

I use my private pem with a password using this:

from requests.adapters import HTTPAdapter

from urllib3.util.ssl_ import create_urllib3_context

class SSLAdapter(HTTPAdapter):
    def __init__(self, certfile, keyfile, password=None, *args, **kwargs):
        self._certfile = certfile
        self._keyfile = keyfile
        self._password = password
        return super(self.__class__, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).proxy_manager_for(*args, **kwargs)

    def _add_ssl_context(self, kwargs):
        context = create_urllib3_context()
        context.load_cert_chain(certfile=self._certfile,
                                keyfile=self._keyfile,
                                password=str(self._password))
        kwargs['ssl_context'] = context

aiguofer commented Oct 12, 2017

@mkane848 saw your original comment where you were getting a ValueError: String expected. You might want to check pyca/pyopenssl#701 and urllib3/urllib3#1275.

I use my private pem with a password using this:

from requests.adapters import HTTPAdapter

from urllib3.util.ssl_ import create_urllib3_context

class SSLAdapter(HTTPAdapter):
    def __init__(self, certfile, keyfile, password=None, *args, **kwargs):
        self._certfile = certfile
        self._keyfile = keyfile
        self._password = password
        return super(self.__class__, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).proxy_manager_for(*args, **kwargs)

    def _add_ssl_context(self, kwargs):
        context = create_urllib3_context()
        context.load_cert_chain(certfile=self._certfile,
                                keyfile=self._keyfile,
                                password=str(self._password))
        kwargs['ssl_context'] = context
@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Dec 4, 2017

For your information, I just implemented PKCS#12 support for requests as a separate library:

The code is a clean implementation: it uses neither monkey patching nor temporary files. Instead, a custom TransportAdapter is used, which provides a custom SSLContext.

Any feedback and improvements are welcome!

Of course, I wish requests would provide this functionality directly, but until we are there, this library will alleviate the pain.

vog commented Dec 4, 2017

For your information, I just implemented PKCS#12 support for requests as a separate library:

The code is a clean implementation: it uses neither monkey patching nor temporary files. Instead, a custom TransportAdapter is used, which provides a custom SSLContext.

Any feedback and improvements are welcome!

Of course, I wish requests would provide this functionality directly, but until we are there, this library will alleviate the pain.

@candlerb

This comment has been minimized.

Show comment
Hide comment
@candlerb

candlerb Dec 20, 2017

It would be very nice if we could simply do this:

    cert=("cert.pem", "key.pem", "somepassphrase")  # separate cert/key

    cert=("keycert.pem", None, "somepassphrase")    # combined cert/key

...even if it only worked on python 3.3+. This would only be a minor addition to the API surface.

AFAICS, this would mean a small change to urllib3 so that HTTPSConnection accepts an optional password argument; this is passed down through ssl_wrap_socket, ending up with:

    if certfile:
        if password is not None:
            context.load_cert_chain(certfile, keyfile, password)
        else:
            context.load_cert_chain(certfile, keyfile)

Then it would be backwards-compatible, raising an exception only if you try to use a private key passphrase on an older platform that doesn't support it.

Note that the contrib/pyopenssl.py adapter already supports this extra argument to load_cert_chain, and so does python 2.7.


Aside: I am using AWS KMS to manage "secret" data, so I would load the key password at runtime from KMS, not hard-code it into the application.

candlerb commented Dec 20, 2017

It would be very nice if we could simply do this:

    cert=("cert.pem", "key.pem", "somepassphrase")  # separate cert/key

    cert=("keycert.pem", None, "somepassphrase")    # combined cert/key

...even if it only worked on python 3.3+. This would only be a minor addition to the API surface.

AFAICS, this would mean a small change to urllib3 so that HTTPSConnection accepts an optional password argument; this is passed down through ssl_wrap_socket, ending up with:

    if certfile:
        if password is not None:
            context.load_cert_chain(certfile, keyfile, password)
        else:
            context.load_cert_chain(certfile, keyfile)

Then it would be backwards-compatible, raising an exception only if you try to use a private key passphrase on an older platform that doesn't support it.

Note that the contrib/pyopenssl.py adapter already supports this extra argument to load_cert_chain, and so does python 2.7.


Aside: I am using AWS KMS to manage "secret" data, so I would load the key password at runtime from KMS, not hard-code it into the application.

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Dec 20, 2017

Member

I personally wouldn’t be against this change, as I think it would greatly improve our user interface for many users across the board.

@sigmavirus24 any thoughts?

Member

kennethreitz commented Dec 20, 2017

I personally wouldn’t be against this change, as I think it would greatly improve our user interface for many users across the board.

@sigmavirus24 any thoughts?

@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Dec 20, 2017

@candlerb @kennethreitz Would it be acceptable to include the PKCS#12 case into that API as well?

cert=('keycert.p12', None, 'somepassphrase')

The distinction could be either by file extension (*.p12 versus *.pem), or by looking at the first bytes of that file.

vog commented Dec 20, 2017

@candlerb @kennethreitz Would it be acceptable to include the PKCS#12 case into that API as well?

cert=('keycert.p12', None, 'somepassphrase')

The distinction could be either by file extension (*.p12 versus *.pem), or by looking at the first bytes of that file.

@candlerb

This comment has been minimized.

Show comment
Hide comment
@candlerb

candlerb Dec 21, 2017

I don't have a problem with allowing requests to take a pkcs#12, as long as it can be done safely - and in my opinion that precludes writing the extracted private key to a temporary file.

Googling for Python pkcs#12, I find:

  • Someone's code which writes out the private key
  • Some other code which I think depends on pyOpenSSL to read in the pkcs#12. It returns the certificate and key as data items.

So doing this, I think it would be necessary to hook things up in such a way that the key/cert themselves are passed to OpenSSL, not the filenames containing those things. That sounds like a much bigger change.

If that's too hard, then it just means that the user has to convert pkcs#12 to PEM off-line, which is pretty straightforward (and can be documented).

candlerb commented Dec 21, 2017

I don't have a problem with allowing requests to take a pkcs#12, as long as it can be done safely - and in my opinion that precludes writing the extracted private key to a temporary file.

Googling for Python pkcs#12, I find:

  • Someone's code which writes out the private key
  • Some other code which I think depends on pyOpenSSL to read in the pkcs#12. It returns the certificate and key as data items.

So doing this, I think it would be necessary to hook things up in such a way that the key/cert themselves are passed to OpenSSL, not the filenames containing those things. That sounds like a much bigger change.

If that's too hard, then it just means that the user has to convert pkcs#12 to PEM off-line, which is pretty straightforward (and can be documented).

@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Dec 21, 2017

@candlerb As I wrote in my previous comment (#1573 (comment)), I already created a clean implementation that integrates well with requests.

So the problems you are describing are already solved.

Right now my implementation adds new pkcs12_* keywords arguments, to stay out of the way as much as possible.

But I think it should be integrated into the cert keyword argument instead, and my question is:

  • Would that be acceptable in general?
  • Would my concrete proposal cert=('keycert.p12', None, 'somepassphrase') be acceptable?
  • How should we distinguish between PKCS#12 and PEM? (By file name suffix, or by file contents?)

(Moreover, I'd prefer to see that into requests rather than my separate requests_pkcs12 library. But given the age of this issue, I have little hope that this will go upstream anytime soon. However, if there was a concrete statement about which kind of implementation exactly is wanted, maybe I could adjust my implementation accordingly and propose a pull request.)

vog commented Dec 21, 2017

@candlerb As I wrote in my previous comment (#1573 (comment)), I already created a clean implementation that integrates well with requests.

So the problems you are describing are already solved.

Right now my implementation adds new pkcs12_* keywords arguments, to stay out of the way as much as possible.

But I think it should be integrated into the cert keyword argument instead, and my question is:

  • Would that be acceptable in general?
  • Would my concrete proposal cert=('keycert.p12', None, 'somepassphrase') be acceptable?
  • How should we distinguish between PKCS#12 and PEM? (By file name suffix, or by file contents?)

(Moreover, I'd prefer to see that into requests rather than my separate requests_pkcs12 library. But given the age of this issue, I have little hope that this will go upstream anytime soon. However, if there was a concrete statement about which kind of implementation exactly is wanted, maybe I could adjust my implementation accordingly and propose a pull request.)

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Dec 22, 2017

Member

So, a few things:

  1. I don't think we should take the cert keyword and expand it like this. It's implicitly structured data and people are already confused by the tuples in the files keyword. I think continuing a known-bad pattern is foolish.

  2. I think that if anything, the pkcs12 adapter should be modified and upstreamed into the requests-toolbelt. I think it would be better to modify it to create the ssl_context once instead of storing the pkcs12 password in memory on that object.

I think there's still other work that needs doing before we can handle this in the more general case no matter what and that includes determining the right API for this for Requests 3.0.

Member

sigmavirus24 commented Dec 22, 2017

So, a few things:

  1. I don't think we should take the cert keyword and expand it like this. It's implicitly structured data and people are already confused by the tuples in the files keyword. I think continuing a known-bad pattern is foolish.

  2. I think that if anything, the pkcs12 adapter should be modified and upstreamed into the requests-toolbelt. I think it would be better to modify it to create the ssl_context once instead of storing the pkcs12 password in memory on that object.

I think there's still other work that needs doing before we can handle this in the more general case no matter what and that includes determining the right API for this for Requests 3.0.

@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Dec 27, 2017

@sigmavirus24 Thanks for the feedback.

  1. Okay, so let's keep the separate pkcs12_* keywords.
  2. Yes, that's definitely worth improving. I created an issue tracker entry for that: m-click/requests_pkcs12#2

How would the PKCS#12 TransportAdapter class be included into requests? Would that class simply be added to requests, or is there another way to include it on a "deeper" level, so it can be used without any request()/get()/... wrappers and without having to explicitly load that adapter?

vog commented Dec 27, 2017

@sigmavirus24 Thanks for the feedback.

  1. Okay, so let's keep the separate pkcs12_* keywords.
  2. Yes, that's definitely worth improving. I created an issue tracker entry for that: m-click/requests_pkcs12#2

How would the PKCS#12 TransportAdapter class be included into requests? Would that class simply be added to requests, or is there another way to include it on a "deeper" level, so it can be used without any request()/get()/... wrappers and without having to explicitly load that adapter?

@rashley-iqt

This comment has been minimized.

Show comment
Hide comment
@rashley-iqt

rashley-iqt Oct 16, 2018

My organization has a need to use PKCS12 certificates and is willing to make the necessary enhancements to your library in order to do so. Decrypting the .p12 files to .pem files is considered too much of a risk and it adds an extra step to deal with. We'd like to add functionality to generate and provide an appropriate ssl_context for a given session. Is this still functionality your team would be willing to accept assuming it is implemented properly?

rashley-iqt commented Oct 16, 2018

My organization has a need to use PKCS12 certificates and is willing to make the necessary enhancements to your library in order to do so. Decrypting the .p12 files to .pem files is considered too much of a risk and it adds an extra step to deal with. We'd like to add functionality to generate and provide an appropriate ssl_context for a given session. Is this still functionality your team would be willing to accept assuming it is implemented properly?

@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Oct 16, 2018

Just a quick reminder: A clean implementation has already been provided by our company, but as a separate adapter: https://github.com/m-click/requests_pkcs12

Feel free to reformat it into a pull request for requests itself.

Along the way, you might want to fix a minor issue: The ssl_context should not be held in memory for a whole session, but as shortly as possible, just for a single given connection. See also:

In case you fix it along the way, it would be nice if you could provide it as a small pull request to https://github.com/m-click/requests_pkcs12 in addition to requests itself.

That way, all people who are using the requests_pkcs12 library right now would automatically benefit from that improvement as well, without having to switch to the (then improved) new API for requests itself.

vog commented Oct 16, 2018

Just a quick reminder: A clean implementation has already been provided by our company, but as a separate adapter: https://github.com/m-click/requests_pkcs12

Feel free to reformat it into a pull request for requests itself.

Along the way, you might want to fix a minor issue: The ssl_context should not be held in memory for a whole session, but as shortly as possible, just for a single given connection. See also:

In case you fix it along the way, it would be nice if you could provide it as a small pull request to https://github.com/m-click/requests_pkcs12 in addition to requests itself.

That way, all people who are using the requests_pkcs12 library right now would automatically benefit from that improvement as well, without having to switch to the (then improved) new API for requests itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment