Allow opt-in insecure requests to HTTPS URLs

[TedDriggs]

The author of a client library/tool isn't always able to stop a server from using self-signed or otherwise-invalid certificates. Is there a way to ask reqwest to ignore validation for a specific request?

(In my specific scenario, the server will only respond over HTTPS, so asking over bare HTTP isn't an option either)

[seanmonstar]

It doesn't exist yet. I hate the concept, but I know it's a reality that many people have to deal with, so I think it should be done in reqwest. Probably something like client.insecure_disable_tls_verification().

[Philipp91]

How about allowing to white-list/pin such self-signed/invalid certificates instead of disabling the verification altogether?
It could even be more secure to use a pinned (invalid) certificate if the private key is not compromised, or it might not be -- I'm not sure. My point is that it would be better not to have a disable-all-verification option available for everyone else to misuse.

[TedDriggs]

@Philipp91, that wouldn't work for the use-case I'm contending with. This tool will be used to connect to client-administered machines, so I won't have the certificates available ahead of time to whitelist.

[cengiz-io]

Hello!

Although I'm very new to reqwest I want to work on this.

Am I allowed to?

[seanmonstar]

@cengizio absolutely you are allowed to. Note, though, that this would need some work in the TLS dependency to allow this, see sfackler/rust-native-tls#13

[yoni386]

any progress with this? the "danger" method is in use but still panic from sone urls in curl is working (curl ssl_verify methods below).
handle.ssl_verify_host(false);
handle.ssl_verify_peer(false);

Thanks

[seanmonstar]

There was functionality just merged in #124 that should allow closing this.

[yoni386]

Sorry but it seems to be the same. This is not the "complicated case" on my attempts it doesn't work -> reqwest::get("https://54.239.168.184").unwrap();

…

On Thu, Jun 1, 2017 at 2:16 AM, Sean McArthur ***@***.***> wrote: There was functionality just merged in #124 <#124> that should allow closing this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2pvbWv0d9K5xy3nhgQRUyO24H22uks5r_fTKgaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[seanmonstar]

@yoni386 the support that was just merged is only on the master branch at the moment, a release hasn't been made. Also, you'd need to make use of the API that was introduced in #124 to disable host verification.

[yoni386]

I think I made it as you did. I also tried to git reset to this commit. which should version contain it? Thank you

On Fri, 2 Jun 2017 at 1:53 Sean McArthur ***@***.***> wrote: @yoni386 <https://github.com/yoni386> the support that was just merged is only on the master branch at the moment, a release hasn't been made. Also, you'd need to make use of the API that was introduced in #124 <#124> to disable host verification. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2rGhmzKx_eQldU8sAZN8oMK9X5BIks5r_0EEgaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[little-dude]

@yoni386 could you show us the code, the error you get (and potentially your Cargo.toml)?

[yoni386]

I discard all the changes. I think it would probably be best if you add how to clone the commit which includes the relevant changes and how to make sure ssl validation is disable https://stackoverflow.com/questions/12122159/golang-how-to-do-a-https-request-with-bad-certificate.
in your attempts connection to "invalid ssl" works?

[seanmonstar]

If you want to try out a dependency from a git repository, you can do this in your Cargo.toml:

[dependencies]
reqwest = { git = "https://github.com/seanmosntar/reqwest" }

[little-dude]

it would probably be best if you add how to clone the commit which includes the relevant changes

In your Cargo.toml, replace your reqwest dependency by:

[dependencies]
reqwest = { git = "https://github.com/seanmonstar/reqwest" }
# [...]

This tell cargo to use the version of reqwest that is on github.

in your attempts connection to "invalid ssl" works?

TLS connections can be "invalid" in many different ways. The server certificates can have been revoked, expired, or it can have a CN different from the one the client contacted, or it can be signed by a Certificate Authority that is not recognized by the client, etc.

To be clear, reqwest won't let you disable the certificate verification completely like curl does. So if you're trying to connect to a server that has an expired or revoked certificate it won't work. What reqwest allows you to do is:

• disable only the hostname verification
• trust a custom Certificate Authority (in that case you need to have the root certificate of this CA)
• a combination of the above

This is all documented, but the documentation is not online yet, because a new version must be released for that. I suggest that you clone the reqwest repo, and build the documentation yourself:

git clone https://github.com/seanmosntar/reqwest
cd reqwest
cargo doc --open

This should open the documentation for the master branch. You will find examples for SSL configuration.

[yoni386]

extern crate reqwest;

fn main() {

    let url = format!("https://some_foo/rest/v1/");

    let res = reqwest::Client::builder()
        .unwrap()
        .danger_disable_hostname_verification()
        .build()
        .unwrap()
        .get(&url)
        .unwrap()
        .send()
        .unwrap();


    println!("res: is {}", res.status());

}

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Http(Ssl(Failure(Error { code: -67843, message: "The trust policy was not trusted." }))), url: Some(("https://some_foo/rest/v1/") }', src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.
MacBook:rilo yoni$ RUST_BACKTRACE=1 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/rilo`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Http(Ssl(Failure(Error { code: -67843, message: "The trust policy was not trusted." }))), url: Some("https://some_foo/rest/v1/") }', src/libcore/result.rs:859
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::panicking::default_hook::{{closure}}
   2: std::panicking::default_hook
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: std::panicking::begin_panic_fmt
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: <core::result::Result<T, E>>::unwrap
  10: rilo::main
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start
  14: main

toml:

[dependencies]
curl = "0.4.6"
serde_json = "1.0.2"
log = "0.3.7"
env_logger = "0.4.2"
chrono = "0.3.1"
clap = "2.24.2"
#threadpool = "1.3.2"
#scoped_threadpool = "0.1.7"
crossbeam = "0.2.10"

reqwest = { git = "https://github.com/seanmonstar/reqwest" }

[seanmonstar]

If it is a self-signed certificate, you can download it and add it as trusted with client.add_root_certificate(cert).

To be fair, it may be that I shouldn't have closed this issue, since at the top, it does ask to completely disable verification, which is not currently possible. If that's really desirable, I can reopen.

[yoni386]

I think most the servers at backend use internal certificate. Internal infrastructure at enterprise usually work this way.

On Fri, 2 Jun 2017 at 21:56 Sean McArthur ***@***.***> wrote: If it is a self-signed certificate, you can download it and add it as trusted with client.add_root_certificate(cert). To be fair, it may be that I shouldn't have closed this issue, since at the top, it does ask to completely disable verification, which is not currently possible. If that's really desirable, I can reopen. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2qXpNbOg_thatE1-xSLmRTxW9Dd6ks5sAFrtgaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[seanmonstar]

I think most the servers at backend use internal certificate. Internal infrastructure at enterprise usually work this way.

That is true, but usually even then you want to make sure you are securely connecting to internal services correctly, not to some wrong service that someone else setup on the same internal network masquerading as another. So you would likely grab that certificate, and add is as trusted to the client.

[yoni386]

Any idea how much overhead is adding? You mean check if certificate is not listed then add it? Or any time get it and use it. It seems to a lot of logic which doesn't fit to every case. I think optional method is a good idea after all curl, go, python, dart, node offer this.

On Fri, 2 Jun 2017 at 22:36 Sean McArthur ***@***.***> wrote: I think most the servers at backend use internal certificate. Internal infrastructure at enterprise usually work this way. That is true, but usually even then you want to make sure you are securely connecting to internal services correctly, not to some wrong service that someone else setup on the same internal network masquerading as another. So you would likely grab that certificate, and add is as trusted to the client. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2tNL6xKB1CU5x9Ozw120tV-bpHCUks5sAGRZgaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[seanmonstar]

No, I mean you would put of a copy of the public certificate with your app, load it into reqwest::Certificate at startup, add it to the Client, and never worry about it again.

[yoni386]

reqwest::Certificate is the certificate which the client connects to right? If it does it can be thousands different machines and also the client source machine might change from time to time

On Sat, 3 Jun 2017 at 0:29 Sean McArthur ***@***.***> wrote: No, I mean you would put of a copy of the public certificate with your app, load it into reqwest::Certificate at startup, add it to the Client, and never worry about it again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2hzGe-GwiIRgYYMYCoYQmgIYnhPlks5sAH67gaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[little-dude]

@yoni386 reqwest::Certificate is the root certificate of the certificate authority that signed your server certificate. You have to somehow get this certificate. It usually comes in the pem format. You'll have to convert it to der to use it with reqwest.

[yoni386]

*The **certificate authority are public HP, dell and etc. I don't think this is a good idea. BTW how fast this library is? Is it consider faster than curl? * *Thank you *

On Sat, 3 Jun 2017 at 1:12 Corentin Henry ***@***.***> wrote: @yoni386 <https://github.com/yoni386> reqwest::Certificate is the *root certificate* of the *certificate authority* that signed your server certificate. You have to somehow get this certificate. It usually come in the pem format. You'll have to convert it to der to use it with reqwest. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALrs2iKUZfWF7KNcZk2_iFDg4awKmLM0ks5sAIUhgaJpZM4K0ZH8> .

-- Best Regards, Yoni Shperling

[MarkDDR]

Hello, I have a use case where an internal development server uses a self-signed and expired certificate. If an option could be added to allow expired certificates that would be great.

[yoni386]

Hello Mark,

I am not a Author. Maybe the above will also help.
As I know, in rust today only curl can provide those ignores.

[seanmonstar]

I have not really looked into how easy or hard it would be to add. To be used in reqwest, such a feature would need to be added to the native-tls crate first.

[mass10]

+1