Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2014-3230 - don't disable verification if only hostnames should not ... #14

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

noxxi
Copy link

@noxxi noxxi commented May 3, 2014

@mschilli
Copy link
Contributor

mschilli commented May 4, 2014

Thanks for the patch, not sure I understand the fix, though :). The bug says that setting HTTPS_CA_DIR or HTTPS_CA_FILE causes the IO::Socket::SSL to skip checking the server cert.

The IO::Socket::SSL docs say that "The default for SSL_verify_mode on the client is currently SSL_VERIFY_NONE, which is a very bad idea, thus the default will change in the near future.
See documentation for SSL_verify_mode for more information." -- I presume the fix changes this?

The fix uses

 $ssl_opts{SSL_verifycn_scheme} = 'none';

instead of the previous setting for SSL_verify_mode, but the docs say that "If you are really sure, that you don't want to verify the identity using the hostname you can use 'none' as a scheme." -- isn't this the same as disabling the cert check?

Would be great if you could clarify, just want to make sure we're doing the right thing here -- maybe it's just the docs that need to be updated?

@noxxi
Copy link
Author

noxxi commented May 5, 2014

Long story: In LWP version<6 Net::SSL/Crypt::SSLeay was used as default to provide https support. This module got configured with environment variables like HTTPS_CA_FILE etc and while it could check the certificate against the given CA file, it would not verify the host name, e.g. not check if the name in the certificate it got when accessing example.com really matched example.com or was instead attacker.com. When changing the default SSL package to IO::Socket::SSL with LWP 6 it tried to keep this behavior for compatibility reasons, when it detected that HTTPS_CA_* variables where used. It did this by setting the verify_hostname option to 0. This option is described as influencing the checking of the host name, not the checking of the certificate itself (there is no extra option in LWP to completely disable all verifications, one would have to use ssl_opts with SSL_verify_mode to do this).

At about the same time LWP 6 was released I decided to change the default verification for the certificate (not host name, but certificate) in IO::Socket::SSL from the default no verification to verify by default - because users usually don't read enough documentation and enable important options as long as it works. In order to not break existing users too much I've started with a check only, e.g. if SSL_verify_mode was not set by the user it complained loudly, that it is using insecure defaults and how to change the code to fix this message. But obviously several people did get this message wrong and instead of making sure, that it has the necessary information to check the certificate (like CA path) they've decided to switch certification checking off completely, thus keeping everything explicitly insecure.

RT#81948 is one example of how such a fix got wrong. Instead of explicitly enabling verification and setting SSL_ca_file, as recommended by the error message, the patch completely disabled verification if verify_hostname was set to 0. This in effect changed the meaning of verify_hostname from disabling verification of host name to disabling all verification, even for the certificate.

This patch restores the intended behavior of the option, e.g. that it should only disable verification of the host name. If the user still uses an old version of IO::Socket::SSL this will result in getting the warning again. But the real fix here would not be to disable verification, but to enable verification and provide CA settings - or better yet to upgrade to a newer version of IO::Socket::SSL which has secure defaults enabled and thus does not complain anymore.

@jwilk
Copy link
Contributor

jwilk commented May 6, 2014

When changing the default SSL package to IO::Socket::SSL with LWP 6 it tried to keep this behavior for compatibility reasons, when it detected that HTTPS_CA_* variables where used.

Why special case users who set HTTPS_CA_*? "For compatibility reasons" doesn't make sense without further justification, given that you broke compatiblity for everyone else. Please keep in mind that:

  • There's nothing in the names of HTTPS_CA_* that would suggests that these variables are specific to Crypt::SSLeay, or LWP, or even Perl. So people might have them set in their environment for purposes unrelated to Crypt::SSLeay.
  • I suspect that these days many users of LWP don't even know what Crypt::SSLeay is.
  • There is nothing in the LWP documentation that suggests that setting HTTPS_CA_* might have negative security effect.

It did this by setting the verify_hostname option to 0. This option is described as influencing the checking of the host name, not the checking of the certificate itself

If would make sense if verify_hostname influenced only hostname verification (after all, this is what the name says), but:

  • That's not what LWP::UserAgent POD says:
When TRUE LWP will for secure protocol schemes ensure it connects to servers
that have a valid certificate matching the expected hostname.  If FALSE no
checks are made and you can't be sure that you communicate with the expected peer.
  • That's also not what LWP/Protocol/https.pm says:
To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
environment variable to 0. If you do this you can't be sure that you
communicate with the expected peer.
  • There is already another, more flexible option for controlling hostname verification: SSL_verifycn_scheme.

@noxxi
Copy link
Author

noxxi commented May 6, 2014

Hi jwilk,
you don't need to argue with me - I only explained what the intention of the code probably was (according to the comments). I did not do the code myself, but I'm involved in libwww a bit because I'm the maintainer of IO::Socket::SSL and I did patches for LWP to fix proxy support for HTTPS.

Why special case users who set HTTPS_CA_*

If you google for HTTPS_CA_FILE you will probably only find references to LWP/Crypt::SSLeay. So, in a way, it makes sense to special case this because these are mostly users of the older LWP versions. Everybody likes a clean cut and nobody likes to maintain all the old code, but if you maintain a heavily used package like LWP you cannot just make changes which break the code for lots of users.

I suspect that these days many users of LWP don't even know what Crypt::SSLeay is.

That would be nice, but if you look at recent discussions at stackoverflow or perlmonks they regularly try to force LWP to use Crypt::SSLeay when they run into problems and I often try to point them into the right direction, e.g. how to configure IO::Socket::SSL correctly (or in most cases just use a recent enough version).

... If FALSE no checks are made and you can't be sure that you communicate with the expected peer.

The wording in LWP::UserAgent about verify_hostname is a bit unclear, but I read "no checks" as "not the checks for hostname verification which were described in the previous sentence". I would not read it as "no certificate verification at all". But a clearer message would be good.

There is already another, more flexible option for controlling hostname verification: SSL_verifycn_scheme.

Yes and no. In most cases I would prefer to just use the options from IO::Socket::SSL directly. But for hostname verification it makes more sense, that LWP sets SSL_verifycn_scheme to www by default. This makes it easier for the user to get it right, because most users don't even know that there are different schemes how hostnames can be verified. And LWP::Protocol::https already does the right thing, but only if verify_hostname is true (which it is by default). So it should (and does) make it easy to do the right thing, and hard to do the wrong thing.

@ppisar
Copy link

ppisar commented May 9, 2014

Here I wrote a test case [https://bugzilla.redhat.com/attachment.cgi?id=893881]. It would be nice if LWP-Protocol-https got more tests.

@ppisar
Copy link

ppisar commented May 13, 2014

Following patch is needed for IO::Socket::SSL < 1.950 to pass LWP::Protocol::https tests.

diff --git a/lib/LWP/Protocol/https.pm b/lib/LWP/Protocol/https.pm
index 73020dc..7c45fd2 100644
--- a/lib/LWP/Protocol/https.pm
+++ b/lib/LWP/Protocol/https.pm
@@ -24,6 +24,7 @@ sub _extra_sock_opts
            $ssl_opts{SSL_verifycn_scheme} = '';
        } else {
            $ssl_opts{SSL_verifycn_scheme} = 'none';
+           $ssl_opts{SSL_verify_mode} ||= 0;
        }
     }
     if ($ssl_opts{SSL_verify_mode}) {

@noxxi
Copy link
Author

noxxi commented May 13, 2014

+           $ssl_opts{SSL_verify_mode} ||= 0;

This effectively disables certificate verification if not specified different and thus overrides the default of newer IO::Socket::SSL to have verification enabled unless specified otherwise. This in effect recreates the original problem and thus it is not a good idea.
If you get the fat warning that SSL_verify_mode is not set, you should better follow the instruction in the warning and set the mode (and better set it to enable verification) - or just upgrade to a recent version of IO::Socket::SSL which has verification enabled by default.

@karenetheridge
Copy link
Member

On Tue, May 13, 2014 at 07:58:33AM -0700, Steffen Ullrich wrote:

+           $ssl_opts{SSL_verify_mode} ||= 0;

This effectively disables certificate verification if not specified different and thus overrides the default of newer IO::Socket::SSL to have verification enabled. Thus it is not a good idea.

I agree - we should never turn off security features unless the user
explicitly requests it.

@ppisar
Copy link

ppisar commented May 14, 2014

On Tue, May 13, 2014 at 07:58:34AM -0700, Steffen Ullrich wrote:

+           $ssl_opts{SSL_verify_mode} ||= 0;

This effectively disables certificate verification if not specified
different

Not exactly. Have you tried that?

The branch is executed only if verify_hostname is 0. And that happens only if
user explicitly set verify_hostname to 0. Moreover it does not clobber the
value if the value was set explicitly before. Either by user or by
LWP::UserAgent when processing the environment variables which this security
bug is about.

I addition I wrote that this is suitable for IO::Socket::SSL < 1.950. And with
these old versions, it does not weaken the security.

With newer IO::Socket:SSL, more suitable code would be:

$ssl_opts{SSL_verify_mode} ||= 1;

However that's not needed because 1 is already the default value there.

I proposed "||= 0" because of backward compatibility. If you want break it,
let use "||= 1".

However without this code, LWP-Protocol-https test suite will not pass with
IO::Socket::SSL < 1.950.

-- Petr

@noxxi
Copy link
Author

noxxi commented May 14, 2014

The default value of IO::Socket::SSL is only used, if there is no value set by the user, e.g. SSL_verify_mode is undef. But with your proposal you define SSL_verify_mode whenever verify_hostname is set to 0, so that the default setting of IO::Socket::SSL gets not used.

In essence this means, that if the user has set verify_hostname to 0 either through an LWP option or though the environment variable PERL_LWP_SSL_VERIFY_HOSTNAME, it will not only disable the checking of the hostname against the certificate, but also the verification of the certificate against the trusted CAs - unless the user has also explicitly set SSL_verify_mode to 1.

Or in other words: it makes verify_hostname disable the complete certificate verification unless the user explicitly said otherwise. And this is not the expected behavior.

In my opinion the test suite should just be fixed to explicitly add SSL_verify_mode => 0 if it needs to disable verification.

@karenetheridge
Copy link
Member

What would help here (for both future correctness, and to be sure that this
specific change is made correctly) is some unit tests that demonstrated
each scenario: the user sets parameters explicitly to true or false, the
user omits a parameter, etc etc -- and testing the outcome in all cases.

@ppisar, can you include such tests with your patch?

@mschilli
Copy link
Contributor

Am I the only one bothered to see that each of the modules involved (LWP::UserAgent, LWP::Protocol::https, CA::Mozilla, IO::Socket::SSL, Net::SSLeay, Crypt::SSLeay) has their own verification policy and variable names (HTTPS_CA_FILE , SSL_verify_mode, PERL_LWP_SSL_VERIFY_HOSTNAME, SSL_verifycn_scheme, etc.) which makes it really hard to understand what policy is indeed in effect?

@noxxi
Copy link
Author

noxxi commented May 14, 2014

You are definitely not the only one. This is a maintenance, usability and security nightmare. But it all has historical reasons :{.

As far as I know Crypt::SSLeay is/was only used by LWP and is only kind of maintained with no new features and will only be used if for whatever reason IO::Socket::SSL/Net::SSLeay is not possible. So it will never have support for hostname verification or do secure defaults. But, IO::Socket::SSL is used in lots of modules, and because you cannot expect everybody to understand the complexity of SSL, it evolved into providing secure as possible defaults. If I would be sure that IO::Socket::SSL/Net::SSLeay supports every platform Crypt::SSLeay does, I would suggest to cleanup LWP and remove any support and compatibility hacks for Crypt::SSLeay.

@ppisar
Copy link

ppisar commented May 15, 2014

On Wed, May 14, 2014 at 09:39:50AM -0700, Karen Etheridge wrote:

What would help here (for both future correctness, and to be sure that this
specific change is made correctly) is some unit tests that demonstrated
each scenario: the user sets parameters explicitly to true or false, the
user omits a parameter, etc etc -- and testing the outcome in all cases.

@ppisar, can you include such tests with your patch?

If you mean test for this CVE, then I've already linked the test before
in comment #14 (comment).

If you mean test for the IO::Socket::SSL warning, then it's already included
in the t/apache.t under name "no warning seen".

If you mean general tests covering all possible scenarios, then it's much more
work.

At the end, instead of adding the controversal "$ssl_opts{SSL_verify_mode} ||=
0", you can just declare that the LWP-Protocl-https nedes IO::Socket::SSL >=
1.950. Or you can skip the "no warning seen" test in case of older
IO::Socket::SSL.

-- Petr

@noxxi
Copy link
Author

noxxi commented May 15, 2014

I've contacted the maintainer of Crypt::SSLeay to see if he sees any problems with removing Crypt::SSLeay support from LWP - https://rt.cpan.org/Ticket/Display.html?id=95663

@noxxi
Copy link
Author

noxxi commented May 15, 2014

At the end, instead of adding the controversal "$ssl_opts{SSL_verify_mode} ||=
0", you can just declare that the LWP-Protocl-https nedes IO::Socket::SSL >=
1.950. Or you can skip the "no warning seen" test in case of older
IO::Socket::SSL.

Yes, if we just merge #15 we would remove all of these problems, because this requires a recent IO::Socket::SSL too.

@ciz
Copy link

ciz commented May 15, 2014

@noxxi , could you please fix the Debian bug number in the pull request,
to prevent confusion?
Debian #746576 is some problem with Italian translation.

@noxxi noxxi changed the title Debian #746576 - don't disable verification if only hostnames should not ... Debian #746579 - don't disable verification if only hostnames should not ... May 15, 2014
@noxxi
Copy link
Author

noxxi commented May 15, 2014

@noxxi , could you please fix the Debian bug number in the pull request,

Thanks for pointing this out. It is fixed now.

@mschilli
Copy link
Contributor

Okay, why don't we take a step back and simplify this to the point where everybody can understand what the cert verification policy actually is? Look at curl, for example:

-k, --insecure
          (SSL) This option explicitly allows curl to  perform  "insecure"
          SSL connections and transfers. All SSL connections are attempted
          to be made secure by using the CA certificate  bundle  installed
          by  default.  This  makes  all connections considered "insecure"
          fail unless -k, --insecure is used.

To me, the cert check is either on or off. Secure or insecure. No one cares whether it's insecure because the host name isn't checked or the cert isn't checked or some other variable has a special value.

So why don't we change IO::Socket::SSL to have an option to switch the cert check on or off (preferably something human-readable like "ssl_insecure => 1/0", defaults to 0) and a path to the CA file (something like ssl_cacert), and get rid of all those other confusing options?

@noxxi
Copy link
Author

noxxi commented May 15, 2014

Since 1.970 the hostname will not be automatically checked, if SSL_verify_mode is 0, so this effectively works like the -k option of curl. As for other options: I think that the average user does not need to deal with any options at all, it should just work like in the browser. Only in special cases the user has to deal with the following options:

  • SSL_fingerprint if he needs to accept a self-signed certificate. Just switching the verification completely off is not a good idea. This also disables hostname verification, so it works with local routers etc which often have a self-signed certificate which does not match the hostname.
  • SSL_ca* if the user needs a special CA store. But usually the default store should be enough.
  • And finally he can switch all verification off with SSL_verify_mode is set to 0, if it does not care about security at all.

I agree that the user should not need to distinguish between verification of the hostname and verification of the certificate: like this bug and this discussion shows it is already hard for developers to distinguish between these things. And it also does not make sense, because if hostname verification is disabled a man-in-the-middle could present just a correctly signed certificate for another host. But this effectively means, that we cannot use Crypt::SSLeay any longer, because it does not support verification of the hostname at all. Or one has to copy the relevant code over from IO::Socket::SSL or reimplement the www parts without getting them wrong like others (Ruby and wget have the relevant checks wrong, Python fixed it in the last version).

And I'm not sure if one should make it too easy for the user to disable verification of the certificate (and definitely not with a simple environment variable). Most users don't understand SSL and don't know that it is not possible to get secure end-to-end encryption without verifying the certificate. Even software like OTRS or FosWiki just disabled certificate verification permanently when sending mail with SSL, without giving the user a way to enable it. It would be nice of more documentation would help, but I doubt it :( Maybe one should name the option "SSL_i_dont_care_about_security_at_all" :)

jsonn pushed a commit to jsonn/pkgsrc that referenced this pull request May 16, 2014
jsonn pushed a commit to jsonn/pkgsrc that referenced this pull request Jun 2, 2014
@thoger
Copy link

thoger commented Jun 6, 2014

Is there a clear conclusion on what setting of verify_hostname or PERL_LWP_SSL_VERIFY_HOSTNAME to 0 is meant to do? It seems one of the problems where is a lack of clear definition of that, possibly combined with a lack of understanding of difference between certificate verification and host name verification (understandable, as you typically want none or both checks done, even though OpenSSL only helps/helped with the former).

As noted in @jwilk's comment, LWP::UserAgent documentation is ambiguous and can be read both ways. On the other hand, error message from LWP/Protocol/https.pm indicates verify_hostname 0 is meant to disable all checks, not only host name check. Proposed fix seems incorrect under that assumption.

Few considerations:

  • Should the meaning of verify_hostname 0 depend on the default value of SSL_verify_mode in the underlying IO::Socket::SSL, or should it be defined in LWP?
  • Is it reasonable to expect that HTTPS_CA_FILE=/path/to/ca.crt, and PERL_LWP_SSL_CA_FILE=/path/to/ca.crt with PERL_LWP_SSL_VERIFY_HOSTNAME=0 do the same thing? It's not the case pre-patch, and it depends on IO::Socket::SSL defaults post-patch.
  • Various Internet resources already document PERL_LWP_SSL_VERIFY_HOSTNAME=0 as the solution to disable all SSL certificate checks in LWP. It does that today, afaics it did that even before commit bcc46ce, when IO::Socket::SSL was still defaulting to SSL_verify_mode=0. Does it make sense to break its meaning and make it only disable host name checks, which is typically not very useful, without providing alternative to disable all checks? I realize that the name is confusing if it's meant to disable all certificate checks.

@noxxi
Copy link
Author

noxxi commented Jun 6, 2014

I think the name of the key "verify_hostname" clearly states, that it is the verification of the hostname only, and nothing else. So any documentation should reflect the naming of the key and not re-interprete "hostname verification" as "certificate verification". And for security it would be fatal, if the user just wants to disable hostname verification, as the name of the key suggests, but it would disable the complete certificate verification.

Apart from that, this key is just a hack. It controls only a small (and often misunderstood) aspect of the certificate verification and there is no configuration in LWP itself available for the rest. And I don't think it is even a good idea to let users disable certificate validation partly or completely in an easy way. Look at http://www2.dcsec.uni-hannover.de/files/android/p50-fahl.pdf what happens, if developers just disable SSL validation because it is more convenient.

I propose to:

  • Remove this key and the environment variable from the code.
  • Leave this key in the documentation for a while, but document it as removed and point to the documentation of IO::Socket::SSL in case the user really wants to disable validation. I plan a major rewrite of the documentation in the next time, so that it should be easy to find what to do in case of validation problems (with disabling validation only as the last resort).
  • Integrate Make explicit requirement of Mozilla::CA obsolete #15, so that it will use the default certificate store of the OS if possible (UNIX etc) or fall back to Mozilla::CA. This will make the validation behave more as expected.

Sure, this might break code which just disabled certificate validation. But, if the user just disabled verification because it somehow did not work at this time, it might magically continue to work now with verification enabled (because of fixes in the code, usable CAs etc). And if the user disabled validation because this were non-public certificates, it might even be a good idea that it fails now and that the user learns how to do a proper validation by reading from documentation (e.g. importing CA or checking against a fingerprint).

Sure, it might break the accuracy of internet resources. But there are enough resources out there which at the slightest problem recommend users to disable validation. Lots of these problems just have to do with missing certificate store or failure to use SNI, which are all solved with current versions of IO::Socket::SSL (but not Crypt::SSLeay). So I'm very glad if these bad recommendations don't work any longer and we can show the users the correct way.

Remember, this is security. So it should be easy to do the right thing and it should be hard to do the wrong thing.

@thoger
Copy link

thoger commented Jun 9, 2014

I think the name of the key "verify_hostname" clearly states, that it is the verification of the hostname only, and nothing else. So any documentation should reflect the naming of the key and not re-interprete "hostname verification" as "certificate verification". And for security it would be fatal, if the user just wants to disable hostname verification, as the name of the key suggests, but it would disable the complete certificate verification.

I agree that the name is plain bad for something that actually disables all checks. I wanted to note that LWP already recommends using that (via the matching environment variable) to disable both checks, at that it seems that it always worked as an option to disable both checks. So harm was done already, and you seemed to prefer to avoid breaking existing behaviour even if it's bad and not documented (name checks disabled for HTTPS_CA_*).

I had a look at the history, and it's quite messy. Initially, LWP documentation mentioned verify_hostname and SSL_verify_mode as separate ssl_opts, with PERL_LWP_SSL_VERIFYPEER environment to disable checks. The code was later "simplified" to only support/document verify_hostname, renaming environment to PERL_LWP_SSL_VERIFY_HOSTNAME. AFAICS, this continued to disable certificate checks. See 62dd581, 4fe1741, 5185183, bd3cc5e, and 48b4a77.

  • Remove this key and the environment variable from the code.
  • Leave this key in the documentation for a while, but document it as removed and point to the documentation of IO::Socket::SSL in case the user really wants to disable validation.

It makes sense to provide environment variable that disables all checks, for use cases where user don't want to or can't change a script to pass correct ssl_opts. This would require a decision on how to interact with conflicting settings from ssl_opts.

@noxxi
Copy link
Author

noxxi commented Jun 9, 2014

..I wanted to note that LWP already recommends using that (via the matching environment variable) to disable both checks, at that it seems that it always worked as an option to disable both checks...

From the documentation the verify_hostname option in LWP::UserAgent version 6.06:

When TRUE LWP will for secure protocol schemes ensure it connects to servers that have a valid
certificate matching the expected hostname. If FALSE no checks are made and you can't be sure
that you communicate with the expected peer. The no checks behaviour was the default for 
libwww-perl-5.837 and earlier releases.
This option is initialized from the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable. If
this environment variable isn't set; then verify_hostname defaults to 1.

So here it states clearly, that

  • The option affects the matching of the hostname: "...valid certificate matching the expected hostname.".
  • It will be initialized from the PERL_LWP_SSL_VERIFY_HOSTNAME variable and will be default to 1 if this variable is not set.

If the part "If FALSE no checks are made" is taken out of the context of this key, it would claim that no (unspecified) checks are done. In this case one could assume that these unspecified checks refer to all certification checks. But, in the context of this key it describes only a single check: the verification of the hostname against the certificate.

That is the only documentation of this variable in the documentation I can find and I cannot agree that it recommends to disable both checks. And from the git history this documentation never claimed this behavior. So there might be articles out there claiming this meaning, but not the documentation of LWP itself.

There is a comment inside LWP::Protocol::https which interprets the key differently, but this is no user documentation (e.g. not available with perldoc).

It makes sense to provide environment variable that disables all checks, for use cases where user don't want to or can't change a script to pass correct ssl_opts. This would require a decision on how to interact with conflicting settings from ssl_opts.

I disagree. If you look into the efforts of the browsers to teach people that disabling verification is bad and to make it hard to disable it, I see no reason to make it easy for script users to disable verification. It should be possible for a developer (but not to easy), but much harder for a user.

But, maybe one can find a compromise by adding a variable, which should not be set simply to a true value, but to a specific string like "I really want to disable any certificate checks and know that this is completely insecure". This is a strategy some tools use to make sure, that the user really read the documentation and is really sure, what (s)he s doing. And it has the added bonus of showing up always when somebody documents how to disable verification for LWP, thus pointing out the problems with disabling verification.

@thoger
Copy link

thoger commented Jun 9, 2014

So there might be articles out there claiming this meaning, but not the documentation of LWP itself.

I as referring to this message, which was already quoted in one of the comments above:

https://github.com/libwww-perl/lwp-protocol-https/blob/c5f6cc5/lib/LWP/Protocol/https.pm#L39

To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
environment variable to 0. If you do this you can't be sure that you
communicate with the expected peer.

@jwilk
Copy link
Contributor

jwilk commented Jun 9, 2014

The option affects the matching of the hostname: "...valid certificate matching the expected hostname.".

Yeah, valid certificate. When I first noticed that PERL_LWP_SSL_VERIFY_HOSTNAME disables all the certificate check I thought “what the hell” and was going to file a bug…
But then I read the LWP::UserAgent POD, and realized that the implementation is consistent with what documentation says. So I assumed it was just a poor terminology choice, and shrugged it off.

Incidentally, LWP::protocol::https POD says:

If hostname verification is requested by LWP::UserAgent's C<ssl_opts>, and
neither C<SSL_ca_file> nor C<SSL_ca_path> is set, then C<SSL_ca_file> is
implied to be the one provided by Mozilla::CA.  If the Mozilla::CA module
isn't available SSL requests will fail.  Either install this module, set up an
alternative C<SSL_ca_file> or disable hostname verification.

which seems to confirm the hypothesis that in the LWP world “hostname verification” might mean something different than in the rest of the universe.

That is the only documentation of this variable in the documentation I can find

There's also LWP POD:

=item PERL_LWP_SSL_VERIFY_HOSTNAME

The default C<verify_hostname> setting for C<LWP::UserAgent>.  If
not set the default will be 1.  Set it as 0 to disable hostname
verification (the default prior to libwww-perl 5.840.

And the changelog:

For https://... default to verified connections with require IO::Socket::SSL
and Mozilla::CA modules to be installed.  Old behaviour can be requested by
setting the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable to 0.  The
LWP::UserAgent got new ssl_opts method to control this as well.

maybe one can find a compromise by adding a variable, which should not be set simply to a true value, but to a specific string like "I really want to disable any certificate checks and know that this is completely insecure".

I don't think offending users' intelligence is a valid strategy for writing security-sensitive software (or any software, really).

@noxxi
Copy link
Author

noxxi commented Jun 9, 2014

@thoger :
You cite a comment inside the source code of LWP::Protocol::https. Like I said, I don't think developer comments inside the source code count as user documentation.

@jwilk :
I'm not a native english speaker, but I agree that your interpretation of the documentation is also possible, even if I would not interpret it that way. Whatever the final decision will be, the documentation should be fixed to make this decision easy to understand.

As for the environment variable: I don't want to offend the user, but I don't like the idea of making it too easy for the user to just disable verification without thinking too much about it. There is a reason, that the browsers made it harder to disable verification over the years and tried to add more documentation in their dialogs to make the users aware of the implications.

If you read stackoverflow or similar, it is often the first attempt on any SSL problems to disable certificate verification. And lots of users assume, that they still have encryption without certificate validation, but don't realize that they are open to man-in-the-middle attacks and that all the encryption does not help against it. Thus my proposal might be a way to get your wish of an easy setting with an environment variable, but make the user at least think a bit longer when using it.

@noxxi noxxi changed the title Debian #746579 - don't disable verification if only hostnames should not ... CVE-2014-3230 - don't disable verification if only hostnames should not ... Jun 10, 2014
@noxxi
Copy link
Author

noxxi commented Jun 10, 2014

The discussion is moving away from the original bug. This bug has a CVE now (e.g. it is considered a security problem) and I would suggest that we make a release out of the fixes, instead of letting everybody just pull in these fixes and apply them to their version (at least Debian and RedHat/Fedora did already).

This patch does not interprets verify_hostname in a new way, but instead restores the behavior this option had from version 6.02 (03/2011) until version 6.04 (03/2013), e.g. before the bug was introduced by trying to fix RT#81948. The documentation for this option did not change all the time.

I would suggest that we continue the rest of the discussion, e.g. adding an environment variable to disable all certification and how to fix the documentation, in new issues.

jsonn pushed a commit to jsonn/pkgsrc that referenced this pull request Jun 11, 2014
jsonn pushed a commit to jsonn/pkgsrc that referenced this pull request Oct 11, 2014
@kappa
Copy link

kappa commented Nov 9, 2015

Any news on this?

@alanrocker
Copy link

Trying to tidy up the outstanding tickets for this. Can this now be closed as fixed?

@noxxi
Copy link
Author

noxxi commented Apr 1, 2017

Trying to tidy up the outstanding tickets for this. Can this now be closed as fixed?

From looking at the code the issue is still the same. It is disabling any kind of validation ( hostname and trust chain and expiration and ...) if verify_hostname is 0 even though the documentation in LWP::UserAgent clearly states that this setting is relevant for hostname validation only:

When TRUE LWP will for secure protocol schemes ensure it connects to servers
that have a valid certificate matching the expected hostname. If FALSE no
checks are made and you can't be sure that you communicate with the expected peer.

This also means that the CVE from 3 years ago are still unresolved and that the issue is not fixed and that various distributions like Debian maintain there own patches to fix the CVE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
ToDo
  
To do
Development

Successfully merging this pull request may close these issues.

None yet

9 participants