Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Security: Disallow anything but TLS 1.0+ #894

Closed
davidstrauss opened this issue Oct 17, 2014 · 43 comments
Closed

Security: Disallow anything but TLS 1.0+ #894

davidstrauss opened this issue Oct 17, 2014 · 43 comments

Comments

@davidstrauss
Copy link

Drush performs multiple operations (downloading modules, etc.) using libcurl, but it does not currently restrict connections to TLS 1.0+. This could allow tampering or intercepting transferred data.

I'm not submitting this privately because POODLE is a generally known vulnerability that affects all SSL/TLS applications, and Drush is no more vulnerable than Chrome or Firefox right now.

@weitzman
Copy link
Member

I pushed one commit which changed two urls to from http to https. Could you elaborate on how Drush should call curl and wget in order to be more secure?

I noticed that the following does not work with https - git clone https://git.drupal.org/project/devel.git.

@davidstrauss
Copy link
Author

For cURL, this will allow any TLS version:

curl_setopt($curl, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1);

For wget, this may work, but I'm unsure whether it prevents use of TLSv1.1+:

--secure-protocol=TLSv1

@davidstrauss
Copy link
Author

As for git.drupal.org, would you mind emailing Rudy to let him know, with your example and output? If you can't, let me know so I can follow up with him.

@weitzman
Copy link
Member

I would love a Pull Request for this issue, as I'm not 100% too familiar
with the problem. The function to look at is _drush_download_file() -

function _drush_download_file($url, $destination, $overwrite = TRUE) {

@weitzman
Copy link
Member

weitzman commented Nov 9, 2014

I added --tlsv1 to the curl invocation.

For wget, --secure-protocol=TLSv1 means that the server can choose any SSL type. At least thats how I read http://www.gnu.org/software/wget/manual/html_node/HTTPS-_0028SSL_002fTLS_0029-Options.html. When I try to use 1_1, and 1_2, I am seeing wget: --secure-protocol: Invalid value ‘TLSv1_2’.. So, I did not change wget at all.

Note that Drush currently uses wget over curl when both are present on the CLI. We might want to change this.

@greg-1-anderson
Copy link
Member

Maybe we should fetch with goutte.

@weitzman
Copy link
Member

We have bash at our disposal. I'm not too keen on adding large dependencies for minor gains.

@jhedstrom
Copy link
Member

Drupal.org switched from Edgecast to Fastly today, and dl or pm-releases fails with this error:

Executing: wget -q --timeout=30 -O /tmp/download_filegZW5p5 https://updates.drupal.org/release-history/drupal/8.x
simplexml_load_file(): I/O warning : failed to load external entity "" Project.php:72 [0.39 sec, 6.3 MB]

Running the wget command manually:

Resolving updates.drupal.org (updates.drupal.org)... 23.235.47.249, 199.27.79.249
Connecting to updates.drupal.org (updates.drupal.org)|23.235.47.249|:443... connected.
ERROR: no certificate subject alternative name matches
    requested host name ‘updates.drupal.org’.
To connect to updates.drupal.org insecurely, use `--no-check-certificate'.

Invalid cert in action: https://updates.drupal.org/release-history/drupal/8.x

@jhedstrom jhedstrom reopened this Jun 9, 2015
@greg-1-anderson
Copy link
Member

If you look at one of those updates.drupal.org URLs in a web browser, you can examine the certificate and see that the name in the cert is not correct.

@greg-1-anderson
Copy link
Member

Maybe the cert is correct, but wget does not interpret it correctly?

c.f. https://docs.fastly.com/guides/ssl/why-do-i-see-ssl-certificate-errors-when-using-wget

@ryanaslett
Copy link

Drupal.org does not support https on updates.drupal.org. It was an accident of edgecast before, but not officially supported. We'll have it soon, but the api for updates.drupal.org is http only at the moment.

@greg-1-anderson
Copy link
Member

--no-check-certificate does appear to happen to work at the moment (an accident of fastly?). Maybe it's better to use that than to go all the way back to http?

@grigarr
Copy link

grigarr commented Jun 9, 2015

Drupal core does not support HTTPS for updates requests, YMMV with any use of https://updates.drupal.org until there is official support in core. HTTPS is currently a best effort thing. That said, we are planning to get https enabled with a proper certificate for updates.d.o and ftp.d.o, but the discussion for adding https support started this morning, and I can't predict the timeline for that quite yet.

@greg-1-anderson
Copy link
Member

So, in my testing, after wget fails, Drush tries again with:

$file = @file_get_contents($url)

This works for me. php 5.5.21. Maybe we can let this go for a while?

@webflo
Copy link
Contributor

webflo commented Jun 10, 2015

@greg-1-anderson It does not work on PHP 5.6 because 5.6 valides the SSL Cert? Here is a Travis Build with 5.6 https://travis-ci.org/hechoendrupal/DrupalConsole/jobs/66205359

@weitzman
Copy link
Member

I understand that Drupal does not use https for updates.drupal.org, but I took David's proposal to mean that infra supports it.

Even if we change this for drush6 and drush7 today, it will take a long time for folks to update their Drush. How Feasible is it for the Drupal Infra team to roll back the Fastly switch for the updates.drupal.org domain? That would fix everyone's woes immediately and then https support at Fastly could be added at a relaxed pace.

@nnewton
Copy link

nnewton commented Jun 10, 2015

hmm...

Indeed HTTPS was not really 'supported' on updates, largely because its not used. (the percentage of https access is minuscule) We were unaware drush started using it and indeed the old versions are a problem.

Rudy is sending the fastly conversation to me so I can look at where things stand. I'll update with how possible getting https working on fastly is and what we decide we can do.

@sajt
Copy link

sajt commented Jun 10, 2015

Hi!

My solution. I created a ~/.wgetrc and I putted the following:

check_certificate = off

Now it is working for me.

@nnewton
Copy link

nnewton commented Jun 10, 2015

@weitzman We have a solution we are going to look at.

The problem (as it always is), is usage stats and purging. Its why this is on fastly to begin with.

@greg-1-anderson
Copy link
Member

Here's a much better workaround for the missing SSL cert on fastly.

For Drush Make:

https://gist.github.com/mikeyp/e517bf75611f69d702e3

For Drush pm-download:

https://gist.github.com/greg-1-anderson/fb1dba054c35c46af644

It is still important to not just set-and-forget this (perhaps we should enhance the pm code to WARN on insecure connections, for future), but at least this way you only affect Drush, not all wget applications.

@grigarr
Copy link

grigarr commented Jun 10, 2015

Unofficial https support is back. We now have the following configuration to alleviate the pains of using drush's https:

edgecast https -> fastly http -> updates-origin http

I have a call with Fastly tomorrow to get things moving on https support with them, and we should have something free of this work around mid-late next week.

@greg-1-anderson
Copy link
Member

Great! Thanks for the fast action!

@grigarr
Copy link

grigarr commented Jun 25, 2015

We now have a shared SAN wildcard cert on Fastly for updates (and ftp.drupal.org). EdgeCast is no longer fronting the updates traffic.

@weitzman
Copy link
Member

Thanks @grigarr. If folks are interested, I can submit a patch for Drupal and packaging script to start using SSL for update XML and ftp site. Not urgent.

@steinmb
Copy link

steinmb commented Jul 7, 2015

Did vi really fix this? I see this running the latest of master branch (commit:ec8089f)

Running drush download in debug

drush dl drupal -d

Starting Drush preflight. [0.01 sec, 2.65 MB]
Cache HIT cid: 8.0-dev-commandfiles-0-ea4a438ec021f3b5262a9027db9d3c35 [0.02 sec, 2.72 MB]
Bootstrap to phase 0. [0.36 sec, 8.44 MB]
Found command: pm-download (commandfile=pm) [0.37 sec, 8.44 MB]
Loading version_control engine. [0.37 sec, 8.45 MB]
Loading package_handler engine. [0.37 sec, 8.47 MB]
Executing: wget --version
Loading release_info engine. [0.38 sec, 8.5 MB]
Including /Users/steinmb/.composer/vendor/drush/drush/commands/pm/download.pm.inc [0.39 sec, 8.55 MB]
Calling hook drush_pm_download_validate [0.39 sec, 8.65 MB]
Returned from hook drush_pm_download_validate [0.39 sec, 8.65 MB]
Calling hook drush_pm_download [0.39 sec, 8.67 MB]
Downloading release history from https://updates.drupal.org/release-history/drupal/7.x [0.39 sec, 8.79 MB]
Executing: wget --version
  GNU Wget 1.16.3 built on darwin14.1.0.

  +digest +https +ipv6 -iri +large-file -nls +ntlm +opie -psl +ssl/openssl

  Wgetrc:
      /usr/local/etc/wgetrc (system)
  Compile:
      clang -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
      -DLOCALEDIR="/usr/local/Cellar/wget/1.16.3/share/locale" -I.
      -I../lib -I../lib -I/usr/local/opt/openssl/include -DNDEBUG
  Link:
      clang -DNDEBUG -L/usr/local/opt/openssl/lib -lssl -lcrypto -ldl -lz
      ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a

  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later
  <http://www.gnu.org/licenses/gpl.html>.
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

  Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
  Please send bug reports and questions to <bug-wget@gnu.org>.
Executing: wget -q --timeout=30 -O /private/tmp/download_filelVzFjp https://updates.drupal.org/release-history/drupal/7.x
simplexml_load_file(): I/O warning : failed to load external entity "" Project.php:72 [0.75 sec, 8.79 MB] [warning]
Failed to get available update data from https://updates.drupal.org/release-history/drupal/7.x [0.75 sec, 8.8 MB] [error]
Could not download requested project(s). [0.75 sec, 8.8 MB][error]
Returned from hook drush_pm_download [0.75 sec, 8.8 MB]
Command dispatch complete [0.75 sec, 8.76 MB]

Manual testing:

wget --timeout=30 https://updates.drupal.org/release-history/drupal/7.x
--2015-07-07 19:45:47--  https://updates.drupal.org/release-history/drupal/7.x
Resolving updates.drupal.org... 185.31.17.68
Connecting to updates.drupal.org|185.31.17.68|:443... connected.
ERROR: cannot verify updates.drupal.org's certificate, issued by 'CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE':
  Issued certificate has expired.
To connect to updates.drupal.org insecurely, use `--no-check-certificate'.

@greg-1-anderson
Copy link
Member

This is working for me right now -- even after removing my cached updatexml files. I did notice, however, that after fetching the location for the release from updatexml via https, Drush goes ahead and uses whatever is specified in the XML file to download the actual file contents. This typically is 'http://ftp.drupal.org'. Manually testing a fetch from 'https://ftp.drupal.org' seemed to work just fine; perhaps we should swap in 'https' in place of 'http' here.

@grigarr
Copy link

grigarr commented Jul 7, 2015

Part of adding the certificate support to Fastly also means we can use https for ftp.drupal.org in addition to updates.drupal.org -- This support shouldn't go away, and is part of our longer term plan to force https on *.drupal.org

@weitzman
Copy link
Member

weitzman commented Jul 7, 2015

@greg-1-anderson Did you test with PHP 5.6? Thats the one that had problems originally.I have not tested

The packaging script is what needs to start pointing to https for ftp.drupal.org. No Drush changes needed there.

@greg-1-anderson
Copy link
Member

I am still on php 5.5; have not tried 5.6. I suppose it's time to upgrade to 5.6 everywhere & not look back. (http://php.net/supported-versions.php)

I added #1481 as a proposal; not sure if it helps enough to be valuable, though.

@steinmb
Copy link

steinmb commented Jul 8, 2015

My report ^^ was by using PHP 5.6. Have not tested with PHP 5.5 but I guess it works with that version. Soon time to leave PHP 5.5 :)

@greg-1-anderson
Copy link
Member

I just upgraded to php 5.6, and still could not reproduce the problem above. I have to imagine that the problem is highly localized to the system that is failing; if pm-download did not work, I am sure that we would get a lot of reports about it.

@steinmb
Copy link

steinmb commented Jul 8, 2015

Thanks for your swift reply. Since it was wget that failed did I remove it (brew uninstall wget) and drush started working again, forcing it to use curl I guess. Reinstalled wget from homebrew again (brew install wget) but was no longer able to reproduce the problem. huh?

wget though still fail if I manually test it

wget --timeout=30 https://updates.drupal.org/release-history/drupal/7.x
--2015-07-08 23:42:23--  https://updates.drupal.org/release-history/drupal/7.x
Resolving updates.drupal.org... 185.31.17.68
Connecting to updates.drupal.org|185.31.17.68|:443... connected.
ERROR: cannot verify updates.drupal.org's certificate, issued by 'CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE':
  Issued certificate has expired.
To connect to updates.drupal.org insecurely, use `--no-check-certificate'.

Is using curl stored somewhere? Does not look like it. It testes and use wget, does not run curl --version if it first find wget:

Executing: wget --version
Loading release_info engine.

@greg-1-anderson
Copy link
Member

Yeah, if wget is in the $PATH, it will be used. Might be reasonable to add an option to select the preferred file-getting utility, but this would be a separate PR.

@geerlingguy
Copy link

I was getting the following with Drush master (just pulled latest changes to make sure I'm up to date, and ran composer install again) and PHP 5.6.10 on my Mac:

Beginning to build                                                   [ok]
/Users/jeff.geerling/Desktop/site/scripts/project.make.yml.
simplexml_load_file(): I/O warning : failed to load external entity  [warning]
"" Project.php:72
Failed to get available update data from                             [error]
https://updates.drupal.org/release-history/zen/7.x

I then did brew uninstall --force wget to forcibly uninstall wget, and everything worked fine.

I then reinstalled wget with brew install wget, and everything continued working correctly. My best guess is there was an old version of wget in the system somewhere clogging up the works with an old cert file or something.

@mccrodp
Copy link

mccrodp commented Oct 27, 2015

Thanks for the above, but it did not work for me, re: reinstalling wget via homebrew.

==> Summary
🍺  /usr/local/Cellar/wget/1.15_2: 9 files, 908K, built in 105 seconds
Pauls-MacBook-Air:contrib paulmccrodden$ composer global update
Changed current directory to /Users/paulmccrodden/.composer
Loading composer repositories with package information
Updating dependencies (including require-dev)
Generating autoload files
Pauls-MacBook-Air:contrib paulmccrodden$ drush dl --select field_collection
simplexml_load_file(): I/O warning : failed to load external entity "" Project.php:72                        [warning]
Failed to get available update data from https://updates.drupal.org/release-history/field_collection/7.x     [error]
Could not download requested project(s).

I'm running MAMP with php5.5.26.

I double checked wget is correctly pointing to this update and it seems to be.

Pauls-MacBook-Air:contrib paulmccrodden$ which wget 
/usr/local/bin/wget
Pauls-MacBook-Air:contrib paulmccrodden$ ls -altr /usr/local/bin/wget
lrwxr-xr-x  1 paulmccrodden  admin  30 27 Oct 09:50 /usr/local/bin/wget -> ../Cellar/wget/1.15_2/bin/wget

Strange thing is though, running drush works with some other projects such as run drush dl --select migrate

@jerbob92
Copy link

The weird thing is:

$ wget http://updates.drupal.org/release-history/linkit/7.x

--2015-10-27 10:52:38--  http://updates.drupal.org/release-history/linkit/7.x
Resolving updates.drupal.org (updates.drupal.org)... 23.235.43.68
Connecting to updates.drupal.org (updates.drupal.org)|23.235.43.68|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-10-27 10:52:38 ERROR 404: Not Found.

$ wget https://updates.drupal.org/release-history/linkit/7.x
--2015-10-27 10:55:13--  https://updates.drupal.org/release-history/linkit/7.x
Resolving updates.drupal.org (updates.drupal.org)... 185.31.17.68
Connecting to updates.drupal.org (updates.drupal.org)|185.31.17.68|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-10-27 10:55:13 ERROR 404: Not Found.

(both http and https)

While visiting same URL through browser gives a correct XML. I think something else is going on right now but I didn't figure it out yet.

@jerbob92
Copy link

I think I figured it out. They now only support compressed requests:

$ wget https://updates.drupal.org/release-history/linkit/7.x --header="accept-encoding: gzip"
--2015-10-27 11:00:14--  https://updates.drupal.org/release-history/linkit/7.x
Resolving updates.drupal.org (updates.drupal.org)... 23.235.43.68
Connecting to updates.drupal.org (updates.drupal.org)|23.235.43.68|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3211 (3,1K) [text/xml]
Saving to: ‘7.x’

100%[==========================================================================================================================================================================>] 3.211       --.-K/s   in 0s      

2015-10-27 11:00:15 (216 MB/s) - ‘7.x’ saved [3211/3211]

@mccrodp
Copy link

mccrodp commented Oct 27, 2015

Interesting, I tried with your example for linkit above in the browser, I saw the XML.
As you found, I still get the same I/O error using drush.

However, I found field_collection's update URL https://updates.drupal.org/release-history/field_collection/7.x gives a "Not Found". Never come across this before, where do you even file an error like that, it's not really on the module...is it on project module https://www.drupal.org/project/project?

EDIT: looks like you're making progress with the compressed requests :)

@jerbob92
Copy link

Ah, found the issue about in Drupal.org while trying to create one:
https://www.drupal.org/node/2602264

@joejoseph00
Copy link

had this same problem on latest drush 7.4.0 , I am using php 5.6.x
workaround here: https://gist.github.com/greg-1-anderson/fb1dba054c35c46af644
did the trick , works for me after following above workaround.

@cecrs
Copy link

cecrs commented Sep 7, 2017

uninstalling and reinstalling wget was what finally resolved this issue for me. Thanks @geerlingguy!

@tanc
Copy link

tanc commented Jan 29, 2018

I had to uninstall brew's openssl and then re-install it with:

brew uninstall --ignore-dependencies openssl
brew install openssl

@anotherjames
Copy link

anotherjames commented Jul 25, 2018

I ran into similar issues to this too, when running drush make/remake with patch files (with errors saying 'Unable to download ...') ... partly because drupal.org amazingly returns a 403 for .diff files when there is no user agent string!
I needed to upgrade my wget with homebrew (to at least 1.15 I believe, though I ended up with 1.19.5), to avoid failing when there are 'non-fatal' TLS alerts (and sets a user agent string). That means wget succeeds in writing the contents to the temporary file, and so drush doesn't end up falling back to file_get_contents(), which doesn't (by default) have a user agent string set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests