Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wp core download gives "unable to decompress gzipped phar archive" error #4370

Closed
Flimm opened this Issue Sep 20, 2017 · 80 comments

Comments

@Flimm
Copy link

Flimm commented Sep 20, 2017

We have a nightly script that downloads and reinstalls WordPress on a testing server. Last night, it failed like this:

$ wp core --debug download
Debug (bootstrap): No readable global config found (0.067s)
Debug (bootstrap): No project config found (0.068s)
Debug (bootstrap): argv: /usr/bin/wp core --debug download (0.068s)
Debug (bootstrap): ABSPATH defined: /var/www/html/ (0.068s)
Debug (bootstrap): Running command: core download (0.069s)
Downloading WordPress 4.8.2 (en_US)...
md5 hash verified: 2e8744a702a3d9527782d9135a4c9544
Error: Couldn't extract WordPress archive. unable to decompress gzipped phar archive "/tmp/wp_59c231a7990f7.tar.gz" to temporary file

I've checked that I have not run out of free disk space. I have also checked that the temporary file is a valid gzipped tarball:

$ cat /tmp/wp_59c231a7990f7.tar.gz | file -
/dev/stdin: gzip compressed data, last modified: Tue Sep 19 21:21:21 2017, from Unix
$ zcat /tmp/wp_59c231a7990f7.tar.gz | file -
/dev/stdin: POSIX tar archive (GNU)
$  tar tf /tmp/wp_59c231a7990f7.tar.gz | head
wordpress/
wordpress/wp-settings.php
wordpress/wp-cron.php
wordpress/wp-comments-post.php
wordpress/wp-activate.php
wordpress/wp-admin/
wordpress/wp-admin/link-parse-opml.php
wordpress/wp-admin/js/
wordpress/wp-admin/js/editor.js
wordpress/wp-admin/js/user-profile.min.js

Here is some more debugging information:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial
$ php -v
PHP 7.0.22-0ubuntu0.16.04.1 (cli) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.22-0ubuntu0.16.04.1, Copyright (c) 1999-2017, by Zend Technologies
$ wp --version
WP-CLI 1.3.0
@OnnoGeorg

This comment has been minimized.

Copy link

OnnoGeorg commented Sep 20, 2017

Same problem here on Linux Mint and PHP 5.6.

$ lsb_release -a  
No LSB modules are available.  
Distributor ID:	LinuxMint  
Description:	Linux Mint 18.1 Serena  
Release:	18.1  
Codename:	serena
$ php -v
PHP 5.6.31-2+ubuntu16.04.1+deb.sury.org+1 (cli) 
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans
$ wp --version
WP-CLI 1.3.0
@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

I have same problem with latest nightly version.

$ wp core download --path=/var/www/html/ --version=latest --force
Downloading WordPress 4.8.2 (en_US)...
md5 hash verified: 2e8744a702a3d9527782d9135a4c9544
Error: Couldn't extract WordPress archive. unable to decompress gzipped phar archive "/tmp/wp_59c24f71218a2.tar.gz" to temporary file
@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

It is working on MacOS, but not working on Ubuntu.
Following is a result of behat.

$ vendor/bin/behat features/core-download.feature 

PHP binary:	/usr/bin/php
PHP version:	7.0.22-0ubuntu0.16.04.1
php.ini used:	/etc/php/7.0/cli/php.ini
WP-CLI root dir:	/vagrant/core-command/vendor/wp-cli/wp-cli
WP-CLI vendor dir:	/vagrant/core-command/vendor
WP_CLI phar path:	
WP-CLI packages dir:	
WP-CLI global config:	/tmp/wp-cli-package-test/config.yml
WP-CLI project config:	/vagrant/core-command/wp-cli.yml
WP-CLI version:	1.3.0


                                                                                                                                         
  [RuntimeException]                                                                                                                     
  Exception has been thrown in "beforeSuite" hook, defined in FeatureContext::prepare()                                                  
  $ wp core download --force --path='/tmp/wp-cli-test-core-download-cache'                                                               
  Creating directory '/tmp/wp-cli-test-core-download-cache/'.                                                                            
  Downloading WordPress 4.8.2 (en_US)...                                                                                                 
  md5 hash verified: 2e8744a702a3d9527782d9135a4c9544                                                                                    
  Error: Couldn't extract WordPress archive. unable to decompress gzipped phar archive "/tmp/wp_59c256f2e306e.tar.gz" to temporary file  
  cwd:                                                                                                                                   
  run time: 13.057976961136                                                                                                              
  exit status: 1                                                                                                                         
                                                                                                                                         

behat [--init] [-f|--format="..."] [--out="..."] [--lang="..."] [--[no-]ansi] [--[no-]time] [--[no-]paths] [--[no-]snippets] [--[no-]snippets-paths] [--[no-]multiline] [--[no-]expand] [--story-syntax] [-d|--definitions="..."] [--name="..."] [--tags="..."] [--cache="..."] [--strict] [--dry-run] [--stop-on-failure] [--rerun="..."] [--append-snippets] [--append-to="..."] [features]
@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

This will work

wp --allow-root core download --locale=sv_SE

The problem is with US files ...

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

Oh, yes...
It looks package problem.

$ ./bin/wp core download --path=/home/vagrant/wp --locale=sv_SE --force
Downloading WordPress 4.8.2 (sv_SE)...
md5 hash verified: bfc6e58c9018e4344620caae85323547
Cleaning up files...
File removed: wp-includes/js/tinymce/skins/lightgray/fonts/readme.md
File removed: wp-includes/js/tinymce/skins/lightgray/fonts/tinymce.json
File removed: wp-includes/js/tinymce/skins/lightgray/fonts/tinymce-small.json
File removed: wp-includes/js/tinymce/skins/lightgray/skin.ie7.min.css
File removed: wp-includes/js/tinymce/plugins/media/moxieplayer.swf
File removed: wp-includes/js/tinymce/plugins/wpembed/plugin.js
File removed: wp-includes/js/tinymce/plugins/wpembed/plugin.min.js
7 files cleaned up.
Success: WordPress downloaded.
@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 20, 2017

@miya0001 I think it's more likely to be a PHP / PharData memory problem or something, and due to a recent change. It fails for me with PHP 7.0.23 and 7.1.9 but succeeds with PHP 7.0.18, PHP 5.6.22, PHP 5.5.38 (all on Ubuntu 17.04) and PHP 7.0.19 (Windows).

What version of php did you run on MacOS?

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

@gitlost :)

$ php --version
PHP 7.1.8 (cli) (built: Aug  7 2017 15:02:45) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 20, 2017

Very suspicious!

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

I think the problem might be the arguments in this line

tar xz --strip-components=1 --directory=%s -f $tarball

When you use wp core download you get a temp archive in /tmp folder. Try to unpack it with these arguments like
tar xz --strip-components=1 --directory=wordpress -f your_archive.tar.gz

I can't make it work , but if I use a much simple line like this it's work.
tar -xvzf wp_59c228df6f742.tar.gz

I think the US package have som issue here ...

@Sidsector9

This comment has been minimized.

Copy link
Member

Sidsector9 commented Sep 20, 2017

I think the compression is faulty.
Steps I followed:

  1. Downloaded the en_US version
  2. $phar = new PharData( 'wordpress-4.8.2.tar.gz' ); echo $phar->getFileName();
    This failed
  3. Extracted the tarball manually and compressed it again.
  4. Tried step 2 and it worked perfectly.
@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

Can we made so WP core download use default locale instead of US?

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

Hmm it is working on MacOS.

$ wp core download --version=latest --path=wordpress --force
Downloading WordPress 4.8.2 (en_US)...
Using cached file '/Users/miyauchi/.wp-cli/cache/core/wordpress-4.8.2-en_US.tar.gz'...
Success: WordPress downloaded.

And I can extract tar.gz on linux.

I am not sure is it problem of the package or not.

Can we made so WP core download use default locale instead of US?

The default locale is en_US.

@andreapinti

This comment has been minimized.

Copy link

andreapinti commented Sep 20, 2017

Hi,
same problem here (ubuntu 14)
Tried with php 5.6, php 7.0 and php 7.1

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

I agree, but it should be some misbehavior in the argument vs the package. Like I told before I can use "tar" with a simple line. The arguments troubled it up ...

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

Can we change the default locale to our own lang?

@OnnoGeorg

This comment has been minimized.

Copy link

OnnoGeorg commented Sep 20, 2017

Yes, the problem is with the US package - at least the German one runs well with wp core download --locale=de_DE

The Swedish as well as @ZeroX66 wrote.

@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 20, 2017

If it's a memory problem or something then different tars could randomly work.

@DeaZy07 could you give the full PHP version numbers please?

(@ZeroX66 I tried with the exact same arguments tar xz --strip-components=1 --directory=%s -f $tarball and it worked (and changed the source to fall back to this on a PharData exception and it worked also).)

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

This is bad coz we use Easy Engine and EE use wp-cli to get the WordPress. So maybe we should talk with these who update WP in US.
What is the differens between a sv/de vs us, I wonder?

@andreapinti

This comment has been minimized.

Copy link

andreapinti commented Sep 20, 2017

@gitlost sure

PHP 7.1.9-1+ubuntu14.04.1+deb.sury.org+1 (cli) (built: Sep  2 2017 06:17:11) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.9-1+ubuntu14.04.1+deb.sury.org+1, Copyright (c) 1999-2017, by Zend Technologies
 PHP 7.0.21-1+ubuntu14.04.1+deb.sury.org+1 (cli) (built: Jul  6 2017 09:38:10) ( NTS )
 Copyright (c) 1997-2017 The PHP Group
 Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
     with Zend OPcache v7.0.21-1~ubuntu14.04.1+deb.sury.org+1, Copyright (c) 1999-2017, by Zend Technologies
 PHP 5.6.31-4+ubuntu14.04.1+deb.sury.org+4 (cli) 
 Copyright (c) 1997-2016 The PHP Group
 Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Anyway i changed the locale to it_IT and now it works.

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

If I use arg --version it works
wp --allow-root core download --version="4.8.1"

@BhargavBhandari90

This comment has been minimized.

Copy link

BhargavBhandari90 commented Sep 20, 2017

seems compressed file is not proper.

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

@ZeroX66
If you want to change default locale, you can set default locale to ~/.wp-cli/config.yml like following.

core download:
  locale: de_DE
@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

@miya0001
I try to add that file , but it seems to still get the us. Do I need to restart something?

@wojsmol

This comment has been minimized.

Copy link
Contributor

wojsmol commented Sep 20, 2017

on enviramant with error plese check if zlib php extesion is enabled
reference composer/composer#984

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 20, 2017

@ZeroX66
Oh! Sorry, the file name of the config is ~/.wp-cli/config.yml.
https://make.wordpress.org/cli/handbook/config/

(I updated above. 😄 )

@JPry

This comment has been minimized.

Copy link
Contributor

JPry commented Sep 20, 2017

@wojsmol

on enviramant with error plese check if zlib php extesion is enabled

That's a good thought, but it appears that zlib is enabled:

php -i | grep zlib
Registered PHP Streams => https, ftps, compress.zlib, php, file, glob, data, http, ftp, phar, ssh2.shell, ssh2.exec, ssh2.tunnel, ssh2.scp, ssh2.sftp, zip
Registered Stream Filters => zlib.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk, convert.iconv.*, mcrypt.*, mdecrypt.*
zlib
Stream Wrapper => compress.zlib://
Stream Filter => zlib.inflate, zlib.deflate
zlib.output_compression => Off => Off
zlib.output_compression_level => -1 => -1
zlib.output_handler => no value => no value

@ZeroX66

I think the problem might be the arguments in this line

tar xz --strip-components=1 --directory=%s -f $tarball

I don't think that line is the problem. That line is only executed if the PharData class does not exist. The class does exist. The exception happens on $phar = new PharData( $tarball ); as was mentioned by @miya0001

So it seems that there's some issue between the package and the PharData class. I'm not sure how to determine which of them is to blame.

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 20, 2017

@miya0001
Great, now it works ...

@danielbachhuber

This comment has been minimized.

Copy link
Member

danielbachhuber commented Sep 21, 2017

Is there any ETA for a new stable release or should I use the nightly version in my Travis builds for now

Not yet scheduled. You should use the nightly build.

@miya0001

This comment has been minimized.

Copy link
Member

miya0001 commented Sep 21, 2017

@gitlost

It worked fine on my custom docker environment which is using phpenv + php-build too.
I saw the php-build repository and I can't find the patch what you said.
https://github.com/php-build/php-build/tree/master/share/php-build/patches

I guess Travis CI is using same solution. :)

vintagesucks added a commit to vintagesucks/caddy-script that referenced this issue Sep 21, 2017

use wp-cli nightly
fixes wordpress build

see wp-cli/wp-cli#4370 (comment)
@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 21, 2017

Ah maybe. How Travis builds its stuff has always been (and likely always will be) a mystery to me.

The patch above is from the Ubuntu php7.0 package, eg for 17.04 https://launchpad.net/ubuntu/+archive/primary/+files/php7.0_7.0.22-0ubuntu0.17.04.1.debian.tar.xz, for 16.04 https://launchpad.net/ubuntu/+archive/primary/+files/php7.0_7.0.22-0ubuntu0.16.04.1.debian.tar.xz, but is present in older distributions also.

@jmazzi

This comment has been minimized.

Copy link

jmazzi commented Sep 21, 2017

The problem seems to be specific to gzip, not tar.

ubuntu@controller-us-east-1:/tmp/f$ php -r '$f=new PharData( "wp_59c3ce5c83746.tar.gz" );$f->extractTo("/tmp/wordpress");'
PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'unable to decompress gzipped phar archive "/tmp/f/wp_59c3ce5c83746.tar.gz" to temporary file' in Command line code:1
Stack trace:
#0 Command line code(1): PharData->__construct('wp_59c3ce5c8374...')
#1 {main}
  thrown in Command line code on line 1
ubuntu@controller-us-east-1:/tmp/f$ gunzip wp_59c3ce5c83746.tar.gz 
ubuntu@controller-us-east-1:/tmp/f$ php -r '$f=new PharData( "wp_59c3ce5c83746.tar" );$f->extractTo("/tmp/wordpress");'
ubuntu@controller-us-east-1:/tmp/f$ ls -al /tmp/wordpress/
total 80
drwxrwxr-x    3 ubuntu ubuntu  4096 Sep 21 14:55 .
drwxrwxrwt 2593 root   root   69632 Sep 21 14:55 ..
drwxrwxr-x    5 ubuntu ubuntu  4096 Sep 21 14:55 wordpress
ubuntu@controller-us-east-1:/tmp/f$ 

vintagesucks added a commit to vintagesucks/caddy-script that referenced this issue Sep 22, 2017

@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 22, 2017

I tracked down the guilty party to here https://github.com/php/php-src/blob/master/ext/zlib/zlib_filter.c#L124 in the zlib_filter where it returns without updating the bytes_consumed variable, as if a .tar.gz is 1 to about 10 bytes over modulo 8192 then no output might be needed (the last 4 bytes are just a checksum anyway) but the bytes have been consumed, and not reporting them as such causes the calling function to error.

As mentioned above this is only triggered for files > 4MB on distributions with the Debian patch but even so it's surprising it hasn't been noticed before.

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Sep 25, 2017

Just to say that I have also hit this issue on Debian Stretch:

wp --allow-root core download
Downloading WordPress 4.8.2 (en_US)...
md5 hash verified: 2e8744a702a3d9527782d9135a4c9544
Error: Couldn't extract WordPress archive. unable to decompress gzipped phar archive "/tmp/wp_59c96e95f100a.tar.gz" to temporary file

wp --allow-root core download --locale=en_GB
Downloading WordPress 4.8.2 (en_GB)...
md5 hash verified: 28571832a9c0c95cbc8767a896c987d4
Success: WordPress downloaded.
@gitlost

This comment has been minimized.

Copy link
Contributor

gitlost commented Sep 25, 2017

@chriscroome Could you post the output of php --version please?

@ZeroX66

This comment has been minimized.

Copy link

ZeroX66 commented Sep 26, 2017

Hm, It's still some trouble with it. I change to this and it works with "nightly".

wp cli update --nightly

@JPry

This comment has been minimized.

Copy link
Contributor

JPry commented Sep 26, 2017

@ZeroX66

Hm, It's still some trouble with it. I change to this and it works with "nightly".

Nightly is the only way to have the fix. There hasn't been a release yet.

@matthewsuan

This comment has been minimized.

Copy link

matthewsuan commented Sep 26, 2017

i had this problem as well. the --version=4.8.1 fix worked for me.

@danielbachhuber

This comment has been minimized.

Copy link
Member

danielbachhuber commented Sep 26, 2017

Thanks everyone who spent time tracking this down. Given this isn't an issue with WP-CLI itself but with specific PHP distros, I think we've fixed it the best we can.

We'll publish WP-CLI v1.4.0 on Tuesday, October 17th. For now, those affected can use wp cli update --nightly.

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Sep 26, 2017

@gitlost sorry not to have replied sooner. the server is running PHP7.0 from Debian Stretch:

php --version
PHP 7.0.19-1 (cli) (built: May 11 2017 14:04:47) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.19-1, Copyright (c) 1999-2017, by Zend Technologies
@surrealchemist

This comment has been minimized.

Copy link

surrealchemist commented Sep 26, 2017

We saw this on our Ubuntu 16/ php70 server last week, and today a developer got it while trying to provision a VCCW vagrant environment.
Had to manually do 4.8.1 to get around the issue.

Also the message "unable to decompress" was misleading at first. It would be more useful if it said why exactly it is having a problem. At first we thought it was permissions and had to go through checking /tmp etc. and disk space till we realized the file was in /tmp and we could untar it fine by hand.

@Sidsector9

This comment has been minimized.

Copy link
Member

Sidsector9 commented Sep 27, 2017

For EasyEngine users, there's a blog with 2 workarounds -
https://easyengine.io/blog/solution-couldnt-extract-wordpress-archive-error/

Thanks to @danielbachhuber, @miya0001 and @gitlost

kadimi added a commit to kadimi/wp.box that referenced this issue Sep 27, 2017

kadimi added a commit to kadimi/wp.box that referenced this issue Sep 27, 2017

@pkarjala

This comment has been minimized.

Copy link
Contributor

pkarjala commented Sep 29, 2017

Can confirm we are also seeing this issue on Ubuntu 16.04.3 with PHP 7.0.22.

Can also confirm that using wp cli update --nightly fixed the issue using the .tar.gz fallback.

@davidfavor

This comment has been minimized.

Copy link

davidfavor commented Oct 3, 2017

Problem: wp-cli attempts unpack files using tar -xJf which works for xz files + not for gzip files.

Solution: Any .tar.gz file should use tar -xzf instead + likely all will be well.

In fact, likely best to write wp-cli to use a switch statement, so any known extensions will be correctly unpacked + if switch falls through, an exception is raised.

This will handle any future cases where WordPress might change their default archive format.

@danielbachhuber

This comment has been minimized.

Copy link
Member

danielbachhuber commented Oct 3, 2017

Please open a new issue for any topics going forward, thanks.

@wp-cli wp-cli locked and limited conversation to collaborators Oct 3, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.