New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drush rsync incorrectly places destination path #3306

Closed
iainhouston opened this Issue Jan 13, 2018 · 9 comments

Comments

4 participants
@iainhouston

iainhouston commented Jan 13, 2018

Note below that the source and target files directories are not sync'd (or overwritten per drush's message) but a new subdirectory sites/default/files/files is created as if the rsync embedded did not use final / for both source and target.

vagrant@vagrant:/var/www/drupal/web$ ll sites/default/files/files 
ls: cannot access 'sites/default/files/files': No such file or directory
.
vagrant@vagrant:/var/www/drupal/web$ drush rsync  @balive:%files @self:%files
You will delete files in /var/www/drupal/web/sites/default/files and replace with data from webmaster@bradford-abbas.uk:/var/www/drupal/web/sites/default/files

 Do you want to continue? (yes/no) [yes]:
 > 

vagrant@vagrant:/var/www/drupal/web$ ll sites/default/files/files
total 12
drwxrwsr-x 23 503 www-data  736 Jan 11 13:14 ./
drwxrwsr-x 24 503 www-data  768 Jan 13 07:58 ../
drwxrwxr-x 25 503 www-data  800 Mar  9  2016 2016-03/
drwxrwxr-x  4 503 www-data  128 Apr 28  2016 2016-04/
drwxrwxr-x  2 503 www-data   64 Jun  6  2016 2016-06/
drwxrwxr-x 15 503 www-data  480 Sep 17  2016 2016-09/
drwxrwxr-x  4 503 www-data  128 Oct 11  2016 2016-10/
drwxrwxr-x  2 503 www-data   64 Jan  3  2017 2017-01/
drwxrwxr-x  3 503 www-data   96 Feb 27  2017 2017-02/
drwxrwxr-x  4 503 www-data  128 Mar  5  2017 2017-03/
drwxrwxr-x  2 503 www-data   64 Apr  3  2017 2017-04/
drwxrwxr-x  3 503 www-data   96 Jun  8  2017 2017-06/
drwxrwxr-x  3 503 www-data   96 Aug 20 09:18 2017-08/
drwxrwxr-x 13 503 www-data  416 Nov 20 13:31 2017-11/
drwxrwxr-x 32 503 www-data 1024 Jan 10 22:59 css/
drwxrwxr-x  3 503 www-data   96 Mar  3  2016 default_images/
-rw-rw-r--  1 503 www-data 6148 Mar  4  2016 .DS_Store
-rw-rw-r--  1 503 www-data  487 Feb 22  2016 .htaccess
drwxrwxr-x 21 503 www-data  672 Jun  8  2017 inline-images/
drwxrwxr-x 16 503 www-data  512 Jan 10 17:41 js/
drwxrwxrwx  3 503 www-data   96 Jan 12 21:45 php/
drwxrwxr-x  6 503 www-data  192 Jun  6  2016 pictures/
drwxrwxr-x  5 503 www-data  160 May  8  2016 styles/
@weitzman

This comment has been minimized.

Member

weitzman commented Jan 13, 2018

@iainhouston

This comment has been minimized.

iainhouston commented Jan 13, 2018

No site/default/files/files on remote

vagrant@vagrant:/var/www/drupal/web$ ssh webmaster@bradford-abbas.uk -o PasswordAuthentication=no -i ~/.ssh/BAPC-2.pem 'ls -la /var/www/drupal/web/sites/default/files'
total 96
drwxrwsr-x 21 www-data  www-data  4096 Jan 11 13:14 .
dr-xr-xr-x  3 webmaster webmaster 4096 Jan 10 17:21 ..
drwxrwxr-x  2 webmaster webmaster 4096 Mar  9  2016 2016-03
drwxrwxr-x  2 webmaster webmaster 4096 Apr 28  2016 2016-04
drwxrwxr-x  2 webmaster webmaster 4096 Jun  6  2016 2016-06
drwxrwxr-x  2 webmaster webmaster 4096 Sep 17  2016 2016-09
drwxrwxr-x  2 webmaster webmaster 4096 Oct 11  2016 2016-10
drwxrwxr-x  2 webmaster webmaster 4096 Jan  3  2017 2017-01
drwxrwxr-x  2 webmaster webmaster 4096 Feb 27  2017 2017-02
drwxrwxr-x  2 webmaster webmaster 4096 Mar  5  2017 2017-03
drwxrwxr-x  2 webmaster webmaster 4096 Apr  3  2017 2017-04
drwxrwxr-x  2 webmaster webmaster 4096 Jun  8  2017 2017-06
drwxrwxr-x  2 webmaster webmaster 4096 Aug 20 09:18 2017-08
drwxrwxr-x  2 webmaster webmaster 4096 Nov 20 13:31 2017-11
drwxrwxr-x  2 webmaster webmaster 4096 Jan 10 22:59 css
drwxrwxr-x  2 webmaster webmaster 4096 Mar  3  2016 default_images
-rw-rw-r--  1 webmaster webmaster 6148 Mar  4  2016 .DS_Store
-rw-rw-r--  1 webmaster webmaster  487 Feb 22  2016 .htaccess
drwxrwxr-x  2 webmaster webmaster 4096 Jun  8  2017 inline-images
drwxrwxr-x  2 webmaster webmaster 4096 Jan 10 17:41 js
drwxrwxrwx  3 webmaster webmaster 4096 Jan 13 18:07 php
drwxrwxr-x  6 webmaster webmaster 4096 Jun  6  2016 pictures
drwxrwxr-x  5 webmaster webmaster 4096 May  8  2016 styles

No site/default/files/files on local

vagrant@vagrant:/var/www/drupal/web$ ll sites/default/files/files
ls: cannot access 'sites/default/files/files': No such file or directory

Local status as requested, public files are at sites/default/files as expected

vagrant@vagrant:/var/www/drupal/web$ drush status
 Drupal version   : 8.4.4                                        
 Site URI         : default                                      
 DB driver        : mysql                                        
 DB hostname      : localhost                                    
 DB port          :                                              
 DB username      : drupal                                       
 DB name          : drupal                                       
 Database         : Connected                                    
 Drupal bootstrap : Successful                                   
 Default theme    : bartik                                       
 Admin theme      : seven                                        
 PHP binary       : /usr/bin/php7.1                              
 PHP config       : /etc/php/7.1/cli/php.ini                     
 PHP OS           : Linux                                        
 Drush script     : /usr/local/bin/drush                         
 Drush version    : 9.0.0-rc2                                    
 Drush temp       : /tmp                                         
 Drush configs    : /var/www/drupal/vendor/drush/drush/drush.yml 
 Install profile  : standard                                     
 Drupal root      : /var/www/drupal/web                          
 Site path        : sites/default                                
 Files, Public    : sites/default/files                          
 Files, Temp      : /tmp           

Aliases:

vagrant@vagrant:/var/www/drupal/web$ drush site:alias
'@badev.badev':
  host: vagrant.bradford-abbas.uk
  ssh:
    options: '-o PasswordAuthentication=no -i ~/.vagrant.d/insecure_private_key'
  root: /var/www/drupal/web
  uri: vagrant.bradford-abbas.uk
  user: vagrant
'@balive.balive':
  host: bradford-abbas.uk
  ssh:
    options: '-o PasswordAuthentication=no -i ~/.ssh/BAPC-2.pem'
  root: /var/www/drupal/web
  uri: bradford-abbas.uk
  user: webmaster

Do the deed and note the comment:

vagrant@vagrant:/var/www/drupal/web$ drush rsync  @balive:%files @self:%files
You will delete files in /var/www/drupal/web/sites/default/files and replace with data from webmaster@bradford-abbas.uk:/var/www/drupal/web/sites/default/files

 Do you want to continue? (yes/no) [yes]:
 > 

sites/default/files/files is now on local machine:

vagrant@vagrant:/var/www/drupal/web$ ll sites/default/files/files
total 12
drwxrwsr-x 23 503 www-data  736 Jan 11 13:14 ./
drwxrwsr-x 24 503 www-data  768 Jan 13 19:05 ../
drwxrwxr-x 25 503 www-data  800 Mar  9  2016 2016-03/
drwxrwxr-x  4 503 www-data  128 Apr 28  2016 2016-04/
drwxrwxr-x  2 503 www-data   64 Jun  6  2016 2016-06/
drwxrwxr-x 15 503 www-data  480 Sep 17  2016 2016-09/
drwxrwxr-x  4 503 www-data  128 Oct 11  2016 2016-10/
drwxrwxr-x  2 503 www-data   64 Jan  3  2017 2017-01/
drwxrwxr-x  3 503 www-data   96 Feb 27  2017 2017-02/
drwxrwxr-x  4 503 www-data  128 Mar  5  2017 2017-03/
drwxrwxr-x  2 503 www-data   64 Apr  3  2017 2017-04/
drwxrwxr-x  3 503 www-data   96 Jun  8  2017 2017-06/
drwxrwxr-x  3 503 www-data   96 Aug 20 09:18 2017-08/
drwxrwxr-x 13 503 www-data  416 Nov 20 13:31 2017-11/
drwxrwxr-x 32 503 www-data 1024 Jan 10 22:59 css/
drwxrwxr-x  3 503 www-data   96 Mar  3  2016 default_images/
-rw-rw-r--  1 503 www-data 6148 Mar  4  2016 .DS_Store
-rw-rw-r--  1 503 www-data  487 Feb 22  2016 .htaccess
drwxrwxr-x 21 503 www-data  672 Jun  8  2017 inline-images/
drwxrwxr-x 16 503 www-data  512 Jan 10 17:41 js/
drwxrwxrwx  3 503 www-data   96 Jan 13 18:07 php/
drwxrwxr-x  6 503 www-data  192 Jun  6  2016 pictures/
drwxrwxr-x  5 503 www-data  160 May  8  2016 styles/
vagrant@vagrant:/var/www/drupal/web$ 
vagrant@vagrant:/var/www/drupal/web$ tree -dL 2 sites/default/files/

sites/default/files/
|-- 2016-03
|-- 2016-04
|-- 2016-06
|-- 2016-09
|-- 2016-10
|-- 2017-01
|-- 2017-02
|-- 2017-03
|-- 2017-04
|-- 2017-06
|-- 2017-08
|-- 2017-11
|-- css
|-- default_images
|-- files
|   |-- 2016-03
|   |-- 2016-04
|   |-- 2016-06
|   |-- 2016-09
|   |-- 2016-10
|   |-- 2017-01
|   |-- 2017-02
|   |-- 2017-03
|   |-- 2017-04
|   |-- 2017-06
|   |-- 2017-08
|   |-- 2017-11
|   |-- css
|   |-- default_images
|   |-- inline-images
|   |-- js
|   |-- php
|   |-- pictures
|   `-- styles
|-- inline-images
|-- js
|-- php
|   `-- twig
|-- pictures
|   |-- 2016-03
|   |-- 2016-04
|   |-- 2016-05
|   `-- 2016-06
`-- styles
    |-- juicebox_square_thumb
    |-- medium
    `-- thumbnail

47 directories

@iainhouston

This comment has been minimized.

iainhouston commented Jan 15, 2018

FYI Using Drush launcher 0.5.0; Drush 9.0.0-rc2 via composer.

@iainhouston

This comment has been minimized.

iainhouston commented Jan 17, 2018

Yes, the rsync is missing both the terminal "/"s so its not replacing files but adding the complete source directory to the target.

So it should be

rsync ...                                                                           \    
webmaster@bradford-abbas.uk:/var/www/drupal/web/sites/default/files/                \       
/var/www/drupal/web/sites/default/files/

rather than

vagrant@vagrant:/var/www/drupal/web$ drush rsync  @balive:%files @self:%files --verbose
 [info] ssh -o PasswordAuthentication=no -i ~/.ssh/BAPC-2.pem webmaster@bradford-abbas.uk 'drush  --backend=2 --root=/var/www/drupal/web --uri=bradford-abbas.uk  core-status   --project=files 2>&1' 2>&1
 [info] /usr/local/bin/drush  --backend=2 --root=/var/www/drupal/web --uri=default  core-status   --project=files 2>&1
You will delete files in /var/www/drupal/web/sites/default/files and replace with data from webmaster@bradford-abbas.uk:/var/www/drupal/web/sites/default/files

 Do you want to continue? (yes/no) [yes]:
 > 

Calling system(rsync -e 'ssh -o PasswordAuthentication=no -i ~/.ssh/BAPC-2.pem' -akzv  --stats --progress webmaster@bradford-abbas.uk:/var/www/drupal/web/sites/default/files /var/www/drupal/web/sites/default/files);
receiving incremental file list
files/
...
@greg-1-anderson

This comment has been minimized.

Member

greg-1-anderson commented Jan 17, 2018

The path component that we started doing likes to normalize paths, removing the trailing slash when present. There is already one workaround in the rsync code for this, but it apparently does not cover @balive:%files, which historically has been interpreted as @balive:%files/ for exactly the reason described in this issue.

@iainhouston

This comment has been minimized.

iainhouston commented Jan 17, 2018

If I recall I did try $ drush rsync @balive:%files/ @self:%files/ explicitly without success.

@weitzman weitzman added this to In Progress in Nine Jan 21, 2018

@rickkowal

This comment has been minimized.

rickkowal commented Feb 19, 2018

I'm having the same issue, is a fix or workaround available?

@greg-1-anderson

This comment has been minimized.

Member

greg-1-anderson commented Feb 20, 2018

Looking into this right now. This is what I'm seeing:

$ ./sut -s rsync %files /tmp
Calling system(rsync -e 'ssh ' -akz /PATH/sites/dev/files /tmp);
$ ./sut -s rsync %files/ /tmp
Calling system(rsync -e 'ssh ' -akz /PATH/sites/dev/files/ /tmp);

Second example looks correct to me; first example should produce the same result as the second example.

greg-1-anderson added a commit that referenced this issue Feb 20, 2018

Fixes #3306: Make sure that rsync with %files resolves to sites/defau…
…lt/files/ as it historically has, not sites/default/files.
@greg-1-anderson

This comment has been minimized.

Member

greg-1-anderson commented Feb 20, 2018

See #3405

@weitzman weitzman moved this from In Progress to Done in Nine Mar 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment