Regression: --exclude '*.*' --rinclude '^one_file$' excludes all files

[NV]

I want to upload several files, but don’t touch anything else:

s3cmd put --exclude '*.*' --rinclude '^one/file$' --rinclude '^another/file$'

Unfortunately, it doesn’t upload anything in 1.5.0-alpha3. It used to work well in 1.5.0-alpha1.

[kormoc]

I'm having an issue with 1.5.0-beta1 where --exclude '*.*' --include '*.png' matches nothing. I wonder if it's related

[mdomsch]

yes. Excludes are processed before includes. Try using --files-from= instead?

On Sat, Dec 28, 2013 at 5:23 PM, Rob Smith notifications@github.com wrote:

I'm having an issue with 1.5.0-beta1 where --exclude '.' --include
'*.png' matches nothing. I wonder if it's related

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-31307203
.

[kormoc]

--files-from=<file with list of files to transfer> does work for my usage.

[mdomsch]

I reviewed the code path for this. --exclude="*" --include="something" is supposed to work. That it's not is indeed a bug that needs further debugging.

[ltmitch49]

I found that if I commented out "dirs.remove(x)" in "handle_exclude_include_walk" in "FileList.py" the --include option works, hopefully correctly.

[mdomsch]

Right. But then you haven't actually excluded any directories in your walk.

I need to think about this code path a little bit more. I'd like it to
work like rsync, in that if an exclude pattern ends in '/', it is treated
as a directory pattern, and excluded from the directory list during
os.walk() (which in commenting out that line as you have done, would defeat
this). We absolutely need to be able to prune directories from the
os.walk() via excludes.

On Thu, Jan 16, 2014 at 12:48 PM, ltmitch49 notifications@github.comwrote:

I found that if I commented out "dirs.remove(x)" in
"handle_exclude_include_walk" in "FileList.py" the --include option works,
hopefully correctly.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-32504740
.

[mdomsch]

Please try upstream master branch now. I reworked how excludes and includes are processed (and where). From my testing, doing a --exclude=* --include=*.jpg does indeed work now. To exclude a directory, be sure you have --exclude=somedir/ (trailing forward slash).

[ltmitch49]

Hello Matt,

I finally got to test the latest s3cmd 1.5.0-beta1 and unfortunately I got the following error on a backup that works fine on an older version (with my simplistic solution to the --exclude --include problem). Here is the error I am getting:

user1@PC6:~$ sudo ./s3bkuptest.sh

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /home/user1/Downloads/s3cmd/s3cmd --dry-run --config=/home/user1/.s3cfg --no-delete-removed --preserve --exclude=* --include-from=/home/user1/s3bkuptest.include sync /home/user1/testbk01 s3://testbk01.ltmProblem: NameError: global name 'src_exclude_list' is not defined
S3cmd: 1.5.0-beta1
python: 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3]
environment LANG=en_CA.UTF-8

Traceback (most recent call last):
File "/home/user1/Downloads/s3cmd/s3cmd", line 2406, in
main()
File "/home/user1/Downloads/s3cmd/s3cmd", line 2328, in main
cmd_func(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1404, in cmd_sync
return cmd_sync_local2remote(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1388, in cmd_sync_local2remote
destination_base_uri = _single_process(local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1199, in _single_process
_child(destination_base, local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1298, in _child
keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
NameError: global name 'src_exclude_list' is not defined

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

----- Original Message -----

From: "Matt Domsch" notifications@github.com
To: "s3tools/s3cmd" s3cmd@noreply.github.com
Cc: "ltmitch49" leonmitchell@shaw.ca
Sent: Saturday, March 15, 2014 7:05:44 PM
Subject: Re: [s3cmd] Regression: --exclude '.' --rinclude '^one_file$' excludes all files (#212)

Please try upstream master branch now. I reworked how excludes and includes are processed (and where). From my testing, doing a --exclude=* --include=*.jpg does indeed work now. To exclude a directory, be sure you have --exclude=somedir/ (trailing forward slash).
—
Reply to this email directly or view it on GitHub .

[mdomsch]

Thanks. I had not tried --dry-run. This should fix it:

diff --git a/s3cmd b/s3cmd
index e5f5fb7..9fa6b0a 100755
--- a/s3cmd
+++ b/s3cmd
@@ -1378,7 +1378,7 @@ def cmd_sync_local2remote(args):
error(u"or disable encryption with --no-encrypt parameter.")
sys.exit(1)

• local_list, single_file_local, exclude_list =
  fetch_local_list(args[:-1], is_src = True, recursive = True)
• local_list, single_file_local, src_exclude_list =
  fetch_local_list(args[:-1], is_src = True, recursive = True)

destinations = [args[-1]]
if cfg.additional_destinations:

On Sun, Mar 16, 2014 at 1:03 PM, ltmitch49 notifications@github.com wrote:

Hello Matt,

I finally got to test the latest s3cmd 1.5.0-beta1 and unfortunately I got
the following error on a backup that works fine on an older version (with
my simplistic solution to the --exclude --include problem). Here is the
error I am getting:

user1@PC6:~$ sudo ./s3bkuptest.sh

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /home/user1/Downloads/s3cmd/s3cmd --dry-run
--config=/home/user1/.s3cfg --no-delete-removed --preserve --exclude=*
--include-from=/home/user1/s3bkuptest.include sync /home/user1/testbk01
s3://testbk01.ltmProblem: NameError: global name 'src_exclude_list' is not
defined
S3cmd: 1.5.0-beta1
python: 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3]
environment LANG=en_CA.UTF-8

Traceback (most recent call last):
File "/home/user1/Downloads/s3cmd/s3cmd", line 2406, in
main()
File "/home/user1/Downloads/s3cmd/s3cmd", line 2328, in main
cmd_func(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1404, in cmd_sync
return cmd_sync_local2remote(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1388, in
cmd_sync_local2remote
destination_base_uri = _single_process(local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1199, in _single_process
_child(destination_base, local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1298, in _child
keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
NameError: global name 'src_exclude_list' is not defined

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

----- Original Message -----

From: "Matt Domsch" notifications@github.com
To: "s3tools/s3cmd" s3cmd@noreply.github.com
Cc: "ltmitch49" leonmitchell@shaw.ca
Sent: Saturday, March 15, 2014 7:05:44 PM
Subject: Re: [s3cmd] Regression: --exclude '.' --rinclude '^one_file$'
excludes all files (#212)

Please try upstream master branch now. I reworked how excludes and
includes are processed (and where). From my testing, doing a --exclude=*
--include=*.jpg does indeed work now. To exclude a directory, be sure you
have --exclude=somedir/ (trailing forward slash).
—
Reply to this email directly or view it on GitHub .

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-37764695
.

[mdomsch]

I tested this locally and pushed it to upstream master now.

Thanks,
Matt

On Sun, Mar 16, 2014 at 1:27 PM, Matt Domsch matt@domsch.com wrote:

Thanks. I had not tried --dry-run. This should fix it:

diff --git a/s3cmd b/s3cmd
index e5f5fb7..9fa6b0a 100755
--- a/s3cmd
+++ b/s3cmd
@@ -1378,7 +1378,7 @@ def cmd_sync_local2remote(args):
error(u"or disable encryption with --no-encrypt parameter.")
sys.exit(1)

• local_list, single_file_local, exclude_list =
  fetch_local_list(args[:-1], is_src = True, recursive = True)
• local_list, single_file_local, src_exclude_list =
  fetch_local_list(args[:-1], is_src = True, recursive = True)

destinations = [args[-1]]
if cfg.additional_destinations:

On Sun, Mar 16, 2014 at 1:03 PM, ltmitch49 notifications@github.comwrote:

Hello Matt,

I finally got to test the latest s3cmd 1.5.0-beta1 and unfortunately I
got the following error on a backup that works fine on an older version
(with my simplistic solution to the --exclude --include problem). Here is
the error I am getting:

user1@PC6:~$ sudo ./s3bkuptest.sh

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /home/user1/Downloads/s3cmd/s3cmd --dry-run
--config=/home/user1/.s3cfg --no-delete-removed --preserve --exclude=*
--include-from=/home/user1/s3bkuptest.include sync /home/user1/testbk01
s3://testbk01.ltmProblem: NameError: global name 'src_exclude_list' is not
defined
S3cmd: 1.5.0-beta1
python: 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3]
environment LANG=en_CA.UTF-8

Traceback (most recent call last):
File "/home/user1/Downloads/s3cmd/s3cmd", line 2406, in
main()
File "/home/user1/Downloads/s3cmd/s3cmd", line 2328, in main
cmd_func(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1404, in cmd_sync
return cmd_sync_local2remote(args)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1388, in
cmd_sync_local2remote
destination_base_uri = _single_process(local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1199, in _single_process
_child(destination_base, local_list)
File "/home/user1/Downloads/s3cmd/s3cmd", line 1298, in _child
keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
NameError: global name 'src_exclude_list' is not defined

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

----- Original Message -----

From: "Matt Domsch" notifications@github.com
To: "s3tools/s3cmd" s3cmd@noreply.github.com
Cc: "ltmitch49" leonmitchell@shaw.ca
Sent: Saturday, March 15, 2014 7:05:44 PM
Subject: Re: [s3cmd] Regression: --exclude '.' --rinclude '^one_file$'
excludes all files (#212)

Please try upstream master branch now. I reworked how excludes and
includes are processed (and where). From my testing, doing a --exclude=*
--include=*.jpg does indeed work now. To exclude a directory, be sure you
have --exclude=somedir/ (trailing forward slash).
—
Reply to this email directly or view it on GitHub .

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-37764695
.