wal-e backup-push does't work with Python 3.6

[yteraoka]

-bash-4.2$ /opt/wal-e/bin/wal-e --aws-instance-profile --s3-prefix s3://****/wal-e backup-push $PGDATA
wal_e.main   INFO     MSG: starting WAL-E
        DETAIL: The subcommand is "backup-push".
        STRUCTURED: time=2017-03-07T15:00:32.823141-00 pid=14150
wal_e.operator.backup INFO     MSG: start upload postgres version metadata
        DETAIL: Uploading to s3://****/wal-e/basebackups_005/base_0000000100000002000000FE_00000040/extended_version.txt.
        STRUCTURED: time=2017-03-07T15:00:35.386069-00 pid=14150
wal_e.operator.backup INFO     MSG: postgres version metadata upload complete
        STRUCTURED: time=2017-03-07T15:00:35.458788-00 pid=14150
wal_e.worker.upload INFO     MSG: beginning volume compression
        DETAIL: Building volume 0.
        STRUCTURED: time=2017-03-07T15:00:35.585929-00 pid=14150
Traceback (most recent call last):
  File "/opt/wal-e/lib64/python3.6/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/worker/upload.py", line 97, in __call__
    tpart.tarfile_write(pl.stdin)
  File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/tar_partition.py", line 315, in tarfile_write
    self._padded_tar_add(tar, et_info)
  File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/tar_partition.py", line 242, in _padded_tar_add
    tar.addfile(et_info.tarinfo, f)
  File "/usr/lib64/python3.6/tarfile.py", line 1973, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
TypeError: copyfileobj() got an unexpected keyword argument 'bufsize'
Wed Mar  8 00:00:35 2017 <Greenlet at 0x7f6121e8e5a0: <wal_e.worker.upload.PartitionUploader object at 0x7f6121e9f940>([ExtendedTarInfo(submitted_path='/var/lib/pgsql/9.)> failed with TypeError

wal_e.operator.backup WARNING  MSG: blocking on sending WAL segments
        DETAIL: The backup was not completed successfully, but we have to wait anyway.  See README: TODO about pg_cancel_backup
        STRUCTURED: time=2017-03-07T15:00:35.701163-00 pid=14150
NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
wal_e.main   CRITICAL MSG: An unprocessed exception has avoided all error handling
        DETAIL: Traceback (most recent call last):
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/cmd.py", line 627, in main
            pool_size=args.pool_size)
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/operator/backup.py", line 197, in database_backup
            **kwargs)
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/operator/backup.py", line 504, in _upload_pg_cluster_dir
            pool.join()
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/worker/upload_pool.py", line 120, in join
            self._wait()
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/worker/upload_pool.py", line 65, in _wait
            raise val
          File "/opt/wal-e/lib64/python3.6/site-packages/gevent/greenlet.py", line 536, in run
            result = self._run(*self.args, **self.kwargs)
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/worker/upload.py", line 97, in __call__
            tpart.tarfile_write(pl.stdin)
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/tar_partition.py", line 315, in tarfile_write
            self._padded_tar_add(tar, et_info)
          File "/opt/wal-e/lib64/python3.6/site-packages/wal_e/tar_partition.py", line 242, in _padded_tar_add
            tar.addfile(et_info.tarinfo, f)
          File "/usr/lib64/python3.6/tarfile.py", line 1973, in addfile
            copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
        TypeError: copyfileobj() got an unexpected keyword argument 'bufsize'

        STRUCTURED: time=2017-03-07T15:00:37.771104-00 pid=14150
-bash-4.2$

It seems to be due to change of tarfile.py.
https://github.com/python/cpython/blob/3.5/Lib/tarfile.py
https://github.com/python/cpython/blob/3.6/Lib/tarfile.py

bufsize is always None?

$ uname -a
Linux ip-172-31-28-236.ap-northeast-1.compute.internal 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Python 3.6 installed from IUC repository.

$ yum list installed | grep python36
python36u.x86_64                           3.6.0-2.ius.centos7         @ius
python36u-libs.x86_64                      3.6.0-2.ius.centos7         @ius
python36u-pip.noarch                       9.0.1-1.ius.centos7         @ius
python36u-setuptools.noarch                32.3.1-1.ius.centos7        @ius

[fdr]

Thanks for the bug report.

Looks like Python 3.6 added a good feature, and that broke my workaround: instead of having to do my weird monkeypatch thing in copyfileobj.py, we might be able set the bufsize on a tarfile object directly (copybufsize).

How would you feel about patching this so that WAL-E's copyfileobj is not used on Python >= 3.6, and instead a large buffer set for tarfile's copyfileobj?

[yteraoka]

Thanks.
That proposal sounds good to me.
I tried deleting the patch.
It worked.

[brikou]

Hello!
As written in readme, v1.1.0 is tested against python v3.6... sadly it still doesn't work with python 3.6, do you have any ETA for this fix?
Thanks a lot

[yteraoka]

@brikou
Hi,
I tried that a half year ago. I did just simple test, and I have completely forgot that.
Before I tried that, the system began with python 3.5 now.
I will try again, so Please give me some time.

[yteraoka]

@brikou
I tried again.

Following patch worked with Python 3.6 and 3.5. Could you please verify that this patch works?

--- /tmp/cmd.py.orig    2017-01-03 21:00:11.000000000 +0000
+++ /opt/wal-e/lib/python3.6/site-packages/wal_e/cmd.py 2017-09-08 10:31:53.567664129 +0000
@@ -589,7 +589,8 @@
         backup_cxt = configure_backup_cxt(args)

         if subcommand == 'backup-fetch':
-            monkeypatch_tarfile_copyfileobj()
+            if not (sys.version_info.major >= 3 and sys.version_info.minor >= 6):
+                monkeypatch_tarfile_copyfileobj()

             external_program_check([LZOP_BIN])
             backup_cxt.database_fetch(
@@ -601,7 +602,8 @@
         elif subcommand == 'backup-list':
             backup_cxt.backup_list(query=args.QUERY, detail=args.detail)
         elif subcommand == 'backup-push':
-            monkeypatch_tarfile_copyfileobj()
+            if not (sys.version_info.major >= 3 and sys.version_info.minor >= 6):
+                monkeypatch_tarfile_copyfileobj()

             if args.while_offline:
                 # we need to query pg_config first for the

[vmihailenco]

Ping. wal-e is still broken with Python 3.6.

[yteraoka]

hmm.. 😕
My tested environment below.

$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)

$ yum list installed | grep postgresql
postgresql96.x86_64                         9.6.5-1PGDG.rhel7          @pgdg96  
postgresql96-libs.x86_64                    9.6.5-1PGDG.rhel7          @pgdg96  
postgresql96-server.x86_64                  9.6.5-1PGDG.rhel7          @pgdg96

$ yum list installed | grep python36
python36u.x86_64                            3.6.2-1.ius.centos7        @ius     
python36u-libs.x86_64                       3.6.2-1.ius.centos7        @ius     
python36u-pip.noarch                        9.0.1-1.ius.centos7        @ius     
python36u-setuptools.noarch                 33.1.1-1.ius.centos7       @ius

Installing WAL-E (S3 storage only)

# python3.6 -m venv /opt/wal-e
# /opt/wal-e/bin/pip install wal-e[aws]==1.1.0b1
# /opt/wal-e/bin/wal-e version
1.1.0b1

then apply the patch that disable monkeypatch_tarfile_copyfileobj() to cmd.py.

Which version do you use?
What error message displayed?

[vmihailenco]

@yteraoka I only mean that master branch is still broken. Your patch is probably working, but it would be nice if we get it into master branch...

[h4k1m0u]

I got the same error as well when I updated to Python3.6 on FreeBSD11.
The version of Wal-e installed is 1.1.0.

[technion]

I can confirm that with 1.1.0 and your patch applied, the issue is resolved for me. It would be great to see this committed.

[loganb]

Hello all,

I'm getting this same failure using Python 3.6.4 and Wal-e 1.1.0b1 on OS X. I'm having trouble following the thread here. Is there a working workaround? Thanks!