Cannot write to S3 device from recent Ubuntu; perhaps because of newer libcurl4

[gerardkok]

Summary

Writing to an S3 device from a recent Ubuntu (tried 19.10 and 20.04) does not work, amanda (version 3.5.1, tried amdump, amlabel and 'amcheck -w') just hangs and never returns. Ubuntu 18.04 (also version 3.5.1) works fine. Downgrading libcurl4 on 19.10 and 20.04 to the version of 18.04 seems to fix the problem (although I haven't tested this in production yet, also because I used a bit of force to downgrade, which I don't want to do in production).

How to reproduce

Install 'amanda-server' on an Ubuntu-20.04 instance (I used a vagrant box based on https://github.com/chef/bento/blob/master/packer_templates/ubuntu/ubuntu-20.04-amd64.json). Configure amanda to use an S3 changer device, and use new virtual tapes. Try to label a tape:

backup@vagrant:~$ amlabel test test-01 slot 1
Reading label...
Found an empty tape.
Writing label 'test-01'...
^C

This does not return (and the virtual tape does not get created). The last lines from amlabel.<timestamp>.debug are:

Wed Jun 03 11:24:48.012207191 2020: pid 14472: thd-0x562400995600: amlabel: Hdr Out: Content-MD5: lYjkchoZihkmrPD9hdJocg==
Wed Jun 03 11:24:48.012210456 2020: pid 14472: thd-0x562400995600: amlabel: Hdr Out: Content-Length: 40
Wed Jun 03 11:24:48.012213590 2020: pid 14472: thd-0x562400995600: amlabel: Hdr Out: Date: Wed, 03 Jun 2020 11:24:50 GMT
Wed Jun 03 11:24:48.012216701 2020: pid 14472: thd-0x562400995600: amlabel: Hdr Out: 

Workaround
Downgrade libcurl4. I did that as follows, but no doubt there are better ways:

• globally replace 'focal' by 'bionic' in /etc/apt/sources.list
• run apt-get update
• run apt-get install libcurl4=7.58.0-2ubuntu3.8:

# apt-get install libcurl4=7.58.0-2ubuntu3.8
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  curl pollinate ubuntu-server
The following packages will be DOWNGRADED:
  libcurl4
0 upgraded, 0 newly installed, 1 downgraded, 3 to remove and 1 not upgraded.
...

Now 'amlabel' works just fine:

backup@vagrant:~$ amlabel test test-01 slot 1
Reading label...
Found an empty tape.
Writing label 'test-01'...
Checking label...
Success!

The relevant lines from amlabel.<timestamp>.debug now read:

Wed Jun 03 12:28:14.777182823 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr Out: Content-MD5: lYjkchoZihkmrPD9hdJocg==
Wed Jun 03 12:28:14.777186131 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr Out: Content-Length: 40
Wed Jun 03 12:28:14.777189284 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr Out: Date: Wed, 03 Jun 2020 12:28:15 GMT
Wed Jun 03 12:28:14.777192351 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr Out: Expect: 100-continue
Wed Jun 03 12:28:14.777195436 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr Out: 
Wed Jun 03 12:28:14.817005276 2020: pid 15279: thd-0x55e019744600: amlabel: Hdr In: HTTP/1.1 100 Continue

[jfarjona]

Same problem here with Debian 11., running libcurl4 7.74.0-1.3+deb11u1

It seems not too many people are paying attention since 18 months. Time to drop amanda.

[shrodi]

This one was a life saver...! I just upgraded from Ubuntu 18.04 LTS to 20.04 LTS and could not figure out why everything looked fine but the writing to S3...! Downgrading libcurl4 did the trick for me also. Some more hints to add to @gerardkok 's solution for posterity and others:

1. Better to go with the Bionic updates instead of base repository, so for example in my case I simply added the following lines to /etc/apt/sources.list:
   deb http://ca.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
   (better use the same base URL you have in your own sources.list, though)

2. After apt-update, you can check available package versions with apt-cache policy libcurl4: you want the the 7.58 one; in my case, the version I picked was 7.58.0-2ubuntu3.16.

3. After install, you can comment out the bionic repositories you added in sources.list and run apt-mark hold libcurl4 to prevent any upgrade of the package until you choose to unhold it.

[chassell]

@jfarjona it looks like our new S3 is working fine. I'd go test all your S3 stuff now... with GCP too.... against 3_6.beta.

@shrodi I think we're fine with the latest curl, no question, with 3_6.beta.