Custom AMI creation or cluster creation/scalup failing on Ubuntu when doing apt update

[lukeseawalker]

Bug description

• Custom AMI creation through the pcluster createami command is failing when the provided base AMI has FSx repository already configured, e.g. when using ParallelCluster AMI as base AMI

• Cluster creation/scale-up is failing when using a pre/post installation script executing a apt update or apt-get update.

In both cases error is:

Get:5 https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease [3275 B]
E: Repository 'https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease' changed its 'Origin' value from '' to 'AWS-FSx-for-Lustre'
E: Repository 'https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease' changed its 'Suite' value from '' to 'stable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

How to reproduce

• Custom AMI creation

pcluster createami -ai ami-07de67a2c91b2605c -os ubuntu1804 

• Cluster creation/scale-up with pre/post installation script containing

apt update

Affected Versions

Version 2.6.0+ and Ubuntu

Root Cause

Upstream FSx repository has different properties than the one known by apt, which are stored in the /var/lib/apt/lists/fsx-lustre-client-repo.s3.amazonaws.com* files.
The value of the property 'Origin' changed from ' ' to 'AWS-FSx-for-Lustre' and the value of the property 'Suite' changed from ' ' to 'stable'.

Workaround

For the custom AMI creation, avoid to use the pcluster createami command and prefer modifying a ParallelCluster AMI, following the doc here

For the cluster creation/scale-up on Ubuntu 18.04, change your pre/post installation to call apt update with additional flag

apt update --allow-releaseinfo-change

On Ubuntu 16.04 that flag is not recognized. The following can be used instead:

rm -rf /var/lib/apt/lists/* && apt update -y

[shijin-aws]

@lukeseawalker I hit such issue as well when running sudo apt-get update remotely (via ssh) on the master node of parallel cluster 2.10.1 with base_os=ubuntu1804. I tried your solution in aws/aws-parallelcluster-cookbook#858. It did work when I login to master node interactively and execute sudo apt update -y. But when I run this command remotely like ssh ubuntu@xx.xx.xxx -i key "sudo apt update -y" I still hit the error:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Get:2 https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease [3275 B]
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists...
E: Repository 'https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease' changed its 'Origin' value from '' to 'AWS-FSx-for-Lustre'
E: Repository 'https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease' changed its 'Suite' value from '' to 'stable'

(pcluster2.10.1) (base) sjina Documents$ echo $?
100

Do you have any suggestion for this issue?

[lukeseawalker]

Hi @shijin-aws,
use sudo apt update --allow-releaseinfo-change

$ ssh ubuntu@x.x.x.x "sudo apt update --allow-releaseinfo-change"                                                                                                              

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:4 https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic InRelease [3275 B]
Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
Get:6 https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu bionic/main amd64 Packages [8389 B]
Fetched 11.7 kB in 0s (30.1 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
73 packages can be upgraded. Run 'apt list --upgradable' to see them.

$ echo $?                                                                                                                                                                                            
0

[shijin-aws]

@lukeseawalker Is this workaround not needed after newer pcluster release?

[tilne]

@shijin-aws Generally speaking we don't anticipate the same issue to occur for the FSx package, but ultimately we don't have control over that for any packages.