Sometimes the EC2 instance fails to configure yum and the cache is corrupted somehow

[switch180]

The symptom is that an instance won't display the right version of python when python --version is run. This is because yum is unable to install packages. Sample logs:

[2020-09-14T23:50:50.840Z] Updating yum packages.
Loaded plugins: priorities, update-motd, upgrade-helper
http://packages.us-west-2.amazonaws.com/2018.03/main/c31535f74c6e/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
Trying other mirror.
...
14 Sep 23:51:00 ntpdate[2500]: adjust time server 47.144.196.17 offset -0.416039 sec
Starting ntpd: [  OK  ]
...
http://packages.ap-southeast-2.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
Trying other mirror.


 One of the configured repositories failed (amzn-updates-Base),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable amzn-updates

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=amzn-updates.skip_if_unavailable=true

failure: repodata/primary.sqlite.bz2 from amzn-updates: [Errno 256] No more mirrors to try.
http://packages.us-west-1.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
...
http://packages.ap-southeast-2.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
Warning: cannot update yum packages. Continue...
[2020-09-14T23:51:21.465Z] Completed updating yum packages.
[2020-09-14T23:51:21.468Z] Installing Python3.6
Loaded plugins: priorities, update-motd, upgrade-helper
Not using downloaded repomd.xml because it is older than what we have:
  Current   : Thu Sep  3 23:14:32 2020
  Downloaded: Thu Nov 14 22:09:23 2019
http://packages.us-east-1.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
Trying other mirror.
...
http://packages.ap-southeast-2.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno 16] error setting timestamp on file /var/cache/yum/x86_64/latest/amzn-updates/primary.sqlite.bz2 from http://packages.ap-southeast-2.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1, OSError: [Errno 2] No such file or directory: '/var/cache/yum/x86_64/latest/amzn-updates/primary.sqlite.bz2'
Trying other mirror.
...
http://packages.ap-northeast-1.amazonaws.com/2018.03/updates/fd3054b03123/x86_64/repodata/primary.sqlite.bz2?instance_id=i-06dbf2976fb3b9fb3&region=us-east-1: [Errno -1] Metadata file does not match checksum
Trying other mirror.


 One of the configured repositories failed (amzn-updates-Base),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable amzn-updates

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=amzn-updates.skip_if_unavailable=true

failure: repodata/primary.sqlite.bz2 from amzn-updates: [Errno 256] No more mirrors to try.

[switch180]

I suggest we add monitoring to this issue by failing the CFN signal when the update_yum_packages command is not successful. Today, we always succeed.

[switch180]

PR accepted and merged.