update images

[edsantiago]

Signed-off-by: Ed Santiago santiago@redhat.com

[github-actions]

Cirrus CI build successful. Found built image names and IDs:

Stage	Image Name	IMAGE_SUFFIX
base	debian	do-not-use
base	fedora	do-not-use
base	fedora-aws	do-not-use
base	fedora-aws-arm64	do-not-use
base	image-builder	do-not-use
base	prior-fedora	do-not-use
cache	build-push	c20231106t160529z-f39f38d13
cache	debian	c20231106t160529z-f39f38d13
cache	fedora	c20231106t160529z-f39f38d13
cache	fedora-aws	c20231106t160529z-f39f38d13
cache	fedora-netavark	c20231106t160529z-f39f38d13
cache	fedora-netavark-aws-arm64	c20231106t160529z-f39f38d13
cache	fedora-podman-aws-arm64	c20231106t160529z-f39f38d13
cache	fedora-podman-py	c20231106t160529z-f39f38d13
cache	prior-fedora	c20231106t160529z-f39f38d13
cache	rawhide	c20231106t160529z-f39f38d13
cache	win-server-wsl	c20231106t160529z-f39f38d13

[edsantiago]

@cevich if you have a spare moment could you look at the fedora-aws Base Image failure please?

==> fedora-aws: Stopping the source instance...
    fedora-aws: Stopping instance
==> fedora-aws: Waiting for the instance to stop...
==> fedora-aws: Error waiting for instance to stop: ResourceNotReady: exceeded wait attempts
==> fedora-aws: Provisioning step had errors: Running the cleanup provisioner, if present...
==> fedora-aws: Terminating the source AWS instance...
==> fedora-aws: ResourceNotReady: failed waiting for successful resource state

I can't find the string "Waiting for the instance to stop" anywhere in the likely source trees, so I have no idea what is running or what the bug is.

FWIW, the python-3.12 bug is a red herring, my last build threw the same error, but worked anyway.

[cevich]

The podman-py stuff I believe Urvashi sorted out. There's an actual bug in pylint and she found a workaround.

The error you got is coming from Packer. I've seen similar things before, it looks like a flake to me. It probably orphaned a VM (we can worry about that later). I restarted the task and will keep an eye on it as I'm able today...

[cevich]

...uggg. Amazon is having a bad day, re-running again...

[edsantiago]

It doesn't seem to be a flake. I restarted it four times yesterday.

[cevich]

I don't think we've changed the packer version recently, so it must be something on the Amazon side. Perhaps triggering a bug in packer.

In my last attempt, I found the line:
fedora-aws: Instance ID: i-0ac23dd69f36c7d41

I looked that instance up on the AWS EC2 console, and it shows the status as "terminated" - which is correct.

I found a few other instances in a "stopped" state, that shouldn't happen.

If you'd like to try figuring and bumping up the packer timeout, that may get you past the hump. Otherwise we may need a newer version (which may not accept our current cloud.yml).

[cevich]

Looking again:

==> fedora-aws: Waiting for the instance to stop...
==> fedora-aws: Error waiting for instance to stop: ResourceNotReady: exceeded wait attempts

I bet amazon changed some timings on their end. Such that (for example) it tries an ACPI shutdown, waits, tries again, waits, then "yanks the plug". If the timings of any of that collide with what packer is expecting, we'd get this problem.

It's highly-likely there's a timeout setting for this, probably needs to be added to the cloud.yml. Sometimes they (HashiCorp) do it as a CLI option or by env. vars. But I doubt it in this case.

[cevich]

If we need to dig deeper, there are options here as well. AWS keeps a log of basically every API request per user. So it's pretty easy to see if and when the request came in. In this case, it does look like a StopInstance is being received, runs for ~10 minutes, then there's a TerminateInstance call.

[edsantiago]

Closing in favor of #312. Hoping all these timeouts and errors go away.