www.podman.io flaking - netavark bug?

[edsantiago]

Tests trying to fetch from www.podman.io are flaking. Sometimes a curl timeout, sometimes a DNS name resolution failure:

curl: (6) Could not resolve host: www.podman.io

I can't tell if it's a real problem with podman.io and/or its nameservers, or a connectivity hiccup, or if it's a Neta/Aardvark problem. The latter seems unlikely because I've only seen this with podman.io.

• fedora-37 : int podman fedora-37 root container
  • PR Quadlet -dryrun arg #16926
    • 01-03 10:37
    • 01-03 10:37
    • 01-03 10:37
    • 01-03 09:18
    • 01-03 09:18
    • 01-03 09:18
• fedora-37 : int podman fedora-37 root host
  • PR oci: terminate all container processes on cleanup #17025
    • 01-07 10:01
    • 01-07 10:01
    • 01-07 10:01
• fedora-37 : int remote fedora-37 root host [remote]
  • PR oci: terminate all container processes on cleanup #17025
    • 01-07 10:38
    • 01-07 10:38
    • 01-07 10:38
  • PR e2e: fix run_volume_test #16989
    • 01-04 02:27
    • 01-04 02:27
    • 01-04 02:27
  • PR Support sig-proxy for podman-remote attach and start #16985
    • 01-04 10:29
    • 01-04 10:29
    • 01-04 10:29
  • PR podman service: close duplicated /dev/null fd #16901
    • 12-20 12:10
    • 12-20 12:10
    • 12-20 12:10
  • PR Allow '/' to prefix container names to match Docker #16820
    • 12-29 06:40
    • 12-29 06:40
    • 12-29 06:40
    • 12-26 08:29

There are many more but I'm getting tired of scanning through flake logs.

[Luap99]

podman.io is hosted on github infra. I think one easy change would be to switch it to something like google.com and hope that this is a server side problem and not a client side bug.

[rhatdan]

Should do redhat.com before google.

[edsantiago]

Yesterday, remote f37 root

[edsantiago]

Well, shucks. It's not a podman.io problem

[edsantiago]

@containers/netavark-maintainers PTAL. It's possible that this is a network-infrastructure problem in Cirrus or gcloud... but it's also possible that our networking is broken.

[Luap99]

Do you know when this started? I know there was a bug in netavark v1.3 which could cause this. VM image updates should include netavark v1.4 but the automation_image CI is still broken so I cannot update it at the moment.
Edit: Updates should be working again so I will try to update the VM images: #16525

[edsantiago]

First instance of Could not resolve host: www.podman.io is Dec 5

First instance of command timed out ... wget www.podman.io is Nov 2

[edsantiago]

Reopening, because #17053 is unlikely to fix anything

[Luap99]

I agree, the common theme here is that it fails with --network private or bridge which means it uses netavark on f37, f36 uses CNI and at least from the instances listed here it is not effected. Therefore it it most likely a netavark bug.

[Luap99]

@edsantiago New images with a newer netavark were merged last week, does it still happen?
I don't remember it seeing it this week so far.

[edsantiago]

Sorry for the late reply: we're having a LOT of flakes this week, and I've had to categorize them one by one, which takes time.

The last two instances of this flake were on January 16, both on a PR that did not include #16525 (the update-images PR). I feel comfortable closing. Thank you!