Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISO: Upgrade podman to 3.1.2 #11727

Closed
wants to merge 1 commit into from

Conversation

afbjorklund
Copy link
Collaborator

@afbjorklund afbjorklund commented Jun 22, 2021

Note: podman3 requires go to be updated first (new OS)

build flag -mod=vendor only valid when using modules

Closes #11706

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 22, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 22, 2021
@afbjorklund afbjorklund changed the title Upgrade podman to 3.1.2 ISO: Upgrade podman to 3.1.2 Jun 22, 2021
@medyagh
Copy link
Member

medyagh commented Jun 23, 2021

ok-to-build-iso

@minikube-bot

This comment has been minimized.

@medyagh
Copy link
Member

medyagh commented Jul 29, 2021

ok-to-build-iso

@minikube-bot

This comment has been minimized.

@afbjorklund
Copy link
Collaborator Author

As stated above, this requires the OS to be updated first

@afbjorklund
Copy link
Collaborator Author

ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @afbjorklund, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 17, 2021
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 3, 2021
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 6, 2021
@spowelljr
Copy link
Member

@afbjorklund Based on #11706 (comment), is this PR something we still want to pursue? And if so, does anything else still have to be done? #11727 (comment)?

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Oct 5, 2021

Well, there is a 3.4.0 now. They say it will be stable. Except of course that podman machine still has a broken ISO, etc

I think we could go for 3.3.1, but it is already EOL and all fixes go in 3.4 - except for the ones that will go into 4.0 instead

Perhaps 3.0.1 is a conservative pick ?

It's used in RHEL 7 and other dinosaurs

@spowelljr
Copy link
Member

I'm happy to go with whatever version you think is best, I can create a PR to update to your preferred version if you don't have the time to do so.

@albertdb
Copy link

albertdb commented Dec 4, 2021

So is v3.1.2 the final choice or is it a reference to then try and compare with v3.4.2?

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux TestFunctional/serial/ComponentHealth (gopogh) 0.00 (chart)
Docker_Cloud_Shell TestFunctional/parallel/MountCmd/specific-port (gopogh) 1.87 (chart)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 1.89 (chart)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 2.80 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 4.32 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/EnableAddonAfterStop (gopogh) 14.15 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/SecondStart (gopogh) 14.15 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/Stop (gopogh) 14.15 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/UserAppExistsAfterStop (gopogh) 14.95 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 16.49 (chart)
Docker_macOS TestNoKubernetes/serial/StartNoArgs (gopogh) 16.78 (chart)
Docker_macOS TestNetworkPlugins/group/kindnet/DNS (gopogh) 28.95 (chart)
Docker_macOS TestNetworkPlugins/group/kubenet/Start (gopogh) 29.92 (chart)
Docker_macOS TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 31.82 (chart)
Docker_Windows TestNetworkPlugins/group/bridge/Start (gopogh) 40.21 (chart)
Docker_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) 46.39 (chart)
Docker_macOS TestNetworkPlugins/group/bridge/DNS (gopogh) 71.58 (chart)
Docker_macOS TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 74.29 (chart)
Hyper-V_Windows TestMultiNode/serial/CopyFile (gopogh) 76.69 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartNoArgs (gopogh) 92.94 (chart)
Docker_Windows TestFunctional/parallel/DryRun (gopogh) 96.91 (chart)
Docker_Windows TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh) 96.91 (chart)
Docker_Windows TestFunctional/parallel/InternationalLanguage (gopogh) 96.91 (chart)
Docker_Windows TestFunctional/parallel/StatusCmd (gopogh) 96.91 (chart)
Docker_Windows TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 96.91 (chart)
Docker_macOS TestDownloadOnly/v1.16.0/preload-exists (gopogh) 100.00 (chart)
Docker_Windows TestInsufficientStorage (gopogh) 100.00 (chart)
Docker_Windows TestMultiNode/serial/CopyFile (gopogh) 100.00 (chart)
Docker_Windows TestNetworkPlugins/group/cilium/Start (gopogh) 100.00 (chart)
Docker_Windows TestPause/serial/VerifyDeletedResources (gopogh) 100.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Dec 4, 2021

So is v3.1.2 the final choice or is it a reference to then try and compare with v3.4.2?

We can bump it further, I don't think the version was ever the problem with the ISO build.

The go version requirement was a bit of a blocker before, but it has 1.15 now at least.
After upgrading the OS version, that version would increase from 1.15.15 to 1.16.10

Usually the versions that RHEL and Debian choose, are better choices for long term

  • RHEL 8.5: v3.3.1
  • Debian 11: v3.0.1

EDIT: Forgot that go was patched to 1.17 already

Can try upgrading it from v3.1.2 to v3.4.2 later on

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 46.1s    | 46.2s               |
| enable ingress | 31.4s    | 30.2s               |
+----------------+----------+---------------------+

Times for minikube start: 46.0s 46.0s 46.7s 45.5s 46.3s
Times for minikube (PR 11727) start: 47.1s 45.3s 46.0s 47.2s 45.2s

Times for minikube ingress: 32.2s 31.7s 30.8s 30.8s 31.7s
Times for minikube (PR 11727) ingress: 31.2s 30.8s 30.2s 29.3s 29.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 20.8s    | 20.3s               |
| enable ingress | 26.5s    | 26.1s               |
+----------------+----------+---------------------+

Times for minikube (PR 11727) start: 20.2s 20.8s 19.9s 20.1s 20.4s
Times for minikube start: 21.9s 21.0s 20.6s 20.2s 20.3s

Times for minikube ingress: 26.4s 25.9s 26.9s 25.9s 27.4s
Times for minikube (PR 11727) ingress: 25.4s 26.9s 24.4s 26.4s 27.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 38.0s    | 37.9s               |
| enable ingress | 26.6s    | 29.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 32.9s 32.4s 32.4s 17.9s 17.4s
Times for minikube (PR 11727) ingress: 32.9s 31.9s 31.4s 31.4s 17.9s

Times for minikube start: 25.3s 42.2s 40.8s 40.8s 41.1s
Times for minikube (PR 11727) start: 25.1s 41.4s 40.8s 41.0s 41.2s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 2.80 (chart)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 3.70 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 4.29 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 4.86 (chart)
Docker_macOS TestStoppedBinaryUpgrade/MinikubeLogs (gopogh) 9.03 (chart)
Docker_macOS TestStoppedBinaryUpgrade/Upgrade (gopogh) 9.03 (chart)
Docker_macOS TestKubernetesUpgrade (gopogh) 12.50 (chart)
Docker_macOS TestNoKubernetes/serial/ProfileList (gopogh) 13.89 (chart)
Docker_macOS TestNoKubernetes/serial/Start (gopogh) 13.89 (chart)
Docker_macOS TestNoKubernetes/serial/Stop (gopogh) 13.89 (chart)
Docker_macOS TestPause/serial/SecondStartNoReconfiguration (gopogh) 13.89 (chart)
Docker_macOS TestPause/serial/Start (gopogh) 13.89 (chart)
Docker_macOS TestPause/serial/Unpause (gopogh) 13.89 (chart)
Docker_macOS TestPause/serial/VerifyDeletedResources (gopogh) 13.89 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/AddonExistsAfterStop (gopogh) 14.02 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/EnableAddonAfterStop (gopogh) 14.02 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/SecondStart (gopogh) 14.02 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/Stop (gopogh) 14.02 (chart)
Docker_macOS TestPause/serial/Pause (gopogh) 14.58 (chart)
Docker_macOS TestPause/serial/PauseAgain (gopogh) 14.58 (chart)
Docker_macOS TestPause/serial/VerifyStatus (gopogh) 14.58 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/UserAppExistsAfterStop (gopogh) 14.81 (chart)
Docker_macOS TestNetworkPlugins/group/auto/Start (gopogh) 16.54 (chart)
Docker_macOS TestNetworkPlugins/group/cilium/Start (gopogh) 16.54 (chart)
Docker_macOS TestNetworkPlugins/group/false/Start (gopogh) 16.54 (chart)
Docker_macOS TestNoKubernetes/serial/StartNoArgs (gopogh) 16.67 (chart)
Docker_macOS TestNetworkPlugins/group/enable-default-cni/Start (gopogh) 17.32 (chart)
Docker_macOS TestNetworkPlugins/group/bridge/Start (gopogh) 25.78 (chart)
Docker_macOS TestStartStop/group/default-k8s-different-port/serial/AddonExistsAfterStop (gopogh) 25.93 (chart)
Docker_macOS TestStartStop/group/default-k8s-different-port/serial/DeployApp (gopogh) 25.93 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 8, 2021
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 8, 2021
@afbjorklund afbjorklund mentioned this pull request Dec 8, 2021
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 45.1s    | 44.8s               |
| enable ingress | 28.7s    | 28.5s               |
+----------------+----------+---------------------+

Times for minikube start: 45.7s 45.0s 44.5s 44.3s 46.2s
Times for minikube (PR 11727) start: 46.0s 45.0s 45.0s 43.6s 44.4s

Times for minikube (PR 11727) ingress: 28.6s 29.1s 26.1s 29.1s 29.6s
Times for minikube ingress: 28.1s 29.6s 31.1s 29.1s 25.6s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 26.3s    | 26.5s               |
| enable ingress | 22.3s    | 22.0s               |
+----------------+----------+---------------------+

Times for minikube start: 27.1s 26.5s 26.6s 25.8s 25.8s
Times for minikube (PR 11727) start: 26.3s 26.5s 26.8s 26.6s 26.1s

Times for minikube ingress: 22.4s 22.9s 21.9s 21.9s 22.4s
Times for minikube (PR 11727) ingress: 21.9s 23.4s 21.4s 22.4s 20.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11727) |
+----------------+----------+---------------------+
| minikube start | 40.3s    | 42.9s               |
| enable ingress | 38.4s    | 31.7s               |
+----------------+----------+---------------------+

Times for minikube start: 30.9s 41.7s 41.2s 41.8s 45.7s
Times for minikube (PR 11727) start: 45.7s 42.0s 41.1s 44.2s 41.4s

Times for minikube ingress: 33.4s 33.4s 33.4s 32.9s 58.9s
Times for minikube (PR 11727) ingress: 29.4s 33.4s 32.4s 29.9s 33.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 0.00 (chart)
KVM_Linux_containerd TestPause/serial/VerifyDeletedResources (gopogh) 0.00 (chart)
KVM_Linux TestAddons/Setup (gopogh) 0.00 (chart)
KVM_Linux TestForceSystemdFlag (gopogh) 0.00 (chart)
KVM_Linux TestNetworkPlugins/group/kubenet/Start (gopogh) 1.45 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 5.13 (chart)
Docker_Windows TestNetworkPlugins/group/bridge/NetCatPod (gopogh) 5.56 (chart)
KVM_Linux TestGvisorAddon (gopogh) 5.80 (chart)
Docker_Windows TestNoKubernetes/serial/Start (gopogh) 7.22 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 7.55 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 7.55 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 7.55 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/VerifyKubernetesImages (gopogh) 7.55 (chart)
Docker_Windows TestNetworkPlugins/group/false/Start (gopogh) 9.43 (chart)
Docker_Windows TestPause/serial/Pause (gopogh) 10.78 (chart)
Docker_Windows TestKubernetesUpgrade (gopogh) 13.21 (chart)
Docker_Windows TestPause/serial/PauseAgain (gopogh) 15.69 (chart)
Docker_Windows TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 16.98 (chart)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 36.43 (chart)
Docker_Cloud_Shell TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 36.92 (chart)
Docker_Windows TestNetworkPlugins/group/kindnet/Start (gopogh) 38.68 (chart)
Docker_macOS TestNetworkPlugins/group/custom-weave/Start (gopogh) 39.50 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithK8s (gopogh) 63.16 (chart)
Docker_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) 84.62 (chart)
Docker_Windows TestPause/serial/VerifyStatus (gopogh) 95.10 (chart)
Docker_Windows TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 96.23 (chart)
Docker_macOS TestDownloadOnly/v1.16.0/preload-exists (gopogh) 100.00 (chart)
Docker_Windows TestMultiNode/serial/CopyFile (gopogh) 100.00 (chart)
Docker_Windows TestNetworkPlugins/group/cilium/Start (gopogh) 100.00 (chart)
Docker_Windows TestNoKubernetes/serial/StartWithK8s (gopogh) 100.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@albertdb
Copy link

albertdb commented Dec 13, 2021

So is v3.1.2 the final choice or is it a reference to then try and compare with v3.4.2?

We can bump it further, I don't think the version was ever the problem with the ISO build.

The go version requirement was a bit of a blocker before, but it has 1.15 now at least. After upgrading the OS version, that version would increase from 1.15.15 to 1.16.10

Usually the versions that RHEL and Debian choose, are better choices for long term

  • RHEL 8.5: v3.3.1
  • Debian 11: v3.0.1

EDIT: Forgot that go was patched to 1.17 already

Can try upgrading it from v3.1.2 to v3.4.2 later on

Did you have success? v3,4,4 is out and v3.4.3 fixes two CVEs.

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Dec 13, 2021

Nope. Not sure how much the podman machine CVE affects minikube, it mostly affects the host Podman ? Nor if anyone uses OCI images, minikube normally uses Docker images.

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Dec 13, 2021

Anyway, ISO build failures were unrelated to podman

@albertdb
Copy link

Anyway, ISO build failures were unrelated to podman

Nice to hear. Any chance to rebase on top of latest stable release? I don't know what is your Git workflow, but if something is polluting the tests, it would be better to separate concerns.

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Dec 13, 2021

It builds OK here

The monolithic build is a stability issue for CI, but also the packages can have their own issues (currently KIC us failing on checksums)

@albertdb
Copy link

It builds OK here

Quite strange then, if the only change is the podman version and the reason for not building is unrelated to podman :S

@albertdb
Copy link

albertdb commented Dec 13, 2021

The monolithic build is a stability issue for CI, but also the packages can have their own issues (currently KIC us failing on checksums)

Oh, ok. So then there is no reason to choose one instead of another, isn't it?

I mean, being up-to-date today means not being too out-of-date soon.

@afbjorklund
Copy link
Collaborator Author

Getting some help now, so hopefully will be back on track soon. First we want to upgrade os version and build for arm64.

But upgrading kernel and container runtimes is also on the wishlist, and long overdue due to the other upgrade issues...

@afbjorklund
Copy link
Collaborator Author

Let's do another PR, when this is possible.

@gallomas
Copy link

@afbjorklund I would be willing to help on this. Looking at comments and trying to understand what was the problem blocking the PR / tests. Is there a issue opened to track?

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jan 26, 2022

Not really, it was a simple version upgrade...

Should probably upgrade to something like 3.4.2, before the big 4.0.0 comes and deletes all the packages.

For minikube purposes, the actual 3.x probably doesn't matter at all. And 3.0.1 is still "supported", for afaik.

Most requests are still for dockerd, some for containerd.

2021-container-orchestration-report-FACT-8v2

(from https://www.datadoghq.com/container-report/#8)

@albertdb
Copy link

albertdb commented Jan 26, 2022

There are legitimate use cases for more recent (better and less buggier) versions of Podman, such as https://github.com/devonfw-forge/podman-minikube (i.e. ditching Docker Desktop and even docker)

@afbjorklund
Copy link
Collaborator Author

There are legitimate use cases for more recent versions of Podman, such as https://github.com/devonfw-forge/podman-minikube (i.e. ditching Docker Desktop and even docker)

Podman has its own built-in solution with CoreOS, but sure it could be useful for minikube start --no-kubernetes

@albertdb
Copy link

albertdb commented Jan 26, 2022

There are legitimate use cases for more recent versions of Podman, such as https://github.com/devonfw-forge/podman-minikube (i.e. ditching Docker Desktop and even docker)

Podman has its own built-in solution with CoreOS, but sure it could be useful for minikube start --no-kubernetes

The idea is to "silently" introduce Kubernetes and let developers naturally use it just because they already have it.

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jan 26, 2022

The plan is still to make both podman and nerdctl versions available, presumably using modern versions of each.

Just have to get the ISO out of the current sad state.

And arm64 still has higher prio, if it comes to that.

@albertdb
Copy link

albertdb commented Jan 26, 2022

There are legitimate use cases for more recent versions of Podman, such as https://github.com/devonfw-forge/podman-minikube (i.e. ditching Docker Desktop and even docker)

Podman has its own built-in solution with CoreOS, but sure it could be useful for minikube start --no-kubernetes

I guess you were talking about Podman machine for MacOS, I was talking about Windows, in the cases where WSL2 is not feasible.

Or does Podman machine offer an independent backend on Windows?

Edit: not yet containers/podman#13006

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stop using podman2 for loading and building images for cri-o
9 participants