Skip to content

fix: add fips tests and downgrade 2004 containerd#8380

Merged
awesomenix merged 1 commit intomainfrom
nishp/fix/e2e/azurecni
Apr 22, 2026
Merged

fix: add fips tests and downgrade 2004 containerd#8380
awesomenix merged 1 commit intomainfrom
nishp/fix/e2e/azurecni

Conversation

@awesomenix
Copy link
Copy Markdown
Contributor

We have an issue where new containerd version causes this

time="2026-04-22T19:11:59.236557991Z" level=errormsg="CreateContainer within sandbox \"1502c6d3e0dcbb098ed7c4170bdddde8c663d5462d31390ddaa1e0e63bd011b4\" for &ContainerMetadata{Name:node-driver-registrar,Attempt:0,} failed"error="failed to create containerd container: load apparmor profile /tmp/cri-containerd.apparmor.d1155998574: parsererror(\"AppArmor parsererrorfor /tmp/cri-containerd.apparmor.d1155998574 in /tmp/cri-containerd.apparmor.d1155998574 at line 2: Could not open 'abi/3.0': No such file or directory\"): exit status 1"

Its because of this PR containerd/containerd#12899

Copilot AI review requested due to automatic review settings April 22, 2026 19:29
@github-actions github-actions Bot added the components This pull request updates cached components on Linux or Windows VHDs label Apr 22, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Addresses an Ubuntu 20.04 FIPS node provisioning/runtime regression caused by newer containerd generating an AppArmor profile that references abi/3.0, by rolling back the Ubuntu 20.04 containerd package version and adding E2E coverage for the affected VHD.

Changes:

  • Downgrade Ubuntu 20.04 moby-containerd version pin in components.json.
  • Add a new E2E scenario for Ubuntu 20.04 FIPS (Gen1) bootstrapping.
  • Register the Ubuntu 20.04 FIPS containerd VHD in E2E VHD config.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
parts/common/components.json Rolls back Ubuntu 20.04 containerd version to avoid the AppArmor abi/3.0 regression.
e2e/scenario_test.go Adds a new Ubuntu 20.04 FIPS E2E scenario.
e2e/config/vhd.go Adds the Ubuntu 20.04 FIPS containerd image definition used by E2E.

Comment on lines 1137 to 1142
"versionsV2": [
{
"renovateTag": "name=moby-containerd, repository=production, os=ubuntu, release=20.04",
"latestVersion": "1.7.31-ubuntu20.04u1"
"latestVersion": "1.7.30-ubuntu20.04u3"
}
]
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This containerd version pin is keyed only by Ubuntu release (r2004), so it will affect every Ubuntu 20.04-based VHD (not just the 20.04 FIPS image). If the AppArmor regression only impacts FIPS kernels, consider scoping the rollback (if possible in this components model) or document in the PR/commit message that the downgrade is intentionally applied to all 20.04 images so future updates don’t accidentally reintroduce the issue.

Copilot uses AI. Check for mistakes.
Comment thread e2e/scenario_test.go
},
VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
},
Validator: func(ctx context.Context, s *Scenario) {
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new Ubuntu 20.04 FIPS scenario doesn’t validate the container runtime package versions. Since this PR is explicitly downgrading Ubuntu 20.04 containerd to address a runtime regression, it would be valuable for this test to assert the installed moby-containerd (and moby-runc) versions using components.GetExpectedPackageVersions("containerd", "ubuntu", "r2004") / ("runc", "ubuntu", "r2004").

Suggested change
Validator: func(ctx context.Context, s *Scenario) {
Validator: func(ctx context.Context, s *Scenario) {
ValidateInstalledPackageVersion(ctx, s, "moby-containerd", components.GetExpectedPackageVersions("containerd", "ubuntu", "r2004")[0])
ValidateInstalledPackageVersion(ctx, s, "moby-runc", components.GetExpectedPackageVersions("runc", "ubuntu", "r2004")[0])

Copilot uses AI. Check for mistakes.
Comment thread e2e/scenario_test.go
VHD: config.VHDUbuntu2004FIPSContainerd,
BootstrapConfigMutator: func(nbc *datamodel.NodeBootstrappingConfiguration) {
},
VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is named as a FIPS scenario, but unlike the other FIPS tests in this file it doesn’t set VMSS AdditionalCapabilities.EnableFips1403Encryption (or apply the Settings/ProtectedSettings swap workaround). If the intent is to exercise FIPS-mode bootstrapping, align the VMConfigMutator with the Ubuntu 2204 FIPS tests; otherwise the test name/description is misleading and coverage is reduced.

Suggested change
VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
vmss.Properties.AdditionalCapabilities = &armcompute.AdditionalCapabilities{
EnableFips1403Encryption: to.Ptr(true),
}
settings := vmss.Properties.VirtualMachineProfile.ExtensionProfile.Extensions[0].Properties.ProtectedSettings
vmss.Properties.VirtualMachineProfile.ExtensionProfile.Extensions[0].Properties.Settings = settings
vmss.Properties.VirtualMachineProfile.ExtensionProfile.Extensions[0].Properties.ProtectedSettings = nil

Copilot uses AI. Check for mistakes.
Comment thread e2e/scenario_test.go
Comment on lines +695 to +698
BootstrapConfigMutator: func(nbc *datamodel.NodeBootstrappingConfiguration) {
},
VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
},
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BootstrapConfigMutator and VMConfigMutator are defined but empty here. Consider omitting them (leave nil) to reduce noise and keep the scenario focused on the meaningful config/validation.

Copilot uses AI. Check for mistakes.
@awesomenix awesomenix merged commit 2052162 into main Apr 22, 2026
22 of 40 checks passed
@awesomenix awesomenix deleted the nishp/fix/e2e/azurecni branch April 22, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

components This pull request updates cached components on Linux or Windows VHDs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants