feat: experimental azure report ready#8935
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an experimental “report ready/failure to Azure fabric” flow intended for Ubuntu images by (a) shipping a standalone Python reporter into the VHD, (b) patching Ubuntu cloud-init’s Azure datasource to optionally skip its built-in ready report based on user-data, and (c) updating e2e VMSS plumbing to optionally skip CustomScript extension/status handling.
Changes:
- Add a new cloud-init provisioning status reporter script and attempt to invoke it from
cse_start.shafter provisioning completes. - Patch Ubuntu cloud-init (
DataSourceAzure.py) during VHD build to treat a# azure-skip-ready-reportuser-data marker as an additional condition to skip the built-in report. - Add an e2e
SkipCustomScriptExtensionStatusflag and use it to alter VMSS extension behavior and status parsing.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| vhdbuilder/packer/vhd-image-builder-cvm.json | Adds a new artifact to the Packer file provisioner list. |
| vhdbuilder/packer/vhd-image-builder-base.json | Adds a new artifact to the Packer file provisioner list. |
| vhdbuilder/packer/vhd-image-builder-arm64-gen2.json | Adds a new artifact to the Packer file provisioner list. |
| vhdbuilder/packer/vhd-image-builder-arm64-gb.json | Adds a new artifact to the Packer file provisioner list. |
| vhdbuilder/packer/packer_source.sh | Copies the new reporter script into /opt/azure/containers/ during image build. |
| vhdbuilder/packer/install-dependencies.sh | Patches Ubuntu cloud-init Azure datasource to support the new “skip ready report” marker. |
| parts/linux/cloud-init/artifacts/cse_start.sh | Attempts to invoke the reporter script to send Ready/NotReady after CSE finishes. |
| parts/linux/cloud-init/artifacts/clout_init_report_ready.py | New Python implementation of Ready/NotReady reporting to wireserver/KVP. |
| e2e/vmss.go | Adds conditional logic to avoid CSE extension status parsing when skipped. |
| e2e/test_helpers.go | Adds conditional logic around fetching CustomScript extension status. |
| e2e/config/vhd.go | Introduces SkipCustomScriptExtensionStatus in the Image config and enables it for several Ubuntu images. |
Comments suppressed due to low confidence (1)
vhdbuilder/packer/vhd-image-builder-arm64-gen2.json:246
- The added file provisioner references
parts/linux/cloud-init/artifacts/cloud_init_report_ready.py, but that file does not exist in the repo (the new script isclout_init_report_ready.py). This will break Packer builds until the filename and all references are made consistent.
"source": "parts/linux/cloud-init/artifacts/cse_send_logs.py",
"destination": "/home/packer/cse_send_logs.py"
},
{
"type": "file",
| { | ||
| "type": "file", | ||
| "source": "parts/linux/cloud-init/artifacts/cloud_init_report_ready.py", | ||
| "destination": "/home/packer/cloud_init_report_ready.py" | ||
| }, |
| { | ||
| "type": "file", | ||
| "source": "parts/linux/cloud-init/artifacts/cloud_init_report_ready.py", | ||
| "destination": "/home/packer/cloud_init_report_ready.py" | ||
| }, |
| { | ||
| "type": "file", | ||
| "source": "parts/linux/cloud-init/artifacts/cloud_init_report_ready.py", | ||
| "destination": "/home/packer/cloud_init_report_ready.py" | ||
| }, |
| if isUbuntu "$OS"; then | ||
| CSE_CLOUD_INIT_REPORT_READY_SRC=/home/packer/cloud_init_report_ready.py | ||
| CSE_CLOUD_INIT_REPORT_READY_DEST=/opt/azure/containers/cloud_init_report_ready.py | ||
| cpAndMode $CSE_CLOUD_INIT_REPORT_READY_SRC $CSE_CLOUD_INIT_REPORT_READY_DEST 0744 | ||
| fi |
| # When this file exists, cloud-init is handling the ready report itself, | ||
| # so this script should not send a duplicate report to the wireserver. | ||
| REPORT_MARKER = "/var/lib/waagent/experimental_skip_ready_report" |
| Skipped if the experimental_skip_ready_report marker exists, which | ||
| indicates cloud-init is handling the report itself. | ||
| """ |
| Skipped if the experimental_skip_ready_report marker exists. | ||
| """ |
| if !s.VHD.SkipCustomScriptExtensionStatus { | ||
| exitCode, ok := getLinuxCSEExitCode(ctx, s) | ||
| if !ok || exitCode != cseExitCodeOutboundConnFail { | ||
| return vm, err | ||
| } | ||
| } | ||
|
|
||
| toolkit.Logf(ctx, "CSE failed with ERR_OUTBOUND_CONN_FAIL (exit %s) on VMSS %q: known transient e2e outbound flake, recreating node (attempt %d/%d)", exitCode, s.Runtime.VMSSName, attempt+1, maxOutboundCSERetries) | ||
| // Close this attempt's bastion tunnel before recreating: the SSH client is established |
| if s.VHD.SkipCustomScriptExtensionStatus { | ||
| err = getCustomScriptExtensionStatus(s, scenarioVM.VM) | ||
| require.NoError(s.T, err) | ||
| } |
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172153655 Detective summary: Build VHD failed before image creation because Likely cause / signature: $skuSig. Confidence: High for immediate cause; medium for source of variable loss. Recommended owner/action: AgentBaker Linux VHD build pipeline/template owner should inspect definition 119535 matrix/variable generation and restore SKU_NAME for Linux VHD jobs. No PR author action unless PR #8935 changed pipeline templates or SKU wiring. Strongest alternative: SIG image lookup race after successful build; less likely because the earlier Build VHD step aborts before image creation with missing SKU_NAME. Evidence: Timeline shows Build VHD failures across multiple jobs; logs show Wiki signature: $skuSig |
awesomenix
force-pushed
the
nishp/experimental/skip
branch
from
2411223 to
ade310f
Compare
| if !s.VHD.SkipCustomScriptExtensionStatus { | ||
| exitCode, ok := getLinuxCSEExitCode(ctx, s) | ||
| if !ok || exitCode != cseExitCodeOutboundConnFail { | ||
| return vm, err | ||
| } | ||
| } | ||
|
|
||
| toolkit.Logf(ctx, "CSE failed with ERR_OUTBOUND_CONN_FAIL (exit %s) on VMSS %q: known transient e2e outbound flake, recreating node (attempt %d/%d)", exitCode, s.Runtime.VMSSName, attempt+1, maxOutboundCSERetries) |
| if s.VHD.SkipCustomScriptExtensionStatus { | ||
| err = getCustomScriptExtensionStatus(s, scenarioVM.VM) | ||
| require.NoError(s.T, err) | ||
| } |
| else | ||
| python3 /opt/azure/containers/cloud_init_report_ready.py -v --failure --description "ExitCode: ${EXIT_CODE}, ${message_string}" || echo "WARNING: Failed to report failure to Azure fabric" | ||
| fi |
| if [ -x /opt/azure/containers/cloud_init_report_ready.py ]; then | ||
| if [ "$EXIT_CODE" -eq 0 ]; then | ||
| python3 /opt/azure/containers/cloud_init_report_ready.py -v || echo "WARNING: Failed to report ready to Azure fabric" |
| Skipped if the experimental_skip_ready_report marker exists, which | ||
| indicates cloud-init is handling the report itself. | ||
| """ |
| Skipped if the experimental_skip_ready_report marker exists. | ||
| """ |
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172164489 Detective summary: Build VHD now gets past the previous SKU_NAME issue but fails during Packer provisioning. The representative failure is a cloud-init patch precondition: Likely cause / signature: $sig. Confidence: High; PR author action recommended. Recommended owner/action: PR author/VHD builder reviewers should make the DataSourceAzure patch cloud-init-version-aware or avoid patching distro-owned cloud-init internals; validate Ubuntu 20.04, 22.04, 24.04 and arm64 paths before rerun. Strongest alternative: Base image cloud-init changed unexpectedly; less likely as an independent infra issue because the newly added patch assumes an exact source line and fails deterministically when it is absent or not unique. Evidence: ADO timeline shows Build VHD failures across multiple lanes; representative Packer log shows the DataSourceAzure precondition failure and cleanup provisioner start. Wiki signature: $sig |
NOTHING TO LOOK HERE