`flatcar-postinst` does not download sysext images from URLs advertised by Nebraska

[simoncampion]

Description

(Disclaimer: This might not be a bug but just me misunderstanding how to use Nebraska to advertise URLs for sysext images.)

When Nebraska sends URLs of OEM or custom extension sysext images in response to update requests from the update engine, then flatcar-postinst should use those URLs to download sysext images. But for me, this does not work: flatcar-postinst falls back to hard-coded URLs of release servers even if Nebraska advertises URLs.

Impact

I cannot get Flatcar to download OEM and extension images using URLs configured in a Nebraska instance.

Environment and steps to reproduce

1. Set-up: Add a package to a Nebraska instance for Flatcar 3794. Add the metadata for flatcar_production_update.gz, and, under "extra files", add the metadata for oem-qemu.gz. As base URL, use a locally running file server that serves those files. Write a Butane config that instructs Flatcar to pull this update package from the Nebraska instance. Then, start a Flatcar 3760.1.0 VM with QEMU using the official QEMU image.
2. Task: Update Flatcar using update_engine_client -check_for_update
3. Action(s):
   a. Run update_engine_client -check_for_update.
   b. Observe in webserver logs that flatcar_production_update.gz is downloaded.
   c. Observe in webserver logs that oem-qemu.gz is not downloaded. Observe in tcpdump that the VM talks to update.release.flatcar-linux.net, the fallback URL defined in the flatcar-postinst Bash script, downloading about 2KB, which is the size of oem-qemu.gz.
4. Error: None. The fallback servers offer oem-qemu.gz, so the installer terminates without error. (To provoke an error, a more complex setup can be used in which a custom extension is defined that is only present on the local file server. In that case, flatcar-postinst will fail and report a 404 error because the fallback servers does not offer the custom extension image.)

Expected behavior

oem-qemu.gz is downloaded from the local webserver, i.e., from the URL defined in Nebraska.

Additional information

Support for this feature was added in this PR.

My speculative explanation for why this happens:

1. The update engine sends an update request to Nebraska. It receives a response that contains URLs for oem-qemu.gz and other files and writes the response to /var/lib/update_engine/prefs/full-response.
2. The update engine downloads flatcar_production_update.gz from the URL specified by Nebraska, i.e., the local webserver.
3. The update engine sends a request to Nebraska to inform it that the download is finished. It writes Nebraska's response to that request to /var/lib/update_engine/prefs/full-response. The file does not contain the URLs anymore.
4. The update engine starts the Bash script flatcar-postinst.
5. flatcar-postinst checks the contents of /var/lib/update_engine/prefs/full-response and can't find URLs in it. Thus, flatcar-postinst falls back to using hard-coded URLs of public Flatcar servers.

Excerpt from the journal (some URLs / names redacted):

Dec 07 13:11:42 localhost systemd[1]: Starting update-engine.service - Update Engine...
Dec 07 13:11:42 localhost update_engine[1419]: I1207 13:11:42.622797  1419 main.cc:92] Flatcar Update Engine starting
Dec 07 13:11:42 localhost systemd[1]: Started update-engine.service - Update Engine.
...
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.626497  1419 update_attempter.cc:275] New update check requested
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.626646  1419 prefs.cc:52] aleph-version not present in /var/lib/update_engine/prefs
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.626989  1419 omaha_request_params.cc:62] Current group set to 2a27a073-f6b9-4cad-a4d8-c6d43a2508dd
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627064  1419 update_attempter.cc:495] Already updated boot flags. Skipping.
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627071  1419 update_attempter.cc:638] Scheduling an action processor start.
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627087  1419 action_processor.cc:36] ActionProcessor::StartProcessing: OmahaRequestAction
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627110  1419 prefs.cc:52] previous-version not present in /var/lib/update_engine/prefs
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627156  1419 omaha_request_action.cc:269] Posting an Omaha request to https://REDACTED/v1/update/
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627161  1419 omaha_request_action.cc:270] Request: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:12:53 localhost update_engine[1419]: <request protocol="3.0" version="update_engine-0.4.10" updaterversion="update_engine-0.4.10" installsource="ondemandupdate" ismachine="1">
Dec 07 13:12:53 localhost update_engine[1419]:     <os version="Chateau" platform="CoreOS" sp="3760.1.0_x86_64"></os>
Dec 07 13:12:53 localhost update_engine[1419]:     <app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" version="3760.1.0" track="2a27a073-f6b9-4cad-a4d8-c6d43a2508dd" bootid="{0e1b1df7-ec8d-4fb2-a9db-7a56a888ed42}" oem="qemu" oemversion="0.
0.2" alephversion="3760.1.0" machineid="9e4a256d90d74407b306fd3d94d2b28c" machinealias="" lang="en-US" board="amd64-usr" hardware_class="" delta_okay="false" >
Dec 07 13:12:53 localhost update_engine[1419]:         <ping active="1"></ping>
Dec 07 13:12:53 localhost update_engine[1419]:         <updatecheck></updatecheck>
Dec 07 13:12:53 localhost update_engine[1419]:         <event eventtype="3" eventresult="2" previousversion="0.0.0.0"></event>
Dec 07 13:12:53 localhost update_engine[1419]:     </app>
Dec 07 13:12:53 localhost update_engine[1419]: </request>
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.627164  1419 libcurl_http_fetcher.cc:47] Starting/Resuming transfer
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.633056  1419 libcurl_http_fetcher.cc:174] Setting up curl options for HTTPS
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.634050  1419 libcurl_http_fetcher.cc:449] Setting up timeout source: 1 seconds.
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.799589  1419 prefs.cc:52] update-server-cert-0-2 not present in /var/lib/update_engine/prefs
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.799768  1419 prefs.cc:52] update-server-cert-0-1 not present in /var/lib/update_engine/prefs
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.799898  1419 prefs.cc:52] update-server-cert-0-0 not present in /var/lib/update_engine/prefs
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.885196  1419 libcurl_http_fetcher.cc:263] HTTP response code: 200
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.886547  1419 libcurl_http_fetcher.cc:320] Transfer completed (200), 1067 bytes downloaded
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.886556  1419 omaha_request_action.cc:615] Omaha request response: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:12:53 localhost update_engine[1419]: <response protocol="3.0" server="nebraska"><daystart elapsed_seconds="0"></daystart><app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" status="ok"><ping status="ok"></ping><updatecheck status=
"ok"><urls><url codebase="http://10.0.2.2:8000/3794.0.0/"></url></urls><manifest version="3794.0.0"><packages><package name="flatcar_production_update.gz" hash="32uV+75xVNSRqf4n/FEHGDWpk2E=" size="457850580" required="true"></package><pac
kage name="oem-qemu.gz" hash="fhKTxnZ7MY3mNlvOTK31ZmXiXqI=" hash_sha256="82e3a680473ec23120adcc9982633d437522d2cb6da54cb2aabcc20de8036934" size="2341" required="false"></package><package name="flatcar-REDACTED-extension.gz" hash="kdhq+kGcmLs
+fsy2WJwMrhp906s=" hash_sha256="347ffb3c9274c095cf3d43a8a3e7c279510ca9bbb18c89d4ab0e76e6b15d02ad" size="103354368" required="false"></package></packages><actions><action event="postinstall" sha256="eH3fPL2gdsF4g8kUiyKTs8+EgYjS4cEx4FgtPHqa
G2g=" DisablePayloadBackoff="true"></action></actions></manifest></updatecheck><event status="ok"></event></app></response>
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.892289  1419 omaha_request_action.cc:448] Omaha Response manifest version = 3794.0.0
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.892318  1419 omaha_request_action.cc:471] Found 1 url(s)
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.892329  1419 omaha_request_action.cc:507] Processing first of 3 package(s)
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.892333  1419 omaha_request_action.cc:514] Omaha Response package name = flatcar_production_update.gz
Dec 07 13:12:53 localhost update_engine[1419]: I1207 13:12:53.892336  1419 omaha_request_action.cc:530] Url0: http://10.0.2.2:8000/3794.0.0/flatcar_production_update.gz
[more lines]
[more lines]
[more lines]
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.168481  1419 omaha_request_action.cc:269] Posting an Omaha request to https://REDACTED/v1/update/
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.168485  1419 omaha_request_action.cc:270] Request: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:12:59 localhost update_engine[1419]: <request protocol="3.0" version="update_engine-0.4.10" updaterversion="update_engine-0.4.10" installsource="ondemandupdate" ismachine="1">
Dec 07 13:12:59 localhost update_engine[1419]:     <os version="Chateau" platform="CoreOS" sp="3760.1.0_x86_64"></os>
Dec 07 13:12:59 localhost update_engine[1419]:     <app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" version="3760.1.0" track="2a27a073-f6b9-4cad-a4d8-c6d43a2508dd" bootid="{0e1b1df7-ec8d-4fb2-a9db-7a56a888ed42}" oem="qemu" oemversion="0.
0.2" alephversion="3760.1.0" machineid="9e4a256d90d74407b306fd3d94d2b28c" machinealias="" lang="en-US" board="amd64-usr" hardware_class="" delta_okay="false" >
Dec 07 13:12:59 localhost update_engine[1419]:         <event eventtype="13" eventresult="1"></event>
Dec 07 13:12:59 localhost update_engine[1419]:     </app>
Dec 07 13:12:59 localhost update_engine[1419]: </request>
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.168489  1419 libcurl_http_fetcher.cc:47] Starting/Resuming transfer
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.170018  1419 libcurl_http_fetcher.cc:174] Setting up curl options for HTTPS
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.170198  1419 libcurl_http_fetcher.cc:449] Setting up timeout source: 1 seconds.
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.396238  1419 libcurl_http_fetcher.cc:263] HTTP response code: 200
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.397956  1419 libcurl_http_fetcher.cc:320] Transfer completed (200), 229 bytes downloaded
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.397974  1419 omaha_request_action.cc:615] Omaha request response: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:12:59 localhost update_engine[1419]: <response protocol="3.0" server="nebraska"><daystart elapsed_seconds="0"></daystart><app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" status="ok"><event status="ok"></event></app></response>
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398073  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished OmahaRequestAction, starting DownloadAction
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398079  1419 install_plan.cc:53] InstallPlan: , new_update, url: http://10.0.2.2:8000/3794.0.0/flatcar_production_update.gz, payload size: 457850580, payload hash: eH3fPL2gdsF4
g8kUiyKTs8+EgYjS4cEx4FgtPHqaG2g=, partition_path: /dev/vda4, kernel_path: /boot/flatcar/vmlinuz-b, pcr_policy_path: /var/lib/update_engine/pcrs-b.zip, old_partition_path: /dev/vda3, old_kernel_path: /boot/flatcar/vmlinuz-a
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398144  1419 update_attempter.cc:420] Download status: active
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398170  1419 multi_range_http_fetcher.cc:30] starting first transfer
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398175  1419 multi_range_http_fetcher.cc:58] starting transfer of range 0+?
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398180  1419 libcurl_http_fetcher.cc:47] Starting/Resuming transfer
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398317  1419 libcurl_http_fetcher.cc:151] Setting up curl options for HTTP
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.398463  1419 libcurl_http_fetcher.cc:449] Setting up timeout source: 1 seconds.
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.400015  1419 update_attempter.cc:442] Downloaded 4320/457850580 bytes (0%)
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.402920  1419 payload_processor.cc:39] PartitionInfo new_partition_info sha256: ul/xQQMWN1J5Z2UioqtFEpKObCTB/IkR1cWuThwzN9A= size: 1073741824
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.403095  1419 payload_processor.cc:226] Starting to apply update payload operations
Dec 07 13:12:59 localhost update_engine[1419]: I1207 13:12:59.407884  1419 payload_processor.cc:282] Completed 1/1077 operations (0%)
[more lines]
[more lines]
[more lines]
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.152225  1419 omaha_request_action.cc:269] Posting an Omaha request to https://REDACTED/v1/update/
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.152228  1419 omaha_request_action.cc:270] Request: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:13:19 localhost update_engine[1419]: <request protocol="3.0" version="update_engine-0.4.10" updaterversion="update_engine-0.4.10" installsource="ondemandupdate" ismachine="1">
Dec 07 13:13:19 localhost update_engine[1419]:     <os version="Chateau" platform="CoreOS" sp="3760.1.0_x86_64"></os>
Dec 07 13:13:19 localhost update_engine[1419]:     <app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" version="3760.1.0" track="2a27a073-f6b9-4cad-a4d8-c6d43a2508dd" bootid="{0e1b1df7-ec8d-4fb2-a9db-7a56a888ed42}" oem="qemu" oemversion="0.
0.2" alephversion="3760.1.0" machineid="9e4a256d90d74407b306fd3d94d2b28c" machinealias="" lang="en-US" board="amd64-usr" hardware_class="" delta_okay="false" >
Dec 07 13:13:19 localhost update_engine[1419]:         <event eventtype="14" eventresult="1"></event>
Dec 07 13:13:19 localhost update_engine[1419]:     </app>
Dec 07 13:13:19 localhost update_engine[1419]: </request>
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.152233  1419 libcurl_http_fetcher.cc:47] Starting/Resuming transfer
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.153443  1419 libcurl_http_fetcher.cc:174] Setting up curl options for HTTPS
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.153630  1419 libcurl_http_fetcher.cc:449] Setting up timeout source: 1 seconds.
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.388578  1419 libcurl_http_fetcher.cc:263] HTTP response code: 200
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.390797  1419 libcurl_http_fetcher.cc:320] Transfer completed (200), 229 bytes downloaded
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.390826  1419 omaha_request_action.cc:615] Omaha request response: <?xml version="1.0" encoding="UTF-8"?>
Dec 07 13:13:19 localhost update_engine[1419]: <response protocol="3.0" server="nebraska"><daystart elapsed_seconds="0"></daystart><app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" status="ok"><event status="ok"></event></app></response>
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.390991  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished OmahaRequestAction, starting FilesystemCopierAction
Dec 07 13:13:19 localhost update_engine[1419]: I1207 13:13:19.391024  1419 filesystem_copier_action.cc:294] Filesystem size: 1073741824
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.810441  1419 filesystem_copier_action.cc:274] Hash: ul/xQQMWN1J5Z2UioqtFEpKObCTB/IkR1cWuThwzN9A=
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.811090  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished FilesystemCopierAction, starting KernelVerifierAction
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.976043  1419 kernel_verifier_action.cc:74] Kernel size: 54755136
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.976070  1419 kernel_verifier_action.cc:75] Kernel hash: 9wLta+mSe9VDUWzDFDhX/T7nqgI/adaCOsPEef9ngJ8=
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.976078  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished KernelVerifierAction, starting PCRPolicyPostAction
Dec 07 13:13:22 localhost update_engine[1419]: I1207 13:13:22.976094  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished PCRPolicyPostAction, starting PostinstallRunnerAction
Dec 07 13:13:23 localhost update_engine[1419]: I1207 13:13:23.012722  1419 postinstall_runner_action.cc:40] Failed to mount install part as ext2. Trying ext3.
Dec 07 13:13:23 localhost update_engine[1419]: I1207 13:13:23.013631  1419 postinstall_runner_action.cc:48] Failed to mount install part as ext2/ext3. Trying btrfs.
Dec 07 13:13:23 localhost update_engine[1419]: I1207 13:13:23.451365  1419 subprocess.cc:46] Subprocess output:
Dec 07 13:13:23 localhost update_engine[1419]: [libprotobuf WARNING /build/amd64-usr/var/tmp/portage/dev-libs/protobuf-21.9/work/protobuf-21.9/src/google/protobuf/compiler/parser.cc:646] No syntax specified for the proto file: update_meta
data.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
Dec 07 13:13:23 localhost update_engine[1419]: [libprotobuf WARNING /build/amd64-usr/var/tmp/portage/dev-libs/protobuf-21.9/work/protobuf-21.9/src/google/protobuf/compiler/parser.cc:646] No syntax specified for the proto file: update_meta
data.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
Dec 07 13:13:23 localhost update_engine[1419]: The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
Dec 07 13:13:23 localhost update_engine[1419]: Signature error (Version: 1, Payload Hash: e8695ca8368680d98db0625f1df80b39f1bcd98aec16b1fb233f93124840273e, SIGDATA: o\027b\031\377Z\321\\\021l\013\210\034\224\320\226x\272\266+z\321\231\230
\315\305Bl\014\014\200\377Wg\252\314\243m\003\201\366[\017%%d\345\321ii\300\372=\r\367W3\372*\3770Y\366\246I=\221;\003\333\350\007\352\225\265k\344\221\306L\354\304q^d\277\013$}\245A\323\276U\033\355\2054*\243\025\266j\323\035\303\323g\34
3\377\257c\260\260\366\263\225\304\300Nas?\275\257\337\245\201\n\315\302\013L\261CM\014k/\317\006\030\321\016x\232x\2448\006-\224s\002\327\014g\'\275\213\331\373?-\214\274,\2110\344\2277\314Dm:KumA\253jTF\313\031G\301\223IS\255\0336Y^\350
\222\263i\220\031s\203\326\220\226z\262\330l\325\341A\316\257\355\326\301c/u\024\337m\350\226\035\345\243\031\221E0!\300\371L\355\311\"\2519\302\315\327\221\0177ZSJ\330,\273\354\277)
Dec 07 13:13:23 localhost update_engine[1419]: The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
Dec 07 13:13:23 localhost update_engine[1419]: Valid signature found (Version: 2, Payload Hash: e8695ca8368680d98db0625f1df80b39f1bcd98aec16b1fb233f93124840273e)
Dec 07 13:13:23 localhost update_engine[1419]: Success
Dec 07 13:13:23 localhost update_engine[1419]: Trying to place /var/lib/update_engine/oem-qemu-3794.0.0.raw on OEM partition
Dec 07 13:13:23 localhost update_engine[1419]: grep: /etc/flatcar/enabled-sysext.conf: No such file or directory
Dec 07 13:13:23 localhost update_engine[1419]: grep: /usr/share/flatcar/enabled-sysext.conf: No such file or directory
Dec 07 13:13:23 localhost update_engine[1419]: grep: /etc/flatcar/enabled-sysext.conf: No such file or directory
Dec 07 13:13:23 localhost update_engine[1419]:        start        size    part  contents
Dec 07 13:13:23 localhost update_engine[1419]:      2367488     2097152       4  Label: "USR-B"
Dec 07 13:13:23 localhost update_engine[1419]:                                   Type: Alias for coreos-rootfs
Dec 07 13:13:23 localhost update_engine[1419]:                                   UUID: E03DD35C-7C2D-4A47-B3FE-27F15780A57C
Dec 07 13:13:23 localhost update_engine[1419]:                                   Attr: priority=2 tries=1 successful=0
Dec 07 13:13:23 localhost update_engine[1419]: FLATCAR_RELEASE_VERSION=3794.0.0
Dec 07 13:13:23 localhost update_engine[1419]: FLATCAR_RELEASE_BOARD=amd64-usr
Dec 07 13:13:23 localhost update_engine[1419]: FLATCAR_RELEASE_APPID={e96281a6-d1af-4bde-9a0a-97b76e56dc57}
Dec 07 13:13:23 localhost update_engine[1419]: Setup USR-B (/dev/vda4) for next boot.
Dec 07 13:13:23 localhost update_engine[1419]: I1207 13:13:23.451395  1419 postinstall_runner_action.cc:104] Postinst command succeeded
Dec 07 13:13:23 localhost update_engine[1419]: I1207 13:13:23.455485  1419 action_processor.cc:82] ActionProcessor::ActionComplete: finished PostinstallRunnerAction, starting OmahaRequestAction

I continuously printed the contents of /var/lib/update_engine/prefs/full-response, and it's visible that the response containing the URLs is not present anymore at 13:13:23, when (as far as I understand) flatcar-postinst runs:

Thu Dec  7 13:12:54 UTC 2023
<?xml version="1.0" encoding="UTF-8"?>
<response protocol="3.0" server="nebraska"><daystart elapsed_seconds="0"></daystart><app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" status="ok"><ping status="ok"></ping><updatecheck status="ok"><urls><url codebase="http://10.0.2.2:8000/
3794.0.0/"></url></urls><manifest version="3794.0.0"><packages><package name="flatcar_production_update.gz" hash="32uV+75xVNSRqf4n/FEHGDWpk2E=" size="457850580" required="true"></package><package name="oem-qemu.gz" hash="fhKTxnZ7MY3mNlvOT
K31ZmXiXqI=" hash_sha256="82e3a680473ec23120adcc9982633d437522d2cb6da54cb2aabcc20de8036934" size="2341" required="false"></package><package name="flatcar-REDACTED-extension.gz" hash="kdhq+kGcmLs+fsy2WJwMrhp906s=" hash_sha256="347ffb3c9274c09
5cf3d43a8a3e7c279510ca9bbb18c89d4ab0e76e6b15d02ad" size="103354368" required="false"></package></packages><actions><action event="postinstall" sha256="eH3fPL2gdsF4g8kUiyKTs8+EgYjS4cEx4FgtPHqaG2g=" DisablePayloadBackoff="true"></action></a
ctions></manifest></updatecheck><event status="ok"></event></app></response>
[more lines]
[more lines]
[more lines]
Thu Dec  7 13:13:19 UTC 2023
<?xml version="1.0" encoding="UTF-8"?>
<response protocol="3.0" server="nebraska"><daystart elapsed_seconds="0"></daystart><app appid="1df13560-b68e-4f4b-9d52-a019d567b26e" status="ok"><event status="ok"></event></app></response>

We're very excited about this new feature! Any advice on what we are doing wrong (if it's not a bug) would be much appreciated.

[pothos]

Thanks for the detailed report, good point about the possible overwriting - maybe that needs to be filtered to only be done for responses to update requests.

(FYI: That doesn't happen with sudo flatcar-update because for simplicity it always returns the full response.)

[pothos]

PR in flatcar/update_engine#33 but it won't make it into the December release

[simoncampion]

Thank you for fixing this so quickly, @pothos 👍