update_engine failure to update when passing through a proxy

[elafontaine]

Description

The update process used to work through proxies with environment variable "ALL_PROXY". Today, we've realized that the update process have failed for all machines that are being the proxy.

This is what we used to inject into the update-engine.service through a dropins;

        - name: "proxy.conf"
          contents: |
            [Service]
            Environment="ALL_PROXY=<REDACTED>"

Impact

Machines behind a proxy aren't updating anymore.

Environment and steps to reproduce

1. Set-up: A Machine that does not have access to the internet directly
2. Task: Let locksmith try to update to the latest stable version
3. Error: observe the following message;

E0119 17:56:40.732053  1189 payload_processor.cc:487] prefs->GetInt64(kPrefsUpdateStateNextOperation, &next_operation) && next_operation != kUpdateStateOperationInvalid && next_operation > 0 failed.
E0119 17:58:48.966884  1189 postinstall_runner_action.cc:94] Postinst command failed with code: 1
E0119 17:58:48.974634  1189 update_attempter.cc:615] Update failed.
E0119 17:58:49.812515  1189 omaha_request_action.cc:626] HTTP reported success but Omaha reports an error.

Expected behavior

Get an updated machine on latest version of flatcar.

Additional information

We've tried to understand what happened, and we believe it's due to some change of the curl library being removed and another fetcher being used in the flatcar-postinst script. The reason we believe so is that we were able to test the download which seems to be successful, but something fail after. We believe it might be linked to the following change ;
flatcar/update_engine@43bb354

We'll keep on posting if we ever find the issue for sure. Please let us know if you require anything else from us.

[elafontaine]

I'm no rust expert, and a library may blind-sight me, but I'm not seeing anything in there to handle proxy cases. Am I wrong ?
https://github.com/flatcar/ue-rs/blob/trunk/src/bin/download_sysext.rs#L361

[pothos]

Correct, thanks for the bug report and testing Alpha! This case slipped through in the migration. Indeed we should add code to support the ALL_PROXY env var.

[elafontaine]

Thanks a lot for the confirmation @pothos , really appreciate the quickness !

May I suggest to look into supporting more than just the "ALL_PROXY" ? curl and wget have evolved to support many other variables (e.g. SOCKS_PROXY, NO_PROXY, HTTPS_PROXY, etc.). As curl was used previously, I believe the customer base would expect the behaviour to be maintained, so preferable to get curl's quirks than other binaries.

I'm just saying this to avoid other customers to have problem as well but having a different implementation then us. By the way, we did test most of those variables in trying to fix our issue. :)

[elafontaine]

By the way, we're not testing "alpha". We're on "stable". I don't think it changes anything, but I wanted the fact to be straight.

[pothos]

Ah, so for Stable we use curl in the postinst script action for downloading, not sure why this doesn't work then - maybe the env var is not passed through.

[pothos]

The ue-rs download_sysext helper also supports the needed proxy env vars when invoked directly.

[pothos]

Filed a PR and will test when the image build is ready: flatcar/update_engine#37 flatcar/scripts#1599

[pothos]

The fix is merged and will land in the next round of releases for all channels

[pothos]

FYI, the fix is required in the OS that does the update. If you don't have connectivity without the proxy, you can do an airgapped update with sudo flatcar-update -P … -E … where you pass the downloaded files from https://update.release.flatcar-linux.net/amd64-usr/

[pothos]

Report from the other thread:
I found one more problem on the way: Even with a newer flatcar-update (wget https://raw.githubusercontent.com/flatcar/init/flatcar-master/bin/flatcar-update) it won't work for old Stable or current LTS because the update_engine client doesn't hand the XML over.
A solution could be to jump through a release that doesn't require OEM payloads but will bring the proxy patch to unlock updating regularly. The next LTS bugfix release should work for that (Once available: sudo flatcar-update -V 3510.3.2).

[pothos]

We are thinking about adding workaround for the two issues of proxy settings and custom payload URLs to make it work with any old client. Will post when done.

[pothos]

Workaround is rolling out for Stable, Beta, Alpha - hope all is good now.

[elafontaine]

We were able to see the nodes update. Some will update tomorrow, so I'll close the ticket when we've done them all. Regards, Éric Lafontaine

…

On Wed, Feb 14, 2024 at 5:16 AM Kai Lüke ***@***.***> wrote: Workaround is rolling out for Stable, Beta, Alpha - hope all is good now. — Reply to this email directly, view it on GitHub <#1326 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXCYVDTQ4B43NWEAHRSBITYTSFJPAVCNFSM6AAAAABCCRKZU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGQ2TMMJWHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>