Hangs on "Waiting for time sync (systemd-timesyncd.service) to complete." during install

[peracchi]

As the title says, archinstall hangs on "Waiting for time sync (systemd-timesyncd.service) to complete." during install IF NETWORK DO NOT ALLOW NTP TRAFFIC TO GO OUT.

It do not timeout as expected.

The only way I found to do my installation was to open another SSH session and:

systemctl stop systemd-timesyncd
nano /etc/systemd/timesyncd.conf
<edit file>
NTP=<my internal NTP server>
<save file>
systemctl start systemd-timesyncd

Then, installation proceeds.

[Torxed]

Can you confirm that this is happening because network configuration is done manually (like wifi, slow DHCP or even static IP)?

[peracchi]

Yes, at work we have a very restrictive firewall, MITM, self-signed certificates and other annoyances.

One or more of these factor prevent NTP communication with outside servers and things only work after I stop NTP service, point it to internal NTP server and then restart NTP service.

[Torxed]

I figured something like this was going on ^^
Without even checking, I'm pretty sure this happens:

1. Boot
2. timesync service tries to sync
3. It fails and restart mechanics sucks
4. You configure network
5. Run archinstall
6. It waits for the next timesync to trigger
7. You SSH because the sync never happens
8. You fix it

This is an issue for manual installs to, and I think instead of fixing it here, I'll discuss with some other upstream devs if we can't solve this permanently for all use cases.

[peracchi]

I'm installing Arch on a VM on Proxmox using the ISO archlinux-2023.09.01-x86_64.iso

There are several other steps I need to do before starting the actual installation.

These steps are:

• boot ISO on Proxmox VM
• change root password to SCP/SSH
• SCP internal self-signed certificates
• trust anchor <certificates>
• pacman-key --init && pacman-key --populate archlinux && pacman -Syy
• pacman -S --noconfirm archinstall
• only then archinstall

[Torxed]

Going back and forth on this, since it's network related on user side - there's really nothing we can do to fix this. We could however add a --skip-ntp to the command line.

It'd be up to the user to verify that time is correct, or as OP mentioned edit the NTP settings and restart the timesyncd-service.

Would this be a viable option?

[peracchi]

We could however add a --skip-ntp to the command line.

This would be sufficient.

But the ideal would be to detect the problem with NTP (by timeout, perhaps?), kill the NTP process and proceed with the installation.

[alan-barzilay]

But the ideal would be to detect the problem with NTP (by timeout, perhaps?), kill the NTP process and proceed with the installation.

Some sort of feedback to the user would also be good. When I got this issue I simply waited to see if it it would magically fix itself haha A disclaimer with possible reasons for hanging would give the user enough information to possibly do something about it himself

[Torxed]

Those are valid suggestions,

kill the NTP process and proceed with the installation.

This is usually not enough tho, as it also requires manual configuration first.

I'll make sure to add the parameter before release and perhaps a note in the while loop waiting for the time sync referencing the docs relating to the issue:

(screenshot from WIP PR #1967)

[peracchi]

Thank you for your work/effort.

[alan-barzilay]

is the ntp process important for any other step in the install? because I have an ansible playbook that later on configures ntp among other things, if I used the proposed '--skip-ntp' flag would I need to do anything else that was skipped/affected during archinstall?

[Torxed]

Yes, the main reason for making sure NTP exists before archinstall does anything is because numerous HTTPS calls, one obvious and big one is the pacman call which will contact HTTPS mirrors at some point. The less obvious is the archlinux-keyring update service that happens on the ISO in the background.

So NTP is a cheap reliable:tm: way of ensuring that such things work.

But as long as you ensure time is accurate to within a couple of seconds from the mirrors etc, any method of setting the correct time should work in theory.

[alan-barzilay]

I see, thank you for the clarification. After a quick search, ntp seems to be ill suited for VMs if they are not constantly running, virtualbox even marked it as wontfix. Letting the user skip ntp and synchronize time on his owns seems more and more like a solid solution (or add something like a vm-mode to synchronize against host time, but no idea how to do that consistently from inside guest for all virtualization providers)

[Torxed]

I see, thank you for the clarification. After a quick search, ntp seems to be ill suited for VMs if they are not constantly running, virtualbox even marked it as wontfix. Letting the user skip ntp and synchronize time on his owns seems more and more like a solid solution (or add something like a vm-mode to synchronize against host time, but no idea how to do that consistently from inside guest for all virtualization providers)

Correct. The issue is that most often, you'll need to configure for instance -rtc needs to be configured before launching a virtual machine and there's not much we can do after-the-fact. And as you eluded to, some virtualization platforms have issues retaining time.
So yea, a parameter might be the only viable option.

I just.. *grinds my teeth*.. foresee the following issue being that the keyring is out of date, followed by pacman calls failing "arbitrarily", followed by some other network related service breaking hehe. Either way, and we'll take it as it comes. The docs now reference those issues that we can foresee.

[Torxed]

For what it's worth, a test page is published for the new docs and it references this issue and it's workaround: https://archlinux.github.io/archinstall/help/known_issues.html#waiting-for-time-sync-2144

Together with PR #2225 - adding --skip-ntp I think we're done :)
I'll close this once the PR is in and the docs is official.

[Torxed]

parameter is now in master and should make it to the next release and ISO :)

[svartkanin]

@Torxed might be worth putting that into the FAQ of the readme as well

[Torxed]

Good idea, we should also swap the link to the documentation to https://archinstall.archlinux.page/!

PR: #2273