Virtualbox time drift due to missing guest additions

[tetsuo55]

Hardware Environment

• Raspberry Pi [1/2/3/4]
• ODROID [C2/C4/N2(+)/XU4]
• ASUS Tinker [S]
• Generic x86-64 (like Intel NUC)
• OVA (Open Virtualization Appliance, on Intel NUC or any other hardware, please add the Hypervisor you are using)
• Khadas VIM3

I'm using virtualbox on windows 10

Home Assistant OS release:

• Fresh installation of release x.y
• Updated from version 6.1
• Additional information (if accessible):

System Health

version	core-2021.9.2
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.9.6
os_name	Linux
os_version	5.10.61
arch	x86_64
timezone	Europe/Amsterdam

Home Assistant Community Store

GitHub API	ok
Github API Calls Remaining	5000
Installed Version	1.15.2
Stage	running
Available Repositories	881
Installed Repositories	13

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 6.3
update_channel	stable
supervisor_version	supervisor-2021.09.0
docker_version	20.10.7
disk_total	30.8 GB
disk_used	14.9 GB
healthy	true
supported	true
board	ova
supervisor_api	ok
version_api	ok
installed_addons	Samba share (9.5.1), Terminal & SSH (9.1.3), File editor (5.3.3), ESPHome (2021.8.2), InfluxDB (4.2.1), Grafana (7.2.0), Mosquitto broker (6.0.1), Check Home Assistant configuration (3.8.0), Zigbee2mqtt (1.21.1-1), Home Assistant Google Drive Backup (0.104.3), Glances (0.13.0), HassOS SSH port 22222 Configurator (0.8), SQLite Web (3.1.0)

Lovelace

dashboards	1
resources	1
views	7
mode	storage

**Supervisor logs:**

Journal logs:

Kernel logs:

Description of problem:

The issue is that the time drifts which breaks time based automations in they they fire minutes to hours to days later then planned because the clock is lagging.

The cause is that virtualbox requires guest additions to be able to communicate with the host and those aren't included.

When guest additions are installed virtualbox will sync time with the guest every 10 seconds to correct for the drift that occurs due to the guest getting paused by things like host power saving, being too busy to l
Provide the needed cycles on time or the guest being paused.

I found an older repo that has a virtualbox guest additions package for buildroot:

https://github.com/xenogenesi/vbox-buildroot-vagrant/tree/master/package/vbox-ga

Hoping this is enough for this issue to be corrected

[tetsuo55]

i found this other repo that has automated updates and is way more recent but its in ruby

https://github.com/dotless-de/vagrant-vbguest

[agners]

What host operating system are you using? What is the setting of "Paravirtualization Interface" in you virtual machine? Does your host has hardware acceleration and is it enabled?

I just tried to reproduce this here, and my virtual machine clock is second accurate even after 6h+. I wonder if this maybe only is a problem in certain environments?

The cause is that virtualbox requires guest additions to be able to communicate with the host and those aren't included.

Linux recently added some "native" VirtualBox drivers: https://cateee.net/lkddb/web-lkddb/VBOXGUEST.html. But I don't think that this is enough to get RTC synchronization going.

[tetsuo55]

Host is windows 10 x64
interface is set to standard with nested pages checked
hardware acceleration is enabled but my cpu doesnt support SLAT so hyperv does not work

that kernel driver is the module that guest additions would install but its only a small part of the whole.
https://forums.whonix.org/t/virtualbox-guest-additions-being-upstreamed-to-linux-kernel-org/8785/2

we need the full guest additions to make time work for everyone.

edit:
i should clarify that any time command you run in template tester will display the correct time, its time based automations that fail

set an automation that sends you a message with that set time and compare to the delivery timestamp
screenshot of broken messages

correct time being shown in home assistant interface

[tetsuo55]

@agners is there anything I can do to help? I really need time based automations

[agners]

correct time being shown in home assistant interface

@tetsuo55 oh that is super weird. I fear that syncing time will then not even help...

Can you add the current time to the message (using {{ now() }})? I wonder if at the time the message get sent (from the automation) the time is really off or just the automation triggered at a completely wrong time.

[tetsuo55]

I have appended all the messages with system time and created a new one as a quick test

[agners]

Can you attach the trace of this automation?

[tetsuo55]

I think i got it figured out.
Without the guest additions the guest is unable to synchronize the CPU clocks.
The CPU speeds are wrong inside hassos causing the internal clocks to tick at the wrong speed, while time synchronisation keeps correcting the displayed time, hiding the actual drift.

As for the trace I'll have to wait for the next one because i rebooted the server to figure out this issue.

Which brings us back to full guest additions, this should provide the bits needed for cpu clock synching, although i can't find the document i read that in again

[tetsuo55]

I have found something to test, maybe the paravirtualization has to be set to kvm

[tetsuo55]

Unfortunately the kvm mode didn't help, maybe made it even worse.

id: '1630511868414'
alias: time trigger test a
description: ''
trigger:
  - platform: time
    at: '04:00'
condition: []
action:
  - service: notify.telegram
    data:
      message: this is the 04:00 test {{ now() }}
mode: single

this:
  entity_id: automation.time_trigger_test_a
  state: 'on'
  attributes:
    last_triggered: '2021-09-19T06:06:17.420544+00:00'
    mode: single
    current: 0
    id: '1630511868414'
    friendly_name: time trigger test a
  last_changed: '2021-09-20T16:36:55.557737+00:00'
  last_updated: '2021-09-20T16:36:55.557737+00:00'
  context:
    id: e0ccf26c8b7b63d32f91141dc1f59aa0
    parent_id: null
    user_id: null
trigger:
  id: '0'
  idx: '0'
  platform: time
  now: '2021-09-21T08:20:55.280725+02:00'
  description: time
  entity_id: null

[guignol8220]

Please add me to the list of people having the same issue. Any time triggered automation I have in NodeRed and HA are not anymore reliable.

Problem appeared (for me) around 1-2 month. After spending quit some time investigating by downgrading different part of my systems and trying fresh image/install, I can now says that the problem is solely related to release 6.3. Everything work fine with a clean 6.2 install, but I get unsync time with clean install of 6.3.

extra info: I'm running HA OS inside a VirtualBox, which run on Ubuntu

[agners]

@tetsuo55 from what I can tell in your system, the system time is correct, its just the automation is triggered too late: E.g. You received the message at 8:20, and the message got generated at 8:20:55.289736. The question is, why did the system did not wake up at 8:00 and send it back then?

@tetsuo55 can you confirm @guignol8220's observation? E.g. does it help for you to roll back to 6.2? (You can use ha os update --version 6.2)

[guignol8220]

for info, such a simple automation is enough to demonstrate the error. I use a basic time_patern that trigger every round 10 min or hours. If I run the same automation inside a fresh 6.2 or 6.1, it run exactly at the expected time, but if I run the same automation inside a fresh 6.3 or 6.4, the trigger time is always later by a few second or up to 40-50 min some times.
what is even more confusing is that the time in the logs is actually correct (by that I mean that when it says that the action was triggered at 19:40:14, this is correct time). Seams like the clock used for trigger action use a different clock than the one used for tagging the logs.

here is supposed to trigger at every round 10min (19:40:00 in this case), but got triggered 14 second to late

here is supposed to trigger every round hours (10:00:00) but got tirgger 12 min later...

[tetsuo55]

@tetsuo55 from what I can tell in your system, the system time is correct, its just the automation is triggered too late: E.g. You received the message at 8:20, and the message got generated at 8:20:55.289736. The question is, why did the system did not wake up at 8:00 and send it back then?

@tetsuo55 can you confirm @guignol8220's observation? E.g. does it help for you to roll back to 6.2? (You can use ha os update --version 6.2)

I'm pretty sure i had the issue on 6.1, but i will test this anyway

[tetsuo55]

Same issue on 6.2

this is the 16:00 test system time is 2021-09-23 16:11:55.672817+02:00 the trigger time is2021-09-23 16:11:55.666064+02:00

@agners , can you make a test build with that buildroot addon thing? Or maybe a way to install the virtual box additions?
I've read dozens of pages where this was resolved by installing the additions (we currently only have 1/2 of the additions in form of the kernel parts)

[tetsuo55]

I found another package, for fedora Linux which is fully up to date.
https://src.fedoraproject.org/rpms/virtualbox-guest-additions/blob/f33/f/virtualbox-guest-additions.spec

[butako]

I faced the same problem. Running HA OS inside VirtualBox upon Ubuntu, automations triggered by time would drift by many minutes. This solved it for me: https://community.home-assistant.io/t/timebased-automations-completely-out-of-sync/302083/5 I switched “Paravirtualization Interface” to None and the time automations have been accurate since.

[guignol8220]

Thanks for sharing @butako!
I’m running the almost same setup as you (differences is it is running on a macmini late 2009).
I did reverted 1 week ago my “production” HA to 6.2 and all time based automation are working as it should.
but reading your comment, I dared yesterday pressing the update button to 6.4, and desactivated in same time in VirtalBox what you point out, and so fare no issues!!

[tetsuo55]

I will disable paravirtualization for now too then.
But it's not the proper solution to this issue as we're giving up a lot of performance

[agners]

Agreed disabling paravirtualization is not ideal as a long term solution.

What is a bit puzzling to me is that we supported Virtual Box since quite some time, but it only seems to crop up now? Which begs the question, did something change which makes it show up today?

I'll try porting the VirtualBox guest addition to Buildroot/HAOS, but its not on-top of my todo list right now.

[tetsuo55]

Yeah that is strange, I'm a new user of home assistant and this was my first attempt at using time of day based automations so i can't speak for how things were before

[ioctl2]

The incomplete virtualbox support also has the effect that detailed memory utilization is not available host-side:

[Zi0mele]

Did anyone find a solution to this problem?
I'm experiencing it with HA OS 9.5, core 2023.4.2, on a MacOS Monterey (Installed with OCLP on a 2009 macbook 5,2). It did not present right away after the installation, It started around the daylight saving time change. Disabling the paravirtualization worked, but it spiked the cpu usage in HA (from 1-2% to 12-13%) and now I have some temperature issues with the macbook.
I've tried setting the paravirtualization to "minimal" instead of "none"; it did not affect the time, but it did not affact the cpu usage eaither.
Suggestions?

EDIT: I'm running VirtualBox 6.1.44. I can't get MacOS to prompt me to allow the kernel modification when installing VirtualBox 7, So I can't upgrade to that, for the moment.

[tetsuo55]

The only solution is for the guess additions to be added to ha os