Timer's that never start

[rsyring]

I ran into problems previously with recurring timers that use on_active_sec and on_inactive_sec and never start. I thought I solved that with: 554b402

However, I recently had a sysdi configured timer that didn't have that change applied. After the server rebooted, the unit stopped being triggered. I updated the sysdi version and re-synced the sysdi managed units.

I assumed that would retroactively kick off the timer similar to how Persistent works for calendar timers but it didn't. The monotonic timers were never triggered:

$ systemctl --user show sysdi-restic-backup.timer -p TimersMonotonic,NextElapseUSecMonotonic
TimersMonotonic={ OnUnitInactiveUSec=4h ; next_elapse=0 }
TimersMonotonic={ OnActiveUSec=1s ; next_elapse=0 }
NextElapseUSecMonotonic=infinity

I then did two things:

• Changed the timer to use OnActiveSec instead of OnStartupSec
• Stopped the timer and started it again

That finally triggered the timers to start working:

systemctl --user show sysdi-restic-backup.timer -p TimersMonotonic,NextElapseUSecMonotonic
TimersMonotonic={ OnUnitInactiveUSec=4h ; next_elapse=6d 17h 39min 13.858895s }
TimersMonotonic={ OnActiveUSec=1s ; next_elapse=6d 13h 38min 30.656349s }
NextElapseUSecMonotonic=6d 17h 39min 13.858895s

I'm going to change sysdi's defaults to use OnActiveSec instead of OnStartupSec. For our intended use case of "just run the task repeatedly", this is superior b/c it accounts for a timer created/started the very first time and doesn't rely on system-start/user-login event (which will already have taken place when the timer is first created).

It's possible, however, that there is still a potential case where we have an "unstarting" timer. Possible, b/c I don't really understand all the failure states around the timer and/or service unit not working as expected and what that might do to the timer's next run. If the timer finds itself in an "unstarting" state, someone might try to fix this using sysdi sync.

BUT, in that case, I don't know if systemctl --user enable --now would cause the timer to reset itself. I think, because it's technically started/activated already, using --now won't restart it. In that case, OnActiveSec isn't going to get activated and the configuration updates won't make any change to the timer starting.

In that case, it's possible that what we should really be doing is a separate systemctl --user enable and systemctl --user restart on the timer. But this has the disadvantage of kicking off the service unit regardless of when it has last ran which could be undesirable in some instances.

[rsyring]

The key difference between "Active" and "Started" for timers: "Active" means the timer is armed, meaning it's loaded and waiting to trigger events. "Started" refers to when the timer unit itself is initially started and armed. The timer remains active and does not restart with each trigger; instead, it just fires and remains in the active state. Starting the timer arm triggers the associated service unit, but the timer unit itself stays active without needing to be repeatedly started.

In systemd, a timer unit's "Active" state means it's loaded and awaiting triggers, similar to being armed. The "Started" state refers to the initial start of the timer, which occurs after the unit is activated. Even though the timer triggers the associated service unit on schedule, the timer itself doesn't restart each time; it remains in the "active" state, awaiting its next trigger. Thus, "Active" shows a persistent armed state, and "Started" denotes the one-time initiation of the timer itself.

In systemd, a timer's "Active" state shows that the timer is loaded and awaiting its next trigger. Meanwhile, "Started" reflects when the timer unit itself was initiated, either manually or during system boot. The timer doesn’t restart when it fires; instead, it stays active and triggers the service unit (e.g., foo.service). Therefore, "Active" is the timer being armed and ready, while "Started" is the initial activation event. The service unit (not the timer) starts when the timer elapses.

When I pushed this change out to the server, I noticed that the service unit was started even though the timer was already activated/armed. Therefore, we know that sysdi's sync, which runs systemctl --user enable --now, is acting like a re-start and, with OnActiveSec, results in the service unit being activated with every sync.

I think we need to be smarter about this an only start/trigger the timer unit when needed. We can read it's status and know if it needs to be enabled and/or do an initial start (or not).