.timer doesn't fire

[utezduyar]

Submission type

• Bug report

systemd version the issue has been seen with

233

Used distribution

Yocto

log-rotate.timer never starts.

root@axis-accc8e5440dc:~# systemctl list-timers
NEXT                          LEFT          LAST                          PASSED    UNIT                         ACTIVATES
Tue 2017-08-29 13:35:00 CEST  23h left      Mon 2017-08-28 13:35:00 CEST  34min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Tue 2017-08-29 15:30:09 CEST  1 day 1h left Sun 2017-08-27 15:30:09 CEST  22h ago   scrubd.timer                 scrubd.service
n/a                           n/a           n/a                           n/a       log-rotate.timer             log-rotate.service

4 timers listed.
Pass --all to see loaded but inactive timers, too.

Status of the timer

root@axis-accc8e5440dc:~# systemctl status log-rotate.timer
● log-rotate.timer - log rotation timer
   Loaded: loaded (/lib/systemd/system/log-rotate.timer; static; vendor preset: enabled)
   Active: active (elapsed) since Mon 2017-08-21 13:20:28 CEST; 1 weeks 0 days ago
root@axis-accc8e5440dc:~# systemctl status log-rotate.service
● log-rotate.service - rotating old logs
   Loaded: loaded (/lib/systemd/system/log-rotate.service; static; vendor preset: enabled)
   Active: inactive (dead)

Timer file

root@axis-accc8e5440dc:~# systemctl cat log-rotate.timer
# /lib/systemd/system/log-rotate.timer
[Unit]
Description=log rotation timer

[Timer]
OnActiveSec=1m
OnUnitActiveSec=45m

root@axis-accc8e5440dc:~# systemctl show log-rotate.timer
Unit=log-rotate.service
NextElapseUSecRealtime=infinity
NextElapseUSecMonotonic=infinity
LastTriggerUSec=0
LastTriggerUSecMonotonic=0
Result=success
AccuracyUSec=1min
RandomizedDelayUSec=0
Persistent=no
WakeSystem=no
RemainAfterElapse=yes
Id=log-rotate.timer
Names=log-rotate.timer
Requires=sysinit.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=multi-user.target shutdown.target log-rotate.service timers.target
After=sysinit.target
Triggers=log-rotate.service
Description=log rotation timer
LoadState=loaded
ActiveState=active
SubState=elapsed
FragmentPath=/lib/systemd/system/log-rotate.timer
UnitFileState=static
UnitFilePreset=enabled
StateChangeTimestamp=Mon 2017-08-21 13:21:48 CEST
StateChangeTimestampMonotonic=129199383
InactiveExitTimestamp=Mon 2017-08-21 13:20:28 CEST
InactiveExitTimestampMonotonic=48002773
ActiveEnterTimestamp=Mon 2017-08-21 13:20:28 CEST
ActiveEnterTimestampMonotonic=48002773
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Mon 2017-08-21 13:20:28 CEST
ConditionTimestampMonotonic=48001480
AssertTimestamp=Mon 2017-08-21 13:20:28 CEST
AssertTimestampMonotonic=48001493
Transient=no
Perpetual=no
StartLimitIntervalSec=10000000
StartLimitBurst=5
StartLimitAction=none
InvocationID=03f1bf6b7e1944a786bbeb8da6f32203

[poettering]

how does your timer service look like? Is that a Type=oneshot service?

[utezduyar]

Yup!

[Unit]
Description=rotating old logs

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate -s /run/logrotate.status /etc/logrotate.conf

[poettering]

hmm, so OnUnitActiveSec= operates relative to a unit becoming active, but Type=oneshot service units actually never become active, unless you combine them with RemainAfterExit=, hence the confusion...

Type=oneshot units after all do stuff during their start-up and when that's complete they go down again, they never stay up continiously... Hence, combining Type=oneshot with OnUnitActiveSec= can't really work... This is a big underdocumented though

[boucman]

would OnUnitInactiveSec work in this case ?

[utezduyar]

How come it never even fired the first time due to OnActiveSec=1m? Also, the problem happens intermittently, that is very strange too.

[utezduyar]

ping

[RobinMcCorkell]

I'm seeing this issue sporadically with some of my timers as well, on systemd 238. Is there any information I can gather to help debug?

[nh2]

I observe this issue as well with systemd 237. My timer at some point just stopped activating the service.

I use OnUnitInactiveSec= and no oneshot.

As in the issue description, my timer now has n/a for PASSED, when before it was working fine and counting up the seconds there:

# systemctl list-timers
NEXT                         LEFT    LAST                         PASSED  UNIT                         ACTIVATES
Sun 2018-11-04 00:32:30 UTC  9h left Sat 2018-11-03 00:32:30 UTC  14h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a                          n/a     n/a                          n/a     cdn-rsync.timer              cdn-rsync.service

Full outputs and unit files: https://gist.github.com/nh2/623947890cf98b9df87fbd652f76b70b

[nh2]

A similar report by somebody else I found: https://serverfault.com/questions/775246/systemd-timer-not-starting-its-service-unit

It suggests that adding an explicit Requires= from the .timer to the .service helped.

[nh2]

I tried stopping and starting the .timer, but to no effect.

Then I tried the above linked suggestion

It suggests that adding an explicit Requires= from the .timer to the .service helped.

Adding an explicit Requires=my.service to my.timer indeed resulted in the .service being started again.

Oddly, the systemctl list-timers output remains unchanged.

Then, after I killed the process controlled by the .service's ExecStart using kill, PASSED switched from being n/a to counting up seconds again, and LAST is set.

[m-khvoinitsky]

I can confirm the problem with systemd 219 on CentOS 7.

I'm using timer with Type=oneshot service which has:

OnActiveSec=<some_time>
OnUnitInactiveSec=<some_time>

And sometimes, after reboot, timer won't start firing (yes, it's enabled). Adding OnBootSec didn't helped. If I manually start the service with systemctl start <service>.service, OnUnitInactiveSec part of the timer starts working. So, it seems to be the problem with OnBootSec and OnActiveSec which are sometimes unreliable (in particular, after reboot).

Adding an explicit Requires=my.service to my.timer indeed resulted in the .service being started again.

I'd suggest adding Wants=<service>.service instead as a workaround because this is more close equivalent of OnActiveSec=0. The problem with Requires is that if your service unit fails on the first run, timer unit may not be activated at all which is probably not what you want (see man 5 systemd.unit).

[m-khvoinitsky]

Should be the label needs-reporter-feedback still there?

[nmahaba]

I am also seeing the same issue where occasionally my log-rotate timers are in active (elapsed) state, my service is a onshot. I am using an Yocto distribution with systemd version 232.

Timer:

[Timer]
OnBootSec=10sec
OnUnitActiveSec=2m

[Install]
WantedBy=timers.target

Some questions,

1. There are suggestions to use Wants=<service>.service and Requires=<service>.service. But I can't explain how they are supposed to solve the problem.
2. I can't reproduce the issue easily. The closest I could get to the problem reproduction was to remove the OnBootSec section which prevents the timer from firing. Then I would see the issue where log-rotate.timer is in active (elapsed) state. Then if I manually start log-rotate.service, the timer starts to work as expected. So I am guessing the issue with the initial firing of the timer. Will this commit help ?
3. If the issue is with initial timer firing, can the below help (along with making the service RemainAfterExit=true) ?

Timer:

[Timer]
OnActiveSec=2m
OnUnitActiveSec=2m

[Install]
WantedBy=timers.target

OnActiveSec ensures that the timer fires in the beginning and OnUnitActiveSec ensures that the timer re-arms.

[dresken]

https://bugzilla.redhat.com/show_bug.cgi?id=1764908

[nmahaba]

@dresken I can't reproduce the issue as mentioned in issue

[dresken]

@nmahaba Did you reboot first?

[nmahaba]

@dresken Yes, I rebooted the system and performed multiple systemctl daemon-reload.
Its difficult to reproduce the issue, in all instances it was just after a reboot.

[nmahaba]

@poettering Do you have any thoughts on this ?

[m-khvoinitsky]

@nmahaba,

There are suggestions to use Wants=.service and Requires=.service. But I can't explain how they are supposed to solve the problem.

As you later wrote, the issue is with initial timer firing. So,

  Wants=

...
Units listed in this option will be started if the configuring unit is. However, if the listed units fail to start or cannot be added to the transaction, this has no impact on the validity of the transaction as a whole, and this unit will still be started. This is the recommended way to hook start-up of one unit to the start-up of another unit.

Which means that your .service unit will be started together with .timer which solves the problem of initial firing.

[joshuamiller01]

The workaround of using Requires for the service isn't great; in my testing (on 244) that triggers the service to fire upon initial setup of the timer (e.g. systemctl start <thing.timer>) regardless of when it's scheduled to actually run. For timers which run every few minutes it probably doesn't matter, but for things which really need to happen at the declared time, that could be really bad.

[dtrauma]

I'm a bit confused. What exactly is the recommended way to run a oneshot service, or if that is difficult, restart a normal service n days after the last successful execution, if

• it shouldn't run before n days
  • even if the machine restarted in between
  • even if the last successful run was manual, not through the timer
• and it should catch up if the machine was down and we are now past the n days mark

I can build a solution with manual timestamping and cron, of course, but I was under the impression this should be easier with systemd. All of these requirements are real, BTW, for letsencrypt renewal, which you don't want to run too often, you need to run regularly, and sometimes you need to run manually, which should push back the next automatic run, which excludes just using oncalendar.

[dresken]

CLOSED WONTFIX for RHEL7 - hopefully gets addressed in RHEL8 - https://bugzilla.redhat.com/show_bug.cgi?id=1764908

[karx1]

Hi all, what solved for me was, I had to start the .service myself one time, then after that the timer started working as expected.

[dresken]

Hi all, what solved for me was, I had to start the .service myself one time, then after that the timer started working as expected.

What about the next time you reboot? This sounds like a work around and not a solution

[karx1]

For that, you can set `Persistent=true`. Also, since this is a server, I probably will reboot very infrequently, so I can start it manually the few times I do reboot.

[dresken]

For that, you can set Persistent=true. Also, since this is a server, I probably will reboot very infrequently, so I can start it manually the few times I do reboot.

Persistent=true does not address this issue - as it only works with OnCalendar (see https://www.freedesktop.org/software/systemd/man/systemd.timer.html)

I am glad you have the ability to be present and manually intervene with your server every time it is rebooted. The sheer scale of the operations we work with makes this impossible - so forgive us that we would like this bug actually addressed in an appropriate fashion.

[nh2]

hopefully gets addressed in RHEL8 - https://bugzilla.redhat.com/show_bug.cgi?id=1764908

@dresken Closed as fixed in February, with the link to the fix being redhat-plumbers/systemd-rhel7#130.

But that appears to be a backport of #10778 which was merged to systemd in 2018.

[tmccombs]

I'm seeing something that may be the same issue, but with an OnCalendar timer. it has

OnCalendar=daily
AccuracySec=12h
Persistent=true

After booting up with a stamp file that is several days old (actually from a disk image, not the same instance), systemctl list-timers lists the last run time, but gives n/a for the next trigger time. systemctl status reports it is active (elapsed). I can't find any documentation on what exactly the (elapsed) means. The associated service is of type oneshot.

This is on systemd 245.

[Alexander-Shukaev]

Can this be closed as per #10778?

[brotaxt]

Is there any news? I'm using systemd-239-58 and still experiencing these problems with oneshot services. @poettering

cat /etc/systemd/system/callback.service

[Unit]
Description=Provisioning callback 
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=SCRIPT_HERE
ExecStartPost=/usr/bin/systemctl disable callback

[Install]
WantedBy=multi-user.target

cat /etc/systemd/system/callback.timer

[Unit]
Description=Schedule callback.service every 24 hours
RefuseManualStart=no
RefuseManualStop=no

[Timer]
Persistent=true
OnUnitInactiveSec=24h
RandomizedDelaySec=240m
OnBootSec=1h

Unit=callback.service

[Install]
WantedBy=timers.target

systemctl list-timers |grep callback

n/a                           n/a           Wed 2023-04-19 06:14:50 CEST  5 days ago   callback.timer       callback.service

systemctl status callback

● callback.service - Provisioning callback
   Loaded: loaded (/etc/systemd/system/callback.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

systemctl status callback.timer

● callback.timer - Schedule callback.service every 24 hours
   Loaded: loaded (/etc/systemd/system/callback.timer; enabled; vendor preset: disabled)
   Active: active (elapsed) since Mon 2023-04-24 12:20:42 CEST; 36min ago
  Trigger: n/a

Apr 24 12:20:42 hostname systemd[1]: Started Schedule callback.service every 24 hours.