st2 cron timer was missed

[antonbezkrovny]

Have problem:

2022-12-02 09:00:01,696 140410779819872 WARNING base [-] Run time of job "St2Timer._emit_trigger_instance (trigger: cron[day_of_week='*', hour='9', minute='0', second='0'], next run at: 2022-12-03 09:00:00 MSK)" was missed by 0:00:01.696190

How can im configure APScheduler misfire_grace_time ?

[antonbezkrovny]

up

[nzlosh]

Can you provide the following information about your set up with the below command:

• St2 version
• OS distro/version
• Rabbitmq version
• MongoDB version

echo StackStorm Version; \
st2 --version; \
echo Distribution; \
egrep 'NAME|ID|VER' /etc/os-release; \
echo RabbitMQ; \
rabbitmqctl --version; \
echo MongoDB; \
mongo --version

What is the StackStorm execution environment. E.g. bare-metal, VM, container, kubernetes (on-premise / cloud)?

[antonbezkrovny]

Ola!

StackStorm Version
st2 3.6.0, on Python 3.8.10
Distribution
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
RabbitMQ
3.9.15
MongoDB
MongoDB shell version v4.4.13
Build Info: {
"version": "4.4.13",
"gitVersion": "df25c71b8674a78e17468f48bcda5285decb9246",
"openSSLVersion": "OpenSSL 1.1.1f 31 Mar 2020",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "ubuntu2004",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}

[lemeur]

I've opened a similar discussion 11 days ago: #5824

the pb for me is that the VM Host may be overloaded sometimes, and thus misses clock ticks.
The current implementation of the CrontTimer in stackstorm uses APScheduler, which has a misfire_grace_time parameter. But I can't see where to pass it to the Stackstorm rule.

[jonathangull]

HI Team I am also facing this issue . Here are my environment details

st2 3.7.0, on Python 3.6.9
Distribution
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
MongoDB shell version v4.0.28
git version: af1a9dc12adcfa83cc19571cb3faba26eeddac92
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1804
distarch: x86_64
target_arch: x86_64

[lemeur]

@jonathangull,
Are you running st2 in a virtual machine ?

If so, can you check on your hypervisor, if the host was busy at that time (CPU, I/O) ?

[antonbezkrovny]

up, problem appears bot if the host was busy or not

[chain312]

@jonathangull, Are you running st2 in a virtual machine ?

If so, can you check on your hypervisor, if the host was busy at that time (CPU, I/O) ?

@lemeur You can look at my pr and add a parameter to the code#5780