New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rancher crontab scheduling doesn't work. It doesn't pay attention to cron rule and executes restarts container every hour. #8979

Closed
ketchoop opened this Issue Jun 6, 2017 · 10 comments

Comments

@ketchoop

ketchoop commented Jun 6, 2017

Rancher versions:
rancher/server: 1.5.10
rancher/agent: 1.2.2
rancher/crontab: 0.1.0-rancher1

Infrastructure Stack versions:
network-services: net-manager 0.6.6

Docker version: (docker version,docker info preferred)

17.04.0-ce

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
RancherOS v1.0.1 kernel 4.9.24

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Parallels VM

Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)

Rancher HA external DB galera cluster

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)

cattle

**Problem: **

I have elasticsearch curator container that has ```cron.schedule=0 0 0 * * *" label and ran with started-once, for example. And logs says that this container restarts every hour. And this is the same problem with the other containers.

@ketchoop

This comment has been minimized.

Show comment
Hide comment
@ketchoop

ketchoop Jun 7, 2017

Maybe it's not cron trouble, but "started-once" label. But it's strange, that containers automatically runs. Can somebody comment this behaviour, please, because there is no info in docs about this.

ketchoop commented Jun 7, 2017

Maybe it's not cron trouble, but "started-once" label. But it's strange, that containers automatically runs. Can somebody comment this behaviour, please, because there is no info in docs about this.

@ketchoop

This comment has been minimized.

Show comment
Hide comment
@ketchoop

ketchoop Jun 8, 2017

Guys, please, say something about it. Because it's not obvious and I think, wrong behaviour, that started once containers restarts, aproximately, every hour. It makes cron container and started-once useless.

ketchoop commented Jun 8, 2017

Guys, please, say something about it. Because it's not obvious and I think, wrong behaviour, that started once containers restarts, aproximately, every hour. It makes cron container and started-once useless.

@aemneina

This comment has been minimized.

Show comment
Hide comment
@aemneina

aemneina Jun 9, 2017

Can you restart the crontab service to see if it picks up the changes on the container. I think the crontab service isnt automatically scanning and detecting new containers.

aemneina commented Jun 9, 2017

Can you restart the crontab service to see if it picks up the changes on the container. I think the crontab service isnt automatically scanning and detecting new containers.

@ketchoop

This comment has been minimized.

Show comment
Hide comment
@ketchoop

ketchoop Jun 9, 2017

@aemneina didn't restart crontab, but upgraded it to avoid side effects . But, It still happens. My guess, that it's because of started-once label. but I'm not really sure about it. Let me check it with another container.

ketchoop commented Jun 9, 2017

@aemneina didn't restart crontab, but upgraded it to avoid side effects . But, It still happens. My guess, that it's because of started-once label. but I'm not really sure about it. Let me check it with another container.

@ketchoop

This comment has been minimized.

Show comment
Hide comment
@ketchoop

ketchoop Jun 9, 2017

@aemneina Yep, it's not connected with cron. Rancher just restarts started-once containers.

P.S. Maybe it'll be helpful: the last time, when I saw restart it showed me deadlock in database.

UPD: I think I found the reason of this behaviour and maybe it's not connected with crontab and started-once.

ketchoop commented Jun 9, 2017

@aemneina Yep, it's not connected with cron. Rancher just restarts started-once containers.

P.S. Maybe it'll be helpful: the last time, when I saw restart it showed me deadlock in database.

UPD: I think I found the reason of this behaviour and maybe it's not connected with crontab and started-once.

@ketchoop

This comment has been minimized.

Show comment
Hide comment
@ketchoop

ketchoop Jun 9, 2017

@aemneina thank you for response! It helped me to find bug on my side, that was connected with hourly cleanup. And crontab seems to be ok and it changes schedule properly.

ketchoop commented Jun 9, 2017

@aemneina thank you for response! It helped me to find bug on my side, that was connected with hourly cleanup. And crontab seems to be ok and it changes schedule properly.

@ketchoop ketchoop closed this Jun 9, 2017

@aemneina

This comment has been minimized.

Show comment
Hide comment
@aemneina

aemneina Jun 9, 2017

@ket4yii I'm going to reopen, i think the catalog item does have a bug where new containers arent detected. That way we can eventually get a fix in for that.

aemneina commented Jun 9, 2017

@ket4yii I'm going to reopen, i think the catalog item does have a bug where new containers arent detected. That way we can eventually get a fix in for that.

@cloudnautique

This comment has been minimized.

Show comment
Hide comment
@cloudnautique

cloudnautique Jun 20, 2017

Member

This is a UX issue. The Docker event to add the job is not logged on create. The restart logs all of the containers it finds on startup. This is confusing behavior and can/will be addressed.

Member

cloudnautique commented Jun 20, 2017

This is a UX issue. The Docker event to add the job is not logged on create. The restart logs all of the containers it finds on startup. This is confusing behavior and can/will be addressed.

@galal-hussein

This comment has been minimized.

Show comment
Hide comment
@galal-hussein

galal-hussein Jul 11, 2017

Contributor

@soumyalj Can you test with v1.6-development branch

Contributor

galal-hussein commented Jul 11, 2017

@soumyalj Can you test with v1.6-development branch

@soumyalj

This comment has been minimized.

Show comment
Hide comment
@soumyalj

soumyalj Jul 13, 2017

Tested with Rancher Container Crontab catalog version v0.2.0 on rancher v1.6.3. Logs are added after each container is added with cron.schedule/cron.action or any relevant cron labels. Logs are added when we remove containers with cron labels.
Example below:

7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Starting Cron"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Scanning for container cron entries"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f, with schedule: 0 */1 * * * *"
7/13/2017 1:35:41 PMtime="2017-07-13T20:35:41Z" level=info msg="Added: 5b974651ea6ce12051c19aab44bfe15a852d1583dbb6cd753f4f705c4ad24573, with schedule: 0 0 */1 * * *"
7/13/2017 1:41:51 PMtime="2017-07-13T20:41:51Z" level=info msg="Removed: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74"
7/13/2017 1:42:35 PMtime="2017-07-13T20:42:35Z" level=info msg="Removed: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f"
7/13/2017 1:42:39 PMtime="2017-07-13T20:42:39Z" level=info msg="Removed: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c"
7/13/2017 1:51:43 PMtime="2017-07-13T20:51:43Z" level=info msg="Added: c9a6da1a590cb92336f177991b5b2e7bebea142f4e821c7c1f8607475257ad5d, with schedule: 0 */1 * * * *"


soumyalj commented Jul 13, 2017

Tested with Rancher Container Crontab catalog version v0.2.0 on rancher v1.6.3. Logs are added after each container is added with cron.schedule/cron.action or any relevant cron labels. Logs are added when we remove containers with cron labels.
Example below:

7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Starting Cron"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Scanning for container cron entries"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f, with schedule: 0 */1 * * * *"
7/13/2017 1:35:41 PMtime="2017-07-13T20:35:41Z" level=info msg="Added: 5b974651ea6ce12051c19aab44bfe15a852d1583dbb6cd753f4f705c4ad24573, with schedule: 0 0 */1 * * *"
7/13/2017 1:41:51 PMtime="2017-07-13T20:41:51Z" level=info msg="Removed: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74"
7/13/2017 1:42:35 PMtime="2017-07-13T20:42:35Z" level=info msg="Removed: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f"
7/13/2017 1:42:39 PMtime="2017-07-13T20:42:39Z" level=info msg="Removed: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c"
7/13/2017 1:51:43 PMtime="2017-07-13T20:51:43Z" level=info msg="Added: c9a6da1a590cb92336f177991b5b2e7bebea142f4e821c7c1f8607475257ad5d, with schedule: 0 */1 * * * *"


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment