Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rancher crontab scheduling doesn't work. It doesn't pay attention to cron rule and executes restarts container every hour. #8979

Closed
ketchoop opened this issue Jun 6, 2017 · 10 comments
Assignees
Milestone

Comments

@ketchoop
Copy link

@ketchoop ketchoop commented Jun 6, 2017

Rancher versions:
rancher/server: 1.5.10
rancher/agent: 1.2.2
rancher/crontab: 0.1.0-rancher1

Infrastructure Stack versions:
network-services: net-manager 0.6.6

Docker version: (docker version,docker info preferred)

17.04.0-ce

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
RancherOS v1.0.1 kernel 4.9.24

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Parallels VM

Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)

Rancher HA external DB galera cluster

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)

cattle

**Problem: **

I have elasticsearch curator container that has ```cron.schedule=0 0 0 * * *" label and ran with started-once, for example. And logs says that this container restarts every hour. And this is the same problem with the other containers.

@ketchoop
Copy link
Author

@ketchoop ketchoop commented Jun 7, 2017

Maybe it's not cron trouble, but "started-once" label. But it's strange, that containers automatically runs. Can somebody comment this behaviour, please, because there is no info in docs about this.

@ketchoop
Copy link
Author

@ketchoop ketchoop commented Jun 8, 2017

Guys, please, say something about it. Because it's not obvious and I think, wrong behaviour, that started once containers restarts, aproximately, every hour. It makes cron container and started-once useless.

@aemneina
Copy link

@aemneina aemneina commented Jun 9, 2017

Can you restart the crontab service to see if it picks up the changes on the container. I think the crontab service isnt automatically scanning and detecting new containers.

@ketchoop
Copy link
Author

@ketchoop ketchoop commented Jun 9, 2017

@aemneina didn't restart crontab, but upgraded it to avoid side effects . But, It still happens. My guess, that it's because of started-once label. but I'm not really sure about it. Let me check it with another container.

@ketchoop
Copy link
Author

@ketchoop ketchoop commented Jun 9, 2017

@aemneina Yep, it's not connected with cron. Rancher just restarts started-once containers.

P.S. Maybe it'll be helpful: the last time, when I saw restart it showed me deadlock in database.

UPD: I think I found the reason of this behaviour and maybe it's not connected with crontab and started-once.

@ketchoop
Copy link
Author

@ketchoop ketchoop commented Jun 9, 2017

@aemneina thank you for response! It helped me to find bug on my side, that was connected with hourly cleanup. And crontab seems to be ok and it changes schedule properly.

@ketchoop ketchoop closed this Jun 9, 2017
@aemneina
Copy link

@aemneina aemneina commented Jun 9, 2017

@ket4yii I'm going to reopen, i think the catalog item does have a bug where new containers arent detected. That way we can eventually get a fix in for that.

@cloudnautique
Copy link
Member

@cloudnautique cloudnautique commented Jun 20, 2017

This is a UX issue. The Docker event to add the job is not logged on create. The restart logs all of the containers it finds on startup. This is confusing behavior and can/will be addressed.

@galal-hussein
Copy link
Contributor

@galal-hussein galal-hussein commented Jul 11, 2017

@soumyalj Can you test with v1.6-development branch

@soumyalj
Copy link

@soumyalj soumyalj commented Jul 13, 2017

Tested with Rancher Container Crontab catalog version v0.2.0 on rancher v1.6.3. Logs are added after each container is added with cron.schedule/cron.action or any relevant cron labels. Logs are added when we remove containers with cron labels.
Example below:

7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Starting Cron"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Scanning for container cron entries"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c, with schedule: 0 */1 * * * *"
7/13/2017 1:28:21 PMtime="2017-07-13T20:28:21Z" level=info msg="Added: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f, with schedule: 0 */1 * * * *"
7/13/2017 1:35:41 PMtime="2017-07-13T20:35:41Z" level=info msg="Added: 5b974651ea6ce12051c19aab44bfe15a852d1583dbb6cd753f4f705c4ad24573, with schedule: 0 0 */1 * * *"
7/13/2017 1:41:51 PMtime="2017-07-13T20:41:51Z" level=info msg="Removed: 5f5c6599e862b122157ed0686639e679f5f1621f409af0180802112b16918f74"
7/13/2017 1:42:35 PMtime="2017-07-13T20:42:35Z" level=info msg="Removed: 36f0639364f4b648fe551d0094918b4c15786e80731eef72eb9fc9927b1f8e4f"
7/13/2017 1:42:39 PMtime="2017-07-13T20:42:39Z" level=info msg="Removed: 9b787825b726f95aae8b3a668cb51d37cc30b0c8c2a50f2ebba6dc57879f181c"
7/13/2017 1:51:43 PMtime="2017-07-13T20:51:43Z" level=info msg="Added: c9a6da1a590cb92336f177991b5b2e7bebea142f4e821c7c1f8607475257ad5d, with schedule: 0 */1 * * * *"


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.