Takeover schedule API is timing out in rundeck 2.6.10 #2176

Closed
nilroy opened this Issue Nov 16, 2016 · 6 comments

Projects

None yet

3 participants

@nilroy
nilroy commented Nov 16, 2016 edited

Bug report

My Rundeck detail

  • Rundeck version: 2.6.10_
  • install type: deb
  • OS Name/version: Ubuntu 14.04

Expected Behavior
Takeover of all schduled jobs from one node to another should work.

Actual Behavior
We are running a two node rundeck cluster (Active/passive). Recently I upgraded one node to rundeck 2.6.10. And after that whenever I am trying to takeover all job schdulees from the rundeck node running 2.6.9 to the latest one the API (/api/17/scheduler/jobs) is giving timeout exception (Timed out reading data from server). This was not the case with earlier versions of rundeck. From rundeck service log I could see that schedule for same job is being claimed multiple times

INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36
INFO  ScheduledExecutionService: claimScheduledJob: schedule claimed for 80 on node ce34b251-c27d-4699-80fb-0ecd723dec36

How to reproduce Behavior
Use two rundeck nodes in cluster mode. Install rundeck 2.6.10 in one and 2.6.9 in another. Try to schedule jobs in 2.6.9 and then try to takeover all schedules from 2.6.9 to 2.6.10 using API


...

@nilroy
nilroy commented Nov 16, 2016

This issue is thre in 2.6.11 also

@nilroy
nilroy commented Nov 17, 2016

@gschueler Any idea?

@nilroy nilroy referenced this issue in Webtrends/rundeck Nov 17, 2016
Merged

Rundeck 2.6.10 #105

@niphlod
niphlod commented Nov 17, 2016

+1. Happening on my env, too...win2012r2 with mssql backend, 2.6.10 and 2.6.11, type: launcher

@nilroy
nilroy commented Nov 30, 2016

Any help?

@niphlod
niphlod commented Nov 30, 2016

yep, me too. I'm kinda stuck pushing rundeck without clustering.

@gschueler
Contributor

I think that the job takeover is repeating for the same job multiple times due to the query that looks for both scheduled jobs and jobs with scheduled executions. I believe it can be fixed to only do the "takeover" once per discovered job

@gschueler gschueler added this to the 2.7.0 milestone Nov 30, 2016
@gschueler gschueler added a commit to gschueler/rundeck that referenced this issue Nov 30, 2016
@gschueler gschueler cluster mode takeover should execute once per job fix #2176 8f762d7
@gschueler gschueler self-assigned this Nov 30, 2016
@gschueler gschueler closed this in #2213 Nov 30, 2016
@gschueler gschueler removed the in progress label Nov 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment