Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Job's references are reported as Killed #821

Closed
schicky opened this issue Jun 4, 2014 · 3 comments
Closed

Job's references are reported as Killed #821

schicky opened this issue Jun 4, 2014 · 3 comments
Labels
bug
Milestone
2.2

Comments

@schicky
Copy link

schicky commented Jun 4, 2014

Jobs are being reported as "Killed" although they are completing successfully. The same steps are being run on multiple nodes and it seems random which ones this happens to. It is happening to about 5-10% of the nodes and the parent job is then "Failed" although all the desired actions were completed. It is also not always the same job reference that is having the problem. I believe it is a false report because the workflow still continues even though it is set to stop with any failure and the actions they are performing are completed successfully on the node. So far the only correlation I've found is on job references. I haven't seen this behavior on jobs with no job references. I'm also receiving a java.util.ConcurrentModificationException but not on the same node that is reporting killed, so it may not be related.

I'm running Rundeck 2.1.0-1 on Tomcat with Java 1.7.0_45, Rhel 6.4
rundeckkilledstep
concurrentexception
@gschueler gschueler added the bug label Jun 4, 2014
@gschueler gschueler added this to the 2.2 milestone Jun 4, 2014
@ahonor
Copy link
Contributor

ahonor commented Jun 6, 2014

Hi @schicky
Is there more stack trace in the service.log? If so, can you attach the complete stack trace?
Also, does the job run sequentially across nodes or does it use a thread count greater than 1?

@schicky
Copy link
Author

schicky commented Jun 6, 2014

There is about 3000 lines of errors, but seems to be two distinct ones (Cannot change from ABORTED to SUCCEEDED and Cannot change from SUCCEEDED to RUNNING). Here is a sample of two of them

2014-06-04 18:49:33,072 ERROR WorkflowService - updateStateForStep([identifier:1/1, index:0, stepStateChange:StepStateChangeImpl{stepState=StepStateImpl{executionState=SUCCEEDED, metadata=null, errorMessage='null'}, nodeName='vqas2128a_tomcat_applyweb-alpha-a3', nodeState=true}, timestamp:Wed Jun 04 18:49:33 GMT 2014]): Cannot change from ABORTED to SUCCEEDED
java.lang.IllegalStateException: Cannot change from ABORTED to SUCCEEDED
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateState(MutableWorkflowStateImpl.groovy:416)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateNodeStepState(MutableWorkflowStateImpl.groovy:192)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groovy:115)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.descendUpdateStateForStep(MutableWorkflowStateImpl.groovy:267)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groovy:106)
at com.dtolabs.rundeck.app.internal.workflow.DelegateMutableWorkflowState.updateStateForStep(DelegateMutableWorkflowState.groovy:41)
at com.dtolabs.rundeck.app.internal.workflow.LogMutableWorkflowState.updateStateForStep(LogMutableWorkflowState.groovy:67)
at com.dtolabs.rundeck.app.internal.workflow.ExceptionHandlingMutableWorkflowState.updateStateForStep(ExceptionHandlingMutableWorkflowState.groovy:35)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateListener.stepStateChanged(MutableWorkflowStateListener.groovy:24)
at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.notifyAllStepState(WorkflowExecutionStateListenerAdapter.java:55)
at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.finishExecuteNodeStep(WorkflowExecutionStateListenerAdapter.java:194)
at com.dtolabs.rundeck.app.internal.workflow.MultiWorkflowExecutionListener.finishExecuteNodeStep(MultiWorkflowExecutionListener.groovy:93)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:161)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:116)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:58)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:177)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:116)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWFItem(BaseWorkflowStrategy.java:162)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:222)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:189)
at com.dtolabs.rundeck.core.execution.workflow.StepFirstWorkflowStrategy.executeWorkflowImpl(StepFirstWorkflowStrategy.java:79)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy$DispatchedWorkflow.dispatch(NodeFirstWorkflowStrategy.java:365)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:118)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:64)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:196)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy.executeWFSectionNodeDispatch(NodeFirstWorkflowStrategy.java:177)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy.executeWorkflowImpl(NodeFirstWorkflowStrategy.java:106)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at rundeck.services.ExecutionService$_runJobRefExecutionItem_closure50_closure89.doCall(ExecutionService.groovy:1860)
at com.codahale.metrics.Timer.time(Timer.java:99)
at MetricswebGrailsPlugin$_addDynamicMetricMethods_closure26.doCall(MetricswebGrailsPlugin.groovy:174)
at rundeck.services.ExecutionService$_runJobRefExecutionItem_closure50.doCall(ExecutionService.groovy:1859)
at rundeck.services.RequestHelper.doWithMockRequest(RequestHelper.groovy:30)
at rundeck.services.ExecutionService.runJobRefExecutionItem(ExecutionService.groovy:1797)
at rundeck.services.ExecutionService.executeNodeStep(ExecutionService.groovy:2150)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:149)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:116)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:58)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:177)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:116)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWFItem(BaseWorkflowStrategy.java:162)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:222)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:189)
at com.dtolabs.rundeck.core.execution.workflow.StepFirstWorkflowStrategy.executeWorkflowImpl(StepFirstWorkflowStrategy.java:79)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy$DispatchedWorkflow.dispatch(NodeFirstWorkflowStrategy.java:365)
at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$1.call(ParallelNodeDispatcher.java:162)
at com.dtolabs.rundeck.core.cli.CallableWrapperTask.execute(CallableWrapperTask.java:52)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run(Parallel.java:428)
at java.lang.Thread.run(Thread.java:744)

014-06-04 18:49:47,892 ERROR WorkflowService - updateStateForStep([identifier:5/1, index:0, stepStateChange:StepStateChangeImpl{stepState=StepStateImpl{executionState=RUNNING, metadata=null, errorMessage='null'}, nodeName='mpdm0823a_tomcat', nodeState=true}, timestamp:Wed Jun 04 18:49:47 GMT 2014]): Cannot change from SUCCEEDED to RUNNING
java.lang.IllegalStateException: Cannot change from SUCCEEDED to RUNNING
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateState(MutableWorkflowStateImpl.groovy:416)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateNodeStepState(MutableWorkflowStateImpl.groovy:192)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groovy:115)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.descendUpdateStateForStep(MutableWorkflowStateImpl.groovy:267)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groovy:106)
at com.dtolabs.rundeck.app.internal.workflow.DelegateMutableWorkflowState.updateStateForStep(DelegateMutableWorkflowState.groovy:41)
at com.dtolabs.rundeck.app.internal.workflow.LogMutableWorkflowState.updateStateForStep(LogMutableWorkflowState.groovy:67)
at com.dtolabs.rundeck.app.internal.workflow.ExceptionHandlingMutableWorkflowState.updateStateForStep(ExceptionHandlingMutableWorkflowState.groovy:35)
at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateListener.stepStateChanged(MutableWorkflowStateListener.groovy:24)
at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.notifyAllStepState(WorkflowExecutionStateListenerAdapter.java:55)
at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.beginExecuteNodeStep(WorkflowExecutionStateListenerAdapter.java:175)
at com.dtolabs.rundeck.app.internal.workflow.MultiWorkflowExecutionListener.beginExecuteNodeStep(MultiWorkflowExecutionListener.groovy:88)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:140)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:116)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:58)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:177)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:116)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWFItem(BaseWorkflowStrategy.java:162)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:222)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:189)
at com.dtolabs.rundeck.core.execution.workflow.StepFirstWorkflowStrategy.executeWorkflowImpl(StepFirstWorkflowStrategy.java:79)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy$DispatchedWorkflow.dispatch(NodeFirstWorkflowStrategy.java:365)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:118)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:64)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:196)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy.executeWFSectionNodeDispatch(NodeFirstWorkflowStrategy.java:177)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy.executeWorkflowImpl(NodeFirstWorkflowStrategy.java:106)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at rundeck.services.ExecutionService$_runJobRefExecutionItem_closure50_closure89.doCall(ExecutionService.groovy:1860)
at com.codahale.metrics.Timer.time(Timer.java:99)
at MetricswebGrailsPlugin$_addDynamicMetricMethods_closure26.doCall(MetricswebGrailsPlugin.groovy:174)
at rundeck.services.ExecutionService$_runJobRefExecutionItem_closure50.doCall(ExecutionService.groovy:1859)
at rundeck.services.RequestHelper.doWithMockRequest(RequestHelper.groovy:30)
at rundeck.services.ExecutionService.runJobRefExecutionItem(ExecutionService.groovy:1797)
at rundeck.services.ExecutionService.executeNodeStep(ExecutionService.groovy:2150)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:149)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:116)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:58)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:177)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:116)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWFItem(BaseWorkflowStrategy.java:162)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:222)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflowItemsForNodeSet(BaseWorkflowStrategy.java:189)
at com.dtolabs.rundeck.core.execution.workflow.StepFirstWorkflowStrategy.executeWorkflowImpl(StepFirstWorkflowStrategy.java:79)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowStrategy.executeWorkflow(BaseWorkflowStrategy.java:120)
at com.dtolabs.rundeck.core.execution.workflow.NodeFirstWorkflowStrategy$DispatchedWorkflow.dispatch(NodeFirstWorkflowStrategy.java:365)
at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$1.call(ParallelNodeDispatcher.java:162)
at com.dtolabs.rundeck.core.cli.CallableWrapperTask.execute(CallableWrapperTask.java:52)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run(Parallel.java:428)
at java.lang.Thread.run(Thread.java:744)

@gschueler gschueler mentioned this issue Jun 23, 2014
Closed
@gschueler gschueler modified the milestones: 2.1.3, 2.2 Jun 27, 2014
gschueler added a commit that referenced this issue Jun 27, 2014
@gschueler
Temporary fix for #821: don't throw illegal state exception
dabe975 
multiple threads might change value of the same state object
when running a node-step job reference in parallel.
don't throw illegal state exception
gschueler added a commit that referenced this issue Jun 30, 2014
@gschueler
Temporary fix for #821: don't throw illegal state exception
695e5a8 
multiple threads might change value of the same state object
when running a node-step job reference in parallel.
don't throw illegal state exception
@gschueler
Copy link
Member

58981b8

  • use a pair of Step,Node as context stack storage
  • remove WFStepContext, replace with existing StepContextId
  • Add parameter map to a step context id
  • If a subworkflow starts within a node context, add the node a parameter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
2.2
Development

No branches or pull requests
3 participants
@gschueler @ahonor @schicky