Skip to content

[JENKINS-3696] Dead Node Causes Exceptions w/ Node Config Site #14297

Closed
Bug
Closed
[JENKINS-3696] Dead Node Causes Exceptions w/ Node Config Site#14297
Bug
Labels
component:coreimported-jira-issuepriority:criticalresolution:fixed

Description

@jenkins-infra-bot
jenkins-infra-bot
opened on May 15, 2009

We encountered an issue causing one of our test nodes to fall into the Dead
state. Our theory at the moment is that some one was trying to reconfigure a
node while it was in use, specifically we think they tried to rename it. We
believe that caused the test node to get into the dead state. After that
happened we were unable to access the node configuration pages for any of the
test nodes in the system. When ever we try to access the configuration page for
a node we get a status code 500 page with a stack trace which I will include below.

We are running everything on SunOS 10, version 1.303 of Hudson, and we use ssh
to connect our slaves.

I am pasting three excerpts from our logs. The first is the output from
nohup.out which I believe shows the exceptions that occurred while some one was
configuring one of the test nodes. The second section of logs is from hudson.log
which shows more exceptions with regards to the test node. The third log is the
error the client receives in the browser.

--------------------
NOHUP.OUT
--------------------

May 15, 2009 10:55:15 AM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect qatest28a.qa - ACE I18N
Exception in thread "Executor #0 for qatest28a.qa" java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:206)
	at hudson.model.Executor.run(Executor.java:130)
May 15, 2009 10:55:25 AM hudson.slaves.CommandLauncher launch
INFO: slave agent launched for qatest28a.qa - ACE I18N
May 15, 2009 10:55:35 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.ComputerRetentionWork@​1d8b7abe failed
java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.getRetentionStrategy(SlaveComputer.java:400)
	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
May 15, 2009 10:56:35 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.ComputerRetentionWork@​1d8b7abe failed
java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.getRetentionStrategy(SlaveComputer.java:400)
	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
May 15, 2009 10:56:56 AM hudson.ExpressionFactory2$JexlExpression evaluate
WARNING: Caught exception evaluating: it.manualLaunchAllowed. Reason:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor570.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at
org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at
org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
	at
org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:71)
	at
org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
	at
org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71)
	at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:41)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at
org.kohsuke.stapler.jelly.CustomTagLibrary$StaplerDynamicTag$1.run(CustomTagLibrary.java:147)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:42)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:81)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at
org.kohsuke.stapler.jelly.CustomTagLibrary$StaplerDynamicTag$1.run(CustomTagLibrary.java:147)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:81)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:29)
	at
org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:43)
	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:75)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:475)
	at org.kohsuke.stapler.MetaClass$8.dispatch(MetaClass.java:221)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:486)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:150)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:486)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:404)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:117)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
	at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
	at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:91)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:83)
	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:35)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at
Originally reported by elismaga, imported from: Dead Node Causes Exceptions w/ Node Config Site
  • assignee: kohsuke
  • status: Resolved
  • priority: Critical
  • component(s): core
  • resolution: Fixed
  • resolved: 2010-12-14T11:58:21+00:00
  • votes: 0
  • watchers: 1
  • imported: 2025-11-24
Raw content of original issue 
We encountered an issue causing one of our test nodes to fall into the Dead
state. Our theory at the moment is that some one was trying to reconfigure a
node while it was in use, specifically we think they tried to rename it. We
believe that caused the test node to get into the dead state. After that
happened we were unable to access the node configuration pages for any of the
test nodes in the system. When ever we try to access the configuration page for
a node we get a status code 500 page with a stack trace which I will include below. 



We are running everything on SunOS 10, version 1.303 of Hudson, and we use ssh
to connect our slaves. 



I am pasting three excerpts from our logs. The first is the output from
nohup.out which I believe shows the exceptions that occurred while some one was
configuring one of the test nodes. The second section of logs is from hudson.log
which shows more exceptions with regards to the test node. The third log is the
error the client receives in the browser.



--------------------
NOHUP.OUT
--------------------




May 15, 2009 10:55:15 AM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect qatest28a.qa - ACE I18N
Exception in thread "Executor #0 for qatest28a.qa" java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:206)
	at hudson.model.Executor.run(Executor.java:130)
May 15, 2009 10:55:25 AM hudson.slaves.CommandLauncher launch
INFO: slave agent launched for qatest28a.qa - ACE I18N
May 15, 2009 10:55:35 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.ComputerRetentionWork@1d8b7abe failed
java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.getRetentionStrategy(SlaveComputer.java:400)
	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
May 15, 2009 10:56:35 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.ComputerRetentionWork@1d8b7abe failed
java.lang.NullPointerException
	at hudson.slaves.SlaveComputer.getRetentionStrategy(SlaveComputer.java:400)
	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
May 15, 2009 10:56:56 AM hudson.ExpressionFactory2$JexlExpression evaluate
WARNING: Caught exception evaluating: it.manualLaunchAllowed. Reason:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor570.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at
org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at
org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
	at
org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:71)
	at
org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
	at
org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71)
	at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:41)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at
org.kohsuke.stapler.jelly.CustomTagLibrary$StaplerDynamicTag$1.run(CustomTagLibrary.java:147)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:42)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:81)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at
org.kohsuke.stapler.jelly.CustomTagLibrary$StaplerDynamicTag$1.run(CustomTagLibrary.java:147)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
	at org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:81)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
	at org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
	at
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:29)
	at
org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:43)
	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:75)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:475)
	at org.kohsuke.stapler.MetaClass$8.dispatch(MetaClass.java:221)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:486)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:150)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:486)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:404)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:117)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
	at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
	at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:91)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:83)
	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:35)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at
  • environment: Platform: Sun, OS: SunOS

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:coreimported-jira-issuepriority:criticalresolution:fixed

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions