Skip to content

[JENKINS-23271] Intermittent Invalid Object ID in remoting module #1101

Description

@jenkins-infra-bot

For the last two weeks, I have been seeing an intermittent failure during the clearcase stage of our builds. It throws a Java exception along the lines of

FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
etc...

It always happens during the Clearcase segment of the build, but on a different step each time. Sometimes after lsview, sometimes after lshistory, sometimes after mkview ...

It happens a very low percentage of the time, and it happens on a different project each time. I have around 300 projects running between a number of matrix jobs- we are seeing one or two failures on each runthrough of this, always on different jobs and on different slaves. It started a couple weeks ago- our clearcase server crashed, and jenkins was having some trouble interacting with the server. I upgraded to 1.563 to try and solve that problem (I wound up having to manually delete all the views that had been created in the period between the backup and the crash). Once that issue had been resolved though, we started seeing this problem.

I upgraded the clearcase plugin to 1.5, and then tried downgrading Jenkins to 1.554.2. I cannot swear to it, but I thought I was on 1.558 before the first problem popped up, and had been very stable for a long time with all build failures being attributed to bad code, so suddenly having Jenkins fail some is kind of jarring.

At this point, I am not sure what I changed to cause the problem. I remember I had to upgrade slave.jar to version 2.41 from whatever it was before when I did the first upgrade to 1.563. Other then that, nothing else substantial has changed. The Jenkins is on a restricted network with no internet access- I am going to type the stack trace below, so forgive any typos.

FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
at hudson.remoting.ExportTable.get(ExportTable.java:261)
at hudson.remoting.Channel.getExportedObject(Channel.java:596)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:292
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:281)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:240)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(UNKNOWN SOURCE)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Cased by: java.lang.Exception: Object was recently deallocated
#18649 (ref.0) : hudson.Launcher$RemoteLaunchCallable$1
created at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.(ExportTable.java:86)
at hudson.remoting.ExportTable.export(ExportTable.java:239)
at hudson.remoting.Channel.export(Channel.java:592)
at hudson.remoting.Channel.export(Channel.java:583)
at hudson.remoting.Channel.export(Channel.java:553)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1025)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:990)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)

at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270)
... 15 more
Caused by: Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)


Originally reported by pbecotte, imported from: Intermittent Invalid Object ID in remoting module
  • assignee: oleg_nenashev
  • status: Resolved
  • priority: Blocker
  • component(s): remoting
  • label(s): 2.32.1-fixed, remoting
  • resolution: Fixed
  • resolved: 2016-12-07T08:11:28+00:00
  • votes: 83
  • watchers: 102
  • imported: 2025-11-25
Raw content of original issue

For the last two weeks, I have been seeing an intermittent failure during the clearcase stage of our builds. It throws a Java exception along the lines of

FATAL: Invalid object ID 18649 iuota=18470 java.lang.IllegalStateException: Invalid object ID 18469 iota=18470 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277) etc...

It always happens during the Clearcase segment of the build, but on a different step each time. Sometimes after lsview, sometimes after lshistory, sometimes after mkview ...

It happens a very low percentage of the time, and it happens on a different project each time. I have around 300 projects running between a number of matrix jobs- we are seeing one or two failures on each runthrough of this, always on different jobs and on different slaves. It started a couple weeks ago- our clearcase server crashed, and jenkins was having some trouble interacting with the server. I upgraded to 1.563 to try and solve that problem (I wound up having to manually delete all the views that had been created in the period between the backup and the crash). Once that issue had been resolved though, we started seeing this problem.

I upgraded the clearcase plugin to 1.5, and then tried downgrading Jenkins to 1.554.2. I cannot swear to it, but I thought I was on 1.558 before the first problem popped up, and had been very stable for a long time with all build failures being attributed to bad code, so suddenly having Jenkins fail some is kind of jarring.

At this point, I am not sure what I changed to cause the problem. I remember I had to upgrade slave.jar to version 2.41 from whatever it was before when I did the first upgrade to 1.563. Other then that, nothing else substantial has changed. The Jenkins is on a restricted network with no internet access- I am going to type the stack trace below, so forgive any typos.

FATAL: Invalid object ID 18649 iuota=18470 java.lang.IllegalStateException: Invalid object ID 18469 iota=18470 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277) at hudson.remoting.ExportTable.get(ExportTable.java:261) at hudson.remoting.Channel.getExportedObject(Channel.java:596) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:292 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:281) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:240) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(UNKNOWN SOURCE) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Cased by: java.lang.Exception: Object was recently deallocated #18649 (ref.0) : hudson.Launcher$RemoteLaunchCallable$1 created at Fri May 30 19:27:39 EDT 2014 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86) at hudson.remoting.ExportTable.export(ExportTable.java:239) at hudson.remoting.Channel.export(Channel.java:592) at hudson.remoting.Channel.export(Channel.java:583) at hudson.remoting.Channel.export(Channel.java:553) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1025) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:990) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Released at Fri May 30 19:27:39 EDT 2014 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303) at hudson.remoting.Channel.unexport(Channel.java:600) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38) at hudson.remoting.Channel$2.handle(Channel.java:475) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)

at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270) ... 15 more Caused by: Released at Fri May 30 19:27:39 EDT 2014 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303) at hudson.remoting.Channel.unexport(Channel.java:600) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38) at hudson.remoting.Channel$2.handle(Channel.java:475) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)

  • environment: 1.544.2, 1.563, 1.625.1 - Windows or Linux Master, various slaves

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions