Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IllegalStateException in cell communication #3219

Closed
jstarek opened this issue Jun 13, 2017 · 1 comment
Closed

IllegalStateException in cell communication #3219

jstarek opened this issue Jun 13, 2017 · 1 comment

Comments

@jstarek
Copy link
Contributor

jstarek commented Jun 13, 2017

During regular Jenkins testing, the following occured in a 3.0 instance:

Found stack traces in /var/log/dcache/dCacheDomain.log

java.lang.IllegalStateException: Cannot send message with callback in state {} [PRE_STARTUP]
	at com.google.common.base.Preconditions.checkState(Preconditions.java:199) ~[guava-19.0.jar:na]
	at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:519) ~[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at dmg.cells.nucleus.CellAdapter.sendMessage(CellAdapter.java:468) ~[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:129) ~[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:99) ~[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at org.dcache.services.info.gathers.SingleMessageDga.trigger(SingleMessageDga.java:83) ~[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at org.dcache.services.info.gathers.DataGatheringScheduler$RegisteredActivity.checkAndTrigger(DataGatheringScheduler.java:131) ~[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at org.dcache.services.info.gathers.DataGatheringScheduler.run(DataGatheringScheduler.java:281) ~[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

Full logs and details at https://ci.dcache.org/job/3.0%20post%20testing/334/ .
@paulmillar
Copy link
Member

See https://rb.dcache.org/r/10234/

paulmillar added a commit to paulmillar/dcache that referenced this issue Jun 14, 2017
@paulmillar
info: avoid sending messages too early.
1a76715 
Motivation:

The info cell can send messages before the cell is properly registered.
Message sending is randomised, so this does not always happen, but when
it does, it can lead to errors like:

    java.lang.IllegalStateException: Cannot send message with callback in state \{\} \[PRE_STARTUP\]
    	at com.google.common.base.Preconditions.checkState(Preconditions.java:199) ~\[guava-19.0.jar:na\]
    	at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:519) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at dmg.cells.nucleus.CellAdapter.sendMessage(CellAdapter.java:468) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:129) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:99) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.SingleMessageDga.trigger(SingleMessageDga.java:83) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler$RegisteredActivity.checkAndTrigger(DataGatheringScheduler.java:131) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler.run(DataGatheringScheduler.java:281) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at java.lang.Thread.run(Thread.java:748) ~\[na:1.8.0_131\]

Modification:

Only start sending messages once the cell nucleus is in POST_STARTUP
phase.

Result:

No more IllegalStateException stack-traces about attempting to send
message with callback in state PRE_STARTUP.

Target: master
Request: 3.1
Request: 3.0
Request: 2.16
Requires-notes: yes
Requires-book: no
Patch: https://rb.dcache.org/r/10234/
Acked-by: Tigran Mkrtchyan
Fixes dCache#3219
@paulmillar paulmillar mentioned this issue Jun 14, 2017
Merged
paulmillar added a commit to paulmillar/dcache that referenced this issue Jun 14, 2017
@paulmillar
info: avoid sending messages too early.
6407eb9 
Motivation:

The info cell can send messages before the cell is properly registered.
Message sending is randomised, so this does not always happen, but when
it does, it can lead to errors like:

    java.lang.IllegalStateException: Cannot send message with callback in state \{\} \[PRE_STARTUP\]
    	at com.google.common.base.Preconditions.checkState(Preconditions.java:199) ~\[guava-19.0.jar:na\]
    	at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:519) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at dmg.cells.nucleus.CellAdapter.sendMessage(CellAdapter.java:468) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:129) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:99) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.SingleMessageDga.trigger(SingleMessageDga.java:83) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler$RegisteredActivity.checkAndTrigger(DataGatheringScheduler.java:131) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler.run(DataGatheringScheduler.java:281) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at java.lang.Thread.run(Thread.java:748) ~\[na:1.8.0_131\]

Modification:

Only start sending messages once the cell nucleus is in POST_STARTUP
phase.

Result:

No more IllegalStateException stack-traces about attempting to send
message with callback in state PRE_STARTUP.

Target: master
Request: 3.1
Request: 3.0
Request: 2.16
Requires-notes: yes
Requires-book: no
Patch: https://rb.dcache.org/r/10234/
Acked-by: Tigran Mkrtchyan
Fixes dCache#3219
@paulmillar paulmillar mentioned this issue Jun 14, 2017
Merged
paulmillar added a commit to paulmillar/dcache that referenced this issue Jun 14, 2017
@paulmillar
info: avoid sending messages too early.
131d94e 
Motivation:

The info cell can send messages before the cell is properly registered.
Message sending is randomised, so this does not always happen, but when
it does, it can lead to errors like:

    java.lang.IllegalStateException: Cannot send message with callback in state \{\} \[PRE_STARTUP\]
    	at com.google.common.base.Preconditions.checkState(Preconditions.java:199) ~\[guava-19.0.jar:na\]
    	at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:519) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at dmg.cells.nucleus.CellAdapter.sendMessage(CellAdapter.java:468) ~\[cells-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:129) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.MessageHandlerChain.sendMessage(MessageHandlerChain.java:99) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.SingleMessageDga.trigger(SingleMessageDga.java:83) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler$RegisteredActivity.checkAndTrigger(DataGatheringScheduler.java:131) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at org.dcache.services.info.gathers.DataGatheringScheduler.run(DataGatheringScheduler.java:281) ~\[dcache-info-3.0.19-SNAPSHOT.jar:3.0.19-SNAPSHOT\]
    	at java.lang.Thread.run(Thread.java:748) ~\[na:1.8.0_131\]

Modification:

Only start sending messages once the cell nucleus is in POST_STARTUP
phase.

Result:

No more IllegalStateException stack-traces about attempting to send
message with callback in state PRE_STARTUP.

Target: master
Request: 3.1
Request: 3.0
Request: 2.16
Requires-notes: yes
Requires-book: no
Patch: https://rb.dcache.org/r/10234/
Acked-by: Tigran Mkrtchyan
Fixes dCache#3219
This was referenced Jun 14, 2017
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@paulmillar @jstarek