Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revert coordinator lambda conversion to fix error #5591

Merged
merged 1 commit into from Apr 6, 2018

Conversation

@clintropolis
Copy link
Member

commented Apr 6, 2018

We have encountered this issue on some installations where the coordinator will explode with an error of the following form:

    2018-04-05T05:18:59,883 ERROR [LeaderSelector[/demo/coordinator/_COORDINATOR]] org.apache.curator.framework.listen.ListenerContainer - Listener (io.druid.curator.discovery.CuratorDruidLeaderSelector$1@1e7f19b4) threw an exception
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    io/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable.<init>(Lio/druid/server/coordinator/DruidCoordinator;I)V @16: invokedynamic
  Reason:
    Type uninitializedThis (current frame, stack[3]) is not assignable to 'io/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable'
  Current Frame:
    bci: @16
    flags: { flagThisUninit }
    locals: { uninitializedThis, 'io/druid/server/coordinator/DruidCoordinator', integer }
    stack: { uninitializedThis, 'io/druid/server/coordinator/DruidCoordinator', 'io/druid/server/coordinator/helper/DruidCoordinatorSegmentInfoLoader', uninitializedThis }
  Bytecode:
    0x0000000: 2a2b b500 012a 2bbb 0002 592b b700 032a
    0x0000010: ba00 0400 00bb 0005 592b b700 06bb 0007
    0x0000020: 592b b700 08bb 0009 592b b700 0abb 000b
    0x0000030: 592b b700 0cbb 000d 592b b700 0eb8 000f
    0x0000040: 1cb7 0010 b1

    at io.druid.server.coordinator.DruidCoordinator.becomeLeader(DruidCoordinator.java:516) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.server.coordinator.DruidCoordinator.access$000(DruidCoordinator.java:95) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.server.coordinator.DruidCoordinator$1.becomeLeader(DruidCoordinator.java:471) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.curator.discovery.CuratorDruidLeaderSelector$1.isLeader(CuratorDruidLeaderSelector.java:98) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:665) ~[curator-recipes-4.0.0.jar:4.0.0]
    at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:661) ~[curator-recipes-4.0.0.jar:4.0.0]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

It appears related to this change. Note that this doesn't happen on all coordinator installations with this patch, as some seem to be functioning correctly. We haven't isolated the steps or environment to reproduce this exactly, it is likely related to some combination of jvm versions and possibly which extensions are loaded. It may be a jvm bug related to lambda this binding affecting some versions?

Regardless, it seems best to revert this change until we can determine the cause and ensure it doesn't happen anymore.

@gianm
gianm approved these changes Apr 6, 2018
@jon-wei
jon-wei approved these changes Apr 6, 2018

@jon-wei jon-wei merged commit ea4f854 into apache:master Apr 6, 2018

2 checks passed

Inspections: pull requests (Druid) TeamCity build finished
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jon-wei jon-wei added the Bug label Apr 6, 2018

jon-wei added a commit to implydata/druid that referenced this pull request Apr 6, 2018
@leventov

This comment has been minimized.

Copy link
Member

commented Apr 6, 2018

Looks like an instance of this bug: https://bugs.openjdk.java.net/browse/JDK-8157142, that's marked as a duplicate of this: https://bugs.openjdk.java.net/browse/JDK-8129740, that's fixed and ported back to OpenJDK 8u121. Note that this is a javac, not a JVM bug; to ensure it doesn't appear again it would have been enough to update Druid binary build tooling to use a fresh version of javac.

@clintropolis

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2018

Thanks @leventov good find! I just bumped into that bug report searching more for a cause after submitting this PR, but missed that it had been backported. We have an environment that we should be able to confirm this with, will update in comments once we do so.

@jon-wei

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2018

I just verified that the lambda there is fine when Druid is built with an up to date javac

@leventov thanks for the find!

@clintropolis clintropolis deleted the clintropolis:fix-lambda-explosion branch Apr 6, 2018

@leventov

This comment has been minimized.

Copy link
Member

commented Apr 7, 2018

Could you create a PR that updates maven-compiler-plugin accordingly?

sathishsri88 pushed a commit to sathishs/druid that referenced this pull request May 8, 2018
gianm added a commit to implydata/druid that referenced this pull request May 16, 2018

@jihoonson jihoonson added this to the 0.12.2 milestone Jul 5, 2018

jihoonson added a commit to implydata/druid that referenced this pull request Jul 6, 2018
jihoonson added a commit to jihoonson/druid that referenced this pull request Jul 6, 2018
gianm added a commit that referenced this pull request Jul 9, 2018
leventov added a commit to metamx/druid that referenced this pull request Jul 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.