revert coordinator lambda conversion to fix error

[clintropolis]

We have encountered this issue on some installations where the coordinator will explode with an error of the following form:

    2018-04-05T05:18:59,883 ERROR [LeaderSelector[/demo/coordinator/_COORDINATOR]] org.apache.curator.framework.listen.ListenerContainer - Listener (io.druid.curator.discovery.CuratorDruidLeaderSelector$1@1e7f19b4) threw an exception
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    io/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable.<init>(Lio/druid/server/coordinator/DruidCoordinator;I)V @16: invokedynamic
  Reason:
    Type uninitializedThis (current frame, stack[3]) is not assignable to 'io/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable'
  Current Frame:
    bci: @16
    flags: { flagThisUninit }
    locals: { uninitializedThis, 'io/druid/server/coordinator/DruidCoordinator', integer }
    stack: { uninitializedThis, 'io/druid/server/coordinator/DruidCoordinator', 'io/druid/server/coordinator/helper/DruidCoordinatorSegmentInfoLoader', uninitializedThis }
  Bytecode:
    0x0000000: 2a2b b500 012a 2bbb 0002 592b b700 032a
    0x0000010: ba00 0400 00bb 0005 592b b700 06bb 0007
    0x0000020: 592b b700 08bb 0009 592b b700 0abb 000b
    0x0000030: 592b b700 0cbb 000d 592b b700 0eb8 000f
    0x0000040: 1cb7 0010 b1

    at io.druid.server.coordinator.DruidCoordinator.becomeLeader(DruidCoordinator.java:516) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.server.coordinator.DruidCoordinator.access$000(DruidCoordinator.java:95) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.server.coordinator.DruidCoordinator$1.becomeLeader(DruidCoordinator.java:471) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at io.druid.curator.discovery.CuratorDruidLeaderSelector$1.isLeader(CuratorDruidLeaderSelector.java:98) ~[druid-server-0.12.0-iap5.jar:0.12.0-iap5]
    at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:665) ~[curator-recipes-4.0.0.jar:4.0.0]
    at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:661) ~[curator-recipes-4.0.0.jar:4.0.0]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

It appears related to this change. Note that this doesn't happen on all coordinator installations with this patch, as some seem to be functioning correctly. We haven't isolated the steps or environment to reproduce this exactly, it is likely related to some combination of jvm versions and possibly which extensions are loaded. It may be a jvm bug related to lambda this binding affecting some versions?

Regardless, it seems best to revert this change until we can determine the cause and ensure it doesn't happen anymore.

[leventov]

Looks like an instance of this bug: https://bugs.openjdk.java.net/browse/JDK-8157142, that's marked as a duplicate of this: https://bugs.openjdk.java.net/browse/JDK-8129740, that's fixed and ported back to OpenJDK 8u121. Note that this is a javac, not a JVM bug; to ensure it doesn't appear again it would have been enough to update Druid binary build tooling to use a fresh version of javac.

[clintropolis]

Thanks @leventov good find! I just bumped into that bug report searching more for a cause after submitting this PR, but missed that it had been backported. We have an environment that we should be able to confirm this with, will update in comments once we do so.

[jon-wei]

I just verified that the lambda there is fine when Druid is built with an up to date javac

@leventov thanks for the find!

[leventov]

Could you create a PR that updates maven-compiler-plugin accordingly?