Fix failing SystemTests

[Prabha-Veerubhotla]

Problem description
AutoScaleTest, ReadWithAutoScaleTest are failing with following errors:
logs :

2017-03-27 08:52:07,552 53929 [taskpool-8] ERROR i.n.u.c.D.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:486)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85)
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:332)
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:126)
        at com.emc.pravega.stream.impl.netty.ConnectionFactoryImpl.establishConnection(ConnectionFactoryImpl.java:105)
        at com.emc.pravega.stream.impl.segment.SegmentOutputStreamImpl.lambda$setupConnection$1(SegmentOutputStreamImpl.java:338)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
        at com.emc.pravega.stream.impl.segment.SegmentOutputStreamImpl.setupConnection(SegmentOutputStreamImpl.java:337)
        at com.emc.pravega.stream.impl.segment.SegmentOutputStreamImpl.lambda$getConnection$0(SegmentOutputStreamImpl.java:327)
        at com.emc.pravega.common.util.Retry$RetryAndThrowBase.run(Retry.java:167)
        at com.emc.pravega.stream.impl.segment.SegmentOutputStreamImpl.getConnection(SegmentOutputStreamImpl.java:326)
        at com.emc.pravega.stream.impl.segment.SegmentOutputStreamImpl.write(SegmentOutputStreamImpl.java:306)
        at com.emc.pravega.stream.impl.EventStreamWriterImpl.writeEventInternal(EventStreamWriterImpl.java:95)
        at com.emc.pravega.stream.impl.EventStreamWriterImpl.writeEvent(EventStreamWriterImpl.java:79)
        at com.emc.pravega.controller.task.Stream.StreamTransactionMetadataTasks.lambda$commitTxnBody$9(StreamTransactionMetadataTasks.java:233)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

//Warning...
2017-03-27 08:52:18,252 64629 [taskpool-7] WARN  c.e.p.c.t.TimerWheelTimeoutService - Rescheduling timeout task for tx testAutoScale-1566118583/testTxnScaleUp/7a499d58-1b85-48b7-ab85-39f617f0f116 because of transient or unknown error
java.util.concurrent.CompletionException: com.emc.pravega.controller.store.task.LockFailedException: Failed locking resource testAutoScale-1566118583/testTxnScaleUp/7a499d58-1b85-48b7-ab85-39f617f0f116.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.emc.pravega.controller.store.task.LockFailedException: Failed locking resource testAutoScale-1566118583/testTxnScaleUp/7a499d58-1b85-48b7-ab85-39f617f0f116.
        at com.emc.pravega.controller.store.task.ZKTaskMetadataStore.acquireLock(ZKTaskMetadataStore.java:56)
        at com.emc.pravega.controller.store.task.AbstractTaskMetadataStore.lambda$lock$0(AbstractTaskMetadataStore.java:44)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        ... 7 common frames omitted
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /pravega/pravega-cluster/taskIndex/testAutoScale-1566118583_%_testTxnScaleUp_%_7a499d58-1b85-48b7-ab85-39f617f0f116
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:735)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:718)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:714)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
        at com.emc.pravega.controller.store.task.ZKTaskMetadataStore.acquireLock(ZKTaskMetadataStore.java:53)
        ... 9 common frames omitted

Problem location
TestFramework-System tests

Suggestions for an improvement

[bhargav-gulavani]

This is a result of @cleanup annotation on clientFactory created in initializeStreamWriters method of StreamTransactionMetadataTasks.

[bhargav-gulavani]

Fixed in #883