Remove node #1053

Maithem · 2017-12-07T23:11:53Z

Overview

Description:
An implementation of a remove node workflow. This workflow extends the system's capability to downsize the cluster by removing nodes.

Why should this be merged:
This PR and is needed to complete the clustering effort.

Related issue: #1039

Checklist (Definition of Done):

There are no TODOs left in the code
Coding conventions (e.g. for logging, unit tests) have been followed
Change is covered by automated tests
Public API has Javadoc

no2chem · 2017-12-07T23:50:43Z

@Maithem Could you add the checklist back to the PR comment? Thanks.

coveralls · 2017-12-08T01:33:25Z

Coverage decreased (-0.4%) to 70.426% when pulling e90f5d2 on removeNode into 8e63b38 on master.

Maithem · 2017-12-09T00:33:34Z

@no2chem ok, added them along with an estimate.

zalokhan · 2017-12-09T02:04:00Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutManagementView.java

+        currentLayout.setRuntime(runtime);
+        sealEpoch(currentLayout);
+
+        LayoutBuilder builder = new LayoutBuilder(currentLayout);


We can build the layout before sealing.
So the order can be :

generate new layout

seal

consensus

reconfigure Sequencer

This way if there is an error while building the new layout, you would not have sealed and blocked the entire system.

Good point.

zalokhan · 2017-12-09T02:05:11Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

@@ -74,6 +80,18 @@ void handleQuery(CorfuPayloadMsg<OrchestratorRequest> msg, ChannelHandlerContext
                .payloadMsg(new OrchestratorResponse(resp )));
    }

+    void removeNode(CorfuPayloadMsg<OrchestratorRequest> msg, ChannelHandlerContext ctx, IServerRouter r) {


Can this be combined in dispatch?

corfudb-performance · 2017-12-12T10:57:43Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit 1378e8a.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

corfudb-performance · 2017-12-13T10:17:38Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit 3b93c48.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

codecov · 2017-12-14T01:06:37Z

Codecov Report

❗ No coverage uploaded for pull request base (master@8315e31). Click here to learn what that means.
The diff coverage is 26.47%.

@@            Coverage Diff            @@
##             master    #1053   +/-   ##
=========================================
  Coverage          ?   66.22%           
=========================================
  Files             ?      217           
  Lines             ?    10206           
  Branches          ?     1002           
=========================================
  Hits              ?     6759           
  Misses            ?     3062           
  Partials          ?      385

Impacted Files	Coverage Δ
...cols/wireprotocol/orchestrator/AddNodeRequest.java	`55.55% <ø> (ø)`
...ols/wireprotocol/orchestrator/AddNodeWorkflow.java	`0% <ø> (ø)`
...tocols/wireprotocol/orchestrator/ActionStatus.java	`0% <ø> (ø)`
...db/protocols/wireprotocol/orchestrator/Action.java	`0% <ø> (ø)`
...ireprotocol/orchestrator/OrchestratorResponse.java	`89.47% <ø> (ø)`
.../protocols/wireprotocol/orchestrator/Workflow.java	`0% <0%> (ø)`
...tocols/wireprotocol/orchestrator/QueryRequest.java	`92.85% <0%> (ø)`
...eprotocol/orchestrator/CreateWorkflowResponse.java	`38.46% <0%> (ø)`
...fudb/infrastructure/orchestrator/Orchestrator.java	`21.42% <20%> (ø)`
...protocol/orchestrator/OrchestratorRequestType.java	`41.66% <41.66%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8315e31...8c678dd. Read the comment docs.

corfudb-performance · 2017-12-14T05:16:49Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit 48fe631.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

no2chem

See comments.

I think this needs to be broken up into two PRs.

The first PR is a change to the orchestrator service. The second PR is to add remove node functionality.

no2chem · 2017-12-14T08:02:40Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

                r.sendResponse(ctx, msg, CorfuMsgType.ORCHESTRATOR_RESPONSE
                        .payloadMsg(resp));
-                run(workflow);
+                CompletableFuture.runAsync(() -> {


Why use a CompletableFuture when you never join on it?

no2chem · 2017-12-14T08:03:42Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

@@ -42,7 +39,7 @@
 public class Orchestrator {

    final Callable<CorfuRuntime> getRuntime;
-    final Map<String, UUID> activeWorkflows = new ConcurrentHashMap();
+    final Map<UUID, String> activeWorkflows = new ConcurrentHashMap();


It seems that you need a BiMap instead...

no2chem · 2017-12-14T08:04:15Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

            activeWorkflows.remove(workflow.getId());
        }
    }
+
+    UUID findUUID(String endpoint) {


See BiMap comment...

no2chem · 2017-12-14T08:06:52Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

-        if (payload.getType().equals(Types.OrchestratorRequestType.ADD_NODE)) {
-            return new AddNodeWorkflow(payload);
+        switch (payload.getType()) {
+            case ADD_NODE:


Can't you just make this part of the OrchestratorRequestType enum?

Add a new method:
OrchestratorRequestType.getWorkflow(OrchestratorRequest request)

Then no need to manage this extra switch statement.

no2chem · 2017-12-14T08:08:20Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Workflow.java

+     * @return true if all actions completed successfully
+     */
+    public boolean completed() {
+        for (Action action : getActions()) {


or getActions().stream().anyMatch(a -> a.equals(ActionStatus.COMPLETED)

no2chem · 2017-12-14T08:10:28Z

test/src/test/java/org/corfudb/runtime/view/LayoutManagementViewTest.java

+
+        // Remove a node from a two node cluster. Verify that a remove operation won't
+        // cause the loss of redundancy
+        assertThatThrownBy(() -> r.getLayoutManagementView().removeNode(l2,


This doesn't seem to do the second part (Verify that a remove operation won't

// cause the loss of redundancy)

Fixed the comment

no2chem · 2017-12-14T08:11:45Z

format/proto/types.proto

 }

 // Orchestrator responses
 enum OrchestratorResponseType {
    // The status of a workflow
    WORKFLOW_STATUS = 0;
    // Id of a created workflow
-    WORKFLOW_ID = 1;
+    WORKFLOW_CREATE = 1;


What is workflow_create? The comments don't match...

Changed the name.

corfudb-performance · 2017-12-15T10:17:21Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit aebbf0a.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

Maithem · 2017-12-18T22:26:21Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutManagementView.java

+            LayoutModificationException, OutrankedException {
+
+        currentLayout.setRuntime(runtime);
+        sealEpoch(currentLayout);


move seal after the layout creation.

Yes i think you can. We should not seal if we are going to throw an exception.

medhavidhawan · 2017-12-19T02:07:46Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

@@ -75,6 +81,30 @@ void handleQuery(CorfuPayloadMsg<OrchestratorRequest> msg, ChannelHandlerContext
                .payloadMsg(new OrchestratorResponse(resp)));
    }

+    void removeNode(CorfuPayloadMsg<OrchestratorRequest> msg, ChannelHandlerContext ctx, IServerRouter r) {


Removed, now we only have dispatched.

medhavidhawan · 2017-12-19T02:09:13Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

+        RemoveNodeRequest req = (RemoveNodeRequest) msg.getPayload().getRequest();
+
+        if (findUUID(req.getEndpoint()) != null) {
+            // An add node workflow is already executing for this endpoint, return


Is this comment out of place or is the findUUID getting add node workflows in remove node ?

Also this looks like similar code to what you do in addNode. Can we consolidate and have less code ?

I refactored in the HOG branch that depends on this, but I'll push the change here.

Removed, now we only have dispatch. Its refactored.

medhavidhawan · 2017-12-19T02:19:43Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/RemoveNodeWorkflow.java

+/**
+ * @author Maithem
+ */
+public class RemoveNodeWorkflow extends Workflow {


A short file level comment would do for this one.

medhavidhawan · 2017-12-19T02:23:40Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

+            return;
+        } else {
+            // Create a new workflow for this endpoint and return a new workflow id
+            Workflow workflow = getWorkflow(msg.getPayload());


Is this best effort or is the driver single threaded? Because this is not protected against multiple threads?

I will make it synchronized.

Done in dispatch method.

medhavidhawan · 2017-12-19T02:24:48Z

infrastructure/src/main/java/org/corfudb/infrastructure/orchestrator/Orchestrator.java

+            OrchestratorResponse resp = new OrchestratorResponse(new CreateWorkflowResponse(workflow.getId()));
+            r.sendResponse(ctx, msg, CorfuMsgType.ORCHESTRATOR_RESPONSE
+                    .payloadMsg(resp));
+            CompletableFuture.runAsync(() -> {


I think you should first start the asyncWorkflow before sending response. Just for sanity.

medhavidhawan · 2017-12-19T02:32:39Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutBuilder.java

+                                        + "No replicas available.");
+                    } else {
+                        if (layoutSegment.getReplicationMode() == Layout.ReplicationMode.CHAIN_REPLICATION) {
+                            if (layoutStripe.getLogServers().size() < 2) {


should 2 go into some constant or properties file. Although 2 is redundancy so I am not sure !

We need a proper way to specify server properties and this seems like one of those properties.
I'll make it a constant for now.

medhavidhawan · 2017-12-19T02:36:19Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutManagementView.java

+            LayoutModificationException, OutrankedException {
+
+        currentLayout.setRuntime(runtime);
+        sealEpoch(currentLayout);


Yes i think you can. We should not seal if we are going to throw an exception.

medhavidhawan · 2017-12-19T02:41:36Z

test/src/test/java/org/corfudb/integration/WorkflowIT.java

@@ -38,7 +38,7 @@ String getConnectionString(int port) {
    }

    @Test
-    public void AddNodeTest() throws Exception {
+    public void AddRemoveNodeTest() throws Exception {


This should have a comment as to what is being tested

Also I think we need to run some combination of these actions in multiple tests.
Like:

add node -> remove node -> add node -> add node

add node -> add node -> remove node -> add node

add node -> add node -> add node -> remove node -> remove node -> add node -> add node
and so on.
I understand these are slow tests but we need to catch as many issues as we can as early as we can.

I can't test these scenarios in this PR because I need force remove. I will add these test cases to the force remove PR.

medhavidhawan · 2017-12-19T02:49:40Z

test/src/test/java/org/corfudb/runtime/view/LayoutManagementViewTest.java

+
+
+    @Test
+    public void removeNodeTest() throws Exception {


comments please!

medhavidhawan · 2017-12-19T02:51:24Z

test/src/test/java/org/corfudb/runtime/view/LayoutManagementViewTest.java

+        // Remove a node from a two node cluster. Verify that a remove operation won't
+        // cause the loss of redundancy
+        assertThatThrownBy(() -> r.getLayoutManagementView().removeNode(l2,
+                getEndpoint(SERVERS.PORT_1))).isInstanceOf(LayoutModificationException.class);


can we also assess what happened to epoch number ? This one tries an attempt where we could lose redundancy. It would be ok if we do not move epochs when such a thing happens.

Added an extra assert.

Maithem · 2017-12-21T19:39:24Z

Ok, it should pass now.

no2chem · 2017-12-21T19:45:52Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutManagementView.java

+            log.error("removeNode: Bootstrapping sequencer failed due to exception : ", ie);
+            throw new UnrecoverableCorfuInterruptedError(ie);
+        } catch (ExecutionException ee) {
+            log.error("removeNode: Bootstrapping sequencer failed due to exception : ", ee);


Sorry, not handling this doesn't "fix" the problem. At the end of removeNode's execution, what are my guarantees? Is the node removed or not? If the node is guaranteed to be removed, this needs to throw an exception. If not, then how the client should handle it (indeterministic behavior) better be documented.

It actually does.

This method doesn't provide guarantees. I added more documentation.

Maithem · 2017-12-21T20:15:25Z

Ok, added more documentation.

no2chem · 2017-12-21T20:36:39Z

I'm not confident about the design of this PR. It seems that since you have information about the failure/success of the reconfiguration, you should present that information to the client whenever possible. @medhavidhawan?

Maithem · 2017-12-21T20:47:10Z

This is a consequence of the orchestrator stateless design, not this specific PR.

Workflows execute optimistically (i.e. no retry logic). The client requesting workflows should retry on failure. Right now this can happen through observing the layout. Since workflows run asynchronously, the orchestrator server returns a workflow id right away. The orchestrator is stateless so it doesn't persist data on workflow status. Based on those two things, the client can observe the layout after it determines that the workflow is no longer running and retry appropriately.

corfudb-performance · 2017-12-23T02:17:22Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit 4b51cfd.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

Maithem · 2017-12-24T07:09:33Z

@no2chem Addressed all comments

no2chem

See comments.

no2chem · 2017-12-24T07:56:29Z

test/src/test/java/org/corfudb/integration/WorkflowIT.java

@@ -38,7 +38,7 @@ String getConnectionString(int port) {
    }

    @Test
-    public void AddNodeIT() throws Exception {
+    public void AddRemoveNodeTest() throws Exception {


Perhaps this should still be named IT?

no2chem · 2017-12-24T07:57:14Z

runtime/src/main/java/org/corfudb/protocols/wireprotocol/orchestrator/RemoveNodeWorkflow.java

+
+    @Override
+    public List<Action> getActions() {
+        return Arrays.asList(new RemoveNode());


Probably you want Collections.singletonList? Using Arrays here on a non-list is a bit odd

no2chem · 2017-12-24T08:01:38Z

test/src/test/java/org/corfudb/runtime/view/LayoutManagementViewTest.java

+        assertThat(r.getLayoutView().getLayout().getEpoch()).isEqualTo(epoch);
+    }
+
+    void unexceptionalRemove(CorfuRuntime rt, Layout layout, String endpoint) {


Not sure exactly what unexceptional remove is doing here. I think you want to use assertThatCode(() -> {})..doesNotThrowAnyException();

Maithem · 2017-12-24T22:12:29Z

@no2chem Ok, I think its time to get this in.

no2chem

I know its a pain, but we're still missing a few javadoc on LayoutBuilder APIs. I missed these on the last round. In the future if you could check that all public APIs are documented, that would be preferable -- it's easy to miss these things on each pass...

We should also try to have checkstyle catch these violations during CI.

no2chem · 2017-12-25T01:18:47Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutBuilder.java

@@ -50,6 +50,11 @@ public LayoutBuilder clearUnResponsiveServers() {
        return this;
    }

+    public LayoutBuilder setEpoch(long epoch) {


missing javadoc

no2chem · 2017-12-25T01:19:00Z

runtime/src/main/java/org/corfudb/runtime/view/LayoutBuilder.java

@@ -72,6 +77,11 @@ public LayoutBuilder removeUnResponsiveServers(Set<String> endpoints) {
        return this;
    }

+    public LayoutBuilder removeUnResponsiveServer(String endpoint) {


@nonnull, javadoc

corfudb-performance · 2017-12-25T02:16:29Z

Results automatically generated by CorfuDB Benchmark Framework to assess the performance of this pull request for commit 45d3fd9.

*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx+Scan workload, 10 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 1 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 5 threads, Disk mode
*** 0.0% transaction FAILURE rate for NonConflictingTx workload, 10 threads, Disk mode

An interactive dashboard with Pull Request Performance Metrics for ALL cluster types and numbers of threads in run, is available at:
Pull Request #1053 Graphs

An implementation of a remove node. This workflow allows the removal of a node from the cluster.

Maithem · 2017-12-25T02:22:23Z

@no2chem addressed all comments.

no2chem

Ok, LGTM

Maithem added the feature label Dec 7, 2017

Maithem added this to the 0.2.0 milestone Dec 7, 2017

Maithem force-pushed the removeNode branch from 2e23d64 to e90f5d2 Compare December 8, 2017 01:08

Maithem self-assigned this Dec 9, 2017

Maithem force-pushed the removeNode branch from e90f5d2 to aaead2f Compare December 9, 2017 01:43

zalokhan reviewed Dec 9, 2017

View reviewed changes

Maithem force-pushed the removeNode branch from 2ef20dc to 1378e8a Compare December 12, 2017 02:08

Maithem requested a review from medhavidhawan December 12, 2017 02:09

Maithem mentioned this pull request Dec 13, 2017

Remove Node #1039

Closed

Maithem force-pushed the removeNode branch 2 times, most recently from 41739cd to 3b93c48 Compare December 13, 2017 08:30

Maithem force-pushed the removeNode branch from 3b93c48 to 78742ee Compare December 14, 2017 00:38

Maithem force-pushed the removeNode branch from 78742ee to 48fe631 Compare December 14, 2017 02:39

no2chem requested changes Dec 14, 2017

View reviewed changes

Maithem force-pushed the removeNode branch 2 times, most recently from c36fe11 to aebbf0a Compare December 15, 2017 09:16

Maithem force-pushed the removeNode branch from aebbf0a to f4ea902 Compare December 18, 2017 21:39

Maithem commented Dec 18, 2017

View reviewed changes

medhavidhawan reviewed Dec 19, 2017

View reviewed changes

Maithem force-pushed the removeNode branch 3 times, most recently from 71d8cd2 to 4a13fb2 Compare December 20, 2017 02:57

Maithem force-pushed the removeNode branch from 2cc04cc to 0cb3948 Compare December 21, 2017 19:38

no2chem requested changes Dec 21, 2017

View reviewed changes

CorfuDB deleted a comment from no2chem Dec 21, 2017

Maithem force-pushed the removeNode branch from 0cb3948 to 0ae35db Compare December 21, 2017 20:14

CorfuDB deleted a comment from Maithem Dec 21, 2017

Maithem closed this Dec 21, 2017

Maithem reopened this Dec 21, 2017

Maithem force-pushed the removeNode branch 3 times, most recently from 7df975e to 4b51cfd Compare December 23, 2017 01:54

no2chem requested changes Dec 24, 2017

View reviewed changes

Maithem force-pushed the removeNode branch from 4b51cfd to 45d3fd9 Compare December 24, 2017 22:11

no2chem requested changes Dec 25, 2017

View reviewed changes

Remove Node

8c678dd

An implementation of a remove node. This workflow allows the removal of a node from the cluster.

Maithem force-pushed the removeNode branch from 45d3fd9 to 8c678dd Compare December 25, 2017 02:20

no2chem approved these changes Dec 25, 2017

View reviewed changes

Maithem merged commit 5b4ffd0 into master Dec 25, 2017

Maithem deleted the removeNode branch December 25, 2017 02:37

This was referenced Dec 27, 2017

RemoveNodeTest is failing #1116

Closed

Revert "Remove node" #1117

Merged

Remove node #1053

Remove node #1053

Conversation

Maithem commented Dec 7, 2017 • edited by no2chem Loading

Overview

Checklist (Definition of Done):

no2chem commented Dec 7, 2017

coveralls commented Dec 8, 2017

Maithem commented Dec 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corfudb-performance commented Dec 12, 2017

corfudb-performance commented Dec 13, 2017

codecov bot commented Dec 14, 2017 • edited Loading

Codecov Report

corfudb-performance commented Dec 14, 2017

no2chem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corfudb-performance commented Dec 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maithem commented Dec 21, 2017

no2chem Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maithem commented Dec 21, 2017

no2chem commented Dec 21, 2017

Maithem commented Dec 21, 2017

corfudb-performance commented Dec 23, 2017

Maithem commented Dec 24, 2017

no2chem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maithem commented Dec 24, 2017

no2chem left a comment

Maithem commented Dec 7, 2017 •

edited by no2chem

Loading

codecov bot commented Dec 14, 2017 •

edited

Loading

no2chem Dec 21, 2017 •

edited

Loading