etcd_test fails with an assertion in gRPC

[prasad-joshi]

I had same problems during compilation as mentioned on #2

Fixed them by changing cmake file like this

prasad@prasad-Inspiron-5567:~/sources/nokia/etcd-cpp-apiv3$ git diff
diff --git a/tst/CMakeLists.txt b/tst/CMakeLists.txt
index e0ce264..53a515b 100644
--- a/tst/CMakeLists.txt
+++ b/tst/CMakeLists.txt
@@ -1,9 +1,12 @@
-find_path(CATCH_INCLUDE_DIR NAMES catch.hpp)
+find_path(CATCH_INCLUDE_DIR NAMES catch.hpp PATHS ${CMAKE_SOURCE_DIR})
 include_directories(${CATCH_INCLUDE_DIR})
 
 add_executable(etcd_test EtcdTest.cpp EtcdSyncTest.cpp WatcherTest.cpp)
 set_property(TARGET etcd_test PROPERTY CXX_STANDARD 11)
 
-target_link_libraries(etcd_test etcd-cpp-api)
+target_link_libraries(etcd_test
+       etcd-cpp-api
+       pthread
+)
 
 add_test(etcd_test etcd_test)

After making above changes the library compiles however, make test fails

root@ubuntu16:~/workspace/nokia/etcd-cpp-apiv3/build# make test
Running tests...
Test project /home/prasad/workspace/nokia/etcd-cpp-apiv3/build
    Start 1: etcd_test
1/1 Test #1: etcd_test ........................***Failed    0.76 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.78 sec

The following tests FAILED:
	  1 - etcd_test (Failed)
Errors while running CTest
Makefile:105: recipe for target 'test' failed
make: *** [test] Error 8

It seems like an assertion in gRPC failed.

root@ubuntu16:~/workspace/nokia/etcd-cpp-apiv3/build# ./tst/etcd_test  
E0522 13:22:31.865897860   10686 channel_cc.cc:136]          assertion failed: GRPC_CALL_OK == grpc_call_start_batch(call->call(), cops, nops, ops, nullptr)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
etcd_test is a Catch v1.5.4 host application.
Run with -? for options

-------------------------------------------------------------------------------
wait for a value change
-------------------------------------------------------------------------------
/home/prasad/workspace/nokia/etcd-cpp-apiv3/tst/EtcdTest.cpp:277
...............................................................................

/home/prasad/workspace/nokia/etcd-cpp-apiv3/tst/EtcdTest.cpp:277: FAILED:
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases:  14 |  13 passed | 1 failed
assertions: 108 | 107 passed | 1 failed

[zhangwuliang]

I have also encountered this problem, Have you solved it? How to solve?

[bjeremy]

I also have the same problem... Has anyone found out what the reason is?

I think I may know what is happening... but not having gone through grpc in depth, my solution may be naive.

Anyway, in AsyncWatchAction it looks like the write operation is at time resulting in a
GRPC_CALL_ERROR_TOO_MANY_OPERATIONS causing the ASSERT. Which led me to believe that for some reason there are outstanding transactions in the stream... why? who knows.

etcdv3::AsyncWatchAction::AsyncWatchAction(...)
{
isCancelled = false;
stream = parameters.watch_stub->AsyncWatch(&context,&cq_,(void*)"create");
....
stream->Write(watch_req, (void*)"write");
stream->Read(&reply, (void*)this);
}

I changed it to read any events from the queue first before the write.

etcdv3::AsyncWatchAction::AsyncWatchAction(...)
{
isCancelled = false;
stream = parameters.watch_stub->AsyncWatch(&context,&cq_,(void*)"create");
....
gpr_timespec deadline;
deadline.clock_type = GPR_TIMESPAN;
deadline.tv_sec = 0;
deadline.tv_nsec = 10000000;
auto status = cq_.AsyncNext(&got_tag, &ok, deadline);

if (status != CompletionQueue::SHUTDOWN)
{
stream->Write(watch_req, (void*)"write");
stream->Read(&reply, (void*)this);
}
}

Is this correct? Not sure yet... but I stopped getting the ASSERT... and I can setup watchers and things work for now (I'm only starting to prototype basic stuff)

However, running the etcd_test seems to hang on the last testcase... I haven't looked at it yest to see why, may be on cleanup or shutdown of the queue, possibly I'll run into a problem when I try and cancel a watcher. Anyway, I just thought I would throw this out there for now and see if this helps someone figure out a better solution.

UPDATE:

The watcher issue was that it was hanging on the watcher->Cancel()... I seemed to get that working by changing the below method... again.. not sure of the side effects yet, but right now the watcher seems to be working without deadlock or ASSERTs.

void etcdv3::AsyncWatchAction::waitForResponse(std::function<void(etcd::Response)> callback)
{
...
if(got_tag == (void*)"writes done") <-- here
{
isCancelled = true;
break; <-- here
}
...
}
}
}

[siredmar]

Same issue here. Testing with the modifications given by bjeremy didn't gave better results. Is this project still maintained?

[chenyicong]

@siredmar：
actually, you can change the code which provided by bjeremy to below：
etcdv3::AsyncWatchAction::AsyncWatchAction(...)
{
isCancelled = false;
stream = parameters.watch_stub->AsyncWatch(&context,&cq_,(void*)"create");
....
gpr_timespec deadline;
deadline.clock_type = GPR_TIMESPAN;
deadline.tv_sec = 0;
deadline.tv_nsec = 10000000;
auto status = cq_.AsyncNext(&got_tag, &ok, deadline);

• if (status != CompletionQueue::SHUTDOWN) ----delete

• if (status == CompletionQueue::GOT_EVENT) ---add
  {
  stream->Write(watch_req, (void*)"write");
  stream->Read(&reply, (void*)this);
  }
  }

because you can see GRPC code， only the GOT_EVENT staus can do any stream action：
/// Tri-state return for AsyncNext: SHUTDOWN, GOT_EVENT, TIMEOUT.
enum NextStatus {
SHUTDOWN, ///< The completion queue has been shutdown and fully-drained
GOT_EVENT, ///< Got a new event; \a tag will be filled in with its
///< associated value; \a ok indicating its success.
TIMEOUT ///< deadline was reached.
};

[sighingnow]

Has been fixed now, by #16.

[sighingnow]

Thanks everyone who are interested in this library!