Database crashing due to "Guarantee failed"

[dancannon]

After enabled host discovery by default for GoRethink one of the tests started failing pretty consistently (this test only connects to a single server). The host discovery part of the driver runs a change feed query to find what servers the driver can connect to. I enabled logging of every query sent to the DB and have posted the output below (The EOF error is from the server_status query).

I realise that there is not much information in this issue but honestly I dont have any idea what is going on, if you could give me any pointers of what to look for I will try to add more information.

INFO[0000] r.DB("rethinkdb").Table("server_status") [1,[15,[[14,["rethinkdb"]],"server_status"]]]
INFO[0000] r.DB("rethinkdb").Table("server_status").Changes() [1,[152,[[15,[[14,["rethinkdb"]],"server_status"]]]]]
INFO[0000] r.DB("test").TableDrop("table") [1,[61,[[14,["test"]],"table"]]]
INFO[0000] r.DB("test").TableCreate("table") [1,[60,[[14,["test"]],"table"]]]
DEBU[0000] Error discovering hosts EOF, waiting 552.330144ms
FATA[0000] Error finding person: dial tcp 127.0.0.1:28015: connection refused

Running rethinkdb 2.0.1 (CLANG 4.2 (clang-425.0.28))...
Running on Darwin 14.0.0 x86_64
Loading data from directory /Users/danielcannon/rethinkdb_data
warn: Cache size does not leave much memory for server and query overhead (available memory: 618 MB).
warn: Cache size is very low and may impact performance.
Listening for intracluster connections on port 29015
Listening for client driver connections on port 28015
Listening for administrative HTTP connections on port 8080
Listening on addresses: 127.0.0.1, 127.94.0.1, 127.94.0.2, ::1
To fully expose RethinkDB on the network, bind to all addresses by running rethinkdb with the `--bind all` command line option.
Server ready, "Daniels_MacBook_Pro_local_pyy" d36c8870-53c1-4ae5-8497-3c852ae6a251
A newer version of the RethinkDB server is available: 2.0.2. You can read the changelog at <https://github.com/rethinkdb/rethinkdb/releases>.
Version: rethinkdb 2.0.1 (CLANG 4.2 (clang-425.0.28))
error: Error in src/rdb_protocol/artificial_table/caching_cfeed_backend.cc at line 98:
error: Guarantee failed: [dirtiness != dirtiness_t::none_or_some || !dirty_keys.empty()] If nothing is dirty, we shouldn't have gotten here
error: Backtrace:
error: Fri May 29 22:30:56 2015

1: 0 rethinkdb 0x0000000101db308b _Z11deserializeIL17cluster_version_t5EE16archive_result_tP13read_stream_tP17repli_timestamp_t + 539 at 0x101db308b ()
2: 0 rethinkdb 0x00000001019854ec _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEExEED0Ev + 2684 at 0x1019854ec ()
3: 0 rethinkdb 0x0000000101985d27 _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEExEED0Ev + 4791 at 0x101985d27 ()
4: 0 rethinkdb 0x000000010198510a _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEExEED0Ev + 1690 at 0x10198510a ()
5: 0 rethinkdb 0x0000000101b8b75c _Z11deserializeIL17cluster_version_t5EE16archive_result_tP13read_stream_tP13name_string_t + 8268 at 0x101b8b75c ()
6: 0 rethinkdb 0x0000000101bdc1d1 _ZN5boost6detail7variant15visitation_implIN4mpl_4int_ILi0EEENS1_20visitation_impl_stepINS_3mpl6l_iterINS7_6l_itemINS3_5long_ILl3EEEN2ql10changefeed9keyspec_t7range_tENS9_INSA_ILl2EEENSE_7limit_tENS9_INSA_ILl1EEENSE_7point_tENS7_5l_endEEEEEEEEENS8_ISK_EEEENS1_9move_intoEPvNS_7variantISF_SH_SJ_NS1_5void_ESU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_E18has_fallback_type_EEENT1_11result_typeEiiRSX_T2_NS3_5bool_ILb0EEET3_PT_PT0_ + 1457 at 0x101bdc1d1 ()
7: 0 rethinkdb 0x0000000101bddc73 _ZN26callable_action_instance_tINSt3__16__bindIMN40caching_cfeed_artificial_table_backend_t19caching_machinery_tEFvN14auto_drainer_t6lock_tEEJPS3_S5_EEEE10run_actionEv + 67 at 0x101bddc73 ()
8: 0 rethinkdb 0x000000010197ad28 _ZN22threaded_context_ref_tC2Ev + 1304 at 0x10197ad28 ()
error: Exiting.
[1] 70682 trace trap rethinkdb

[timmaxw]

Hi @dancannon. Sorry you ran into this, and thanks for the bug report.

@AtnNn Will there be any more point releases before 2.1? This looks pretty easy to fix; the solution is to just delete the offending assertion, I think.

[dancannon]

Glad its an easy fix :) I was just quite surprised to see the crash after such a small change (especially since it only affected one test case out of the ~200 in gorethink)

[AtnNn]

Yes, there will be another point release.

[danielmewes]

@timmaxw could you put the fix up for code review please?

[timmaxw]

Fixed on v2.0.x in 6b14db2.

[timmaxw]

Cherry-picked into next in 448cde0.

[dancannon]

Thanks for fixing so quickly 👍