Upgrade to 1.12: Segmentation fault from reading the address 0x2492c

[davidspiess]

Today i updated from 1.11 to 1.12. I followed Migrating to RethinkDB 1.12 and almost everything worked fine.

But this query (in express.js with 1.12 node driver) kills now my rethinkdb process and i have no idea why. There is no group_by or grouped_map_reduce in here :(

app.get('/api/quiz/b', function(req, res, next) {
    var conn = app.get('connection');

    // http://stackoverflow.com/questions/20980162/rethinkdb-getting-random-documents-for-each-category
    r.table('b_categories').concatMap(function(cat) {
        return r.table('b_questions').filter({category_id:cat('id')}).sample(cat('random_questions'));
    }).orderBy('id').run(conn, function(err, cursor) {
        if(err) return next(new Error(err.msg));

        cursor.toArray(function(err, questions) {
            if(err) return next(new Error(err.msg));
            res.json(questions);
        });
    });
});

This is the error log in the console:

info: Running rethinkdb 1.12.0 (CLANG 4.2 (clang-425.0.28))...
info: Running on Darwin 13.1.0 x86_64
info: Using cache size of 2130 MB
info: Loading data from directory /Users/david/rethinkdb_data
info: Listening for intracluster connections on port 29015
info: Listening for client driver connections on port 28015
info: Listening for administrative HTTP connections on port 8080
info: Listening on addresses: 127.0.0.1, ::1
info: To fully expose RethinkDB on the network, bind to all addresses
info: by running rethinkdb with the `--bind all` command line option.
info: Server ready
Version: rethinkdb 1.12.0 (CLANG 4.2 (clang-425.0.28))
error: Error in src/arch/runtime/thread_pool.cc at line 341:
error: Segmentation fault from reading the address 0x2492c.
error: Backtrace:
error: Wed Mar 26 21:51:23 2014
error: Exiting.
Abort trap: 6

Any ideas?
Thank you in advance for your help!

[coffeemug]

Hi, this is almost certainly a bug. Sorry you ran into this, we'll fix it ASAP.

Would you be able to share what data you already have in the tables to help us reproduce the problem? If you could e-mail me at slava@rethinkdb.com, I'll help sort it out!

[davidspiess]

Email out :)

[coffeemug]

For the internal folks, the data is on newton in /home/coffeemug/issue_2178/.

@danielmewes @Tryneus -- could you grab this issue?

[coffeemug]

Also, pinging @mlucy, you might be able to grab this too.

[coffeemug]

Ok, @mlucy grabbed this.

[marshall007]

Same issue here, but after the crash rethinkdb refuses to start up again:

info: Running rethinkdb 1.12.0 (CLANG 4.2 (clang-425.0.28))...
info: Running on Darwin 13.1.0 x86_64
info: Using cache size of 1746 MB
info: Loading data from directory /private/var/lib/rethinkdb/data-pipeline
info: Listening for intracluster connections on port 29015
info: Listening for client driver connections on port 28015
info: Listening for administrative HTTP connections on port 8080
info: Listening on addresses: 127.0.0.1, ::1
info: To fully expose RethinkDB on the network, bind to all addresses
info: by running rethinkdb with the `--bind all` command line option.
info: Server ready
Version: rethinkdb 1.12.0 (CLANG 4.2 (clang-425.0.28))
error: Error in src/buffer_cache/alt/evicter.cc at line 124:
error: Guarantee failed: [initialized_]
error: Backtrace:
error: Wed Mar 26 17:00:28 2014

       1: 0   rethinkdb                           0x00000001015dcda2 _ZN2ql33wire_func_serialization_visitor_tD0Ev + 786 at 0x1015dcda2 ()
       2: 0   rethinkdb                           0x00000001010aa62c _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEEiEED0Ev + 2380 at 0x1010aa62c ()
       3: 0   rethinkdb                           0x00000001010aae67 _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEEiEED0Ev + 4487 at 0x1010aae67 ()
       4: 0   rethinkdb                           0x00000001010aa24a _ZN26callable_action_instance_tI21pmap_runner_one_arg_tINSt3__16__bindIM29traversal_progress_combiner_tFviEJPS3_RNS1_12placeholders4__phILi1EEEEEEiEED0Ev + 1386 at 0x1010aa24a ()
       5: 0   rethinkdb                           0x000000010148046c _ZNSt3__16vectorIhNS_9allocatorIhEEE8__appendEm + 15132 at 0x10148046c ()
       6: 0   rethinkdb                           0x00000001010e08fc _ZN16cache_balancer_tD0Ev + 2012 at 0x1010e08fc ()
       7: 0   rethinkdb                           0x00000001010e2c38 _ZNSt3__16vectorIPN18segmented_vector_tIyLm16384EE9segment_tENS_9allocatorIS4_EEE21__push_back_slow_pathIS4_EEvOT_ + 2920 at 0x1010e2c38 ()
       8: 0   rethinkdb                           0x00000001010e48a0 _ZNK3alt13page_loader_t17is_really_loadingEv + 768 at 0x1010e48a0 ()
       9: 0   rethinkdb                           0x00000001010edc8a _ZN13thread_doer_tINSt3__16__bindIMN3alt12page_cache_tEFvyP12ser_buffer_tRK9counted_tI24ls_block_token_pointee_tEEJRPS3_RyS5_SA_EEEE16on_thread_switchEv + 74 at 0x1010edc8a ()
       10: 0   rethinkdb                           0x00000001010a502f _ZNSt3__16__treeINS_4pairIiP22linux_event_callback_tEENS_19__map_value_compareIiS3_NS_4lessIiEELb1EEENS_9allocatorIS4_EEE7destroyEPNS_11__tree_nodeIS4_PvEE + 4255 at 0x1010a502f ()
       11: 0   rethinkdb                           0x00000001010a39fa _ZN14coro_globals_tD2Ev + 1322 at 0x1010a39fa ()
       12: 0   rethinkdb                           0x00000001010a6c64 _ZNKSt3__110__function6__funcINS_6__bindIM9starter_tFvRKNS_8functionIFvvEEEEJPS3_S8_EEENS_9allocatorISC_EES5_E11target_typeEv + 3700 at 0x1010a6c64 ()
       13: 0   libsystem_pthread.dylib             0x00007fff915cd899 _pthread_body + 138 at 0x7fff915cd899 ()
       14: 0   libsystem_pthread.dylib             0x00007fff915cd72a _pthread_struct_init + 0 at 0x7fff915cd72a ()
       15: 0   libsystem_pthread.dylib             0x00007fff915d1fc9 thread_start + 13 at 0x7fff915d1fc9 ()
error: Exiting.
Version: rethinkdb 1.12.0 (CLANG 4.2 (clang-425.0.28))
Crashing while already crashed. Printing error message to stderr.

[mlucy]

I'm on this. I need to get RethinkDB running on an OS X server first. @Davbeer, @marshall007 -- are you both running 10.9?

[coffeemug]

You can just ssh into gantz.

[marshall007]

@mlucy Yep

[mlucy]

Ah, found it. We were incrementing an invalidated iterator. A fix is in review 1373 by @Tryneus .

[mlucy]

Alright, this is in next and 1.12.x. @AtnNn -- can we release tomorrow?

[AtnNn]

@mlucy yes.

[coffeemug]

We might want to wait on a few more bugs (like #2182). Or we can do two releases. @AtnNn should definitely take a vacation after this.

[mlucy]

This bug basically makes it impossible to use 1.12 on that particular version of OS X, so the sooner we get it out the better.

[coffeemug]

Ok, let's do it tomorrow.

[mglukhovsky]

@Davbeer, RethinkDB 1.12.1 was released today, and includes the fix @mlucy made for this issue.

Can you update and let us know if it works for you?

[davidspiess]

Works now 👍 Thank you a lot for helping me out so quickly, you guys are awesome!