Core Dump #3393
Comments
|
Thank you for the bug report @internalfx . @mlucy can you take a look? |
|
I will check on the possibility of giving data... I can at least give you the script that was running.... sync = require('synchronize')
async = require('async')
_ = require('lodash')
r = require('rethinkdb')
moment = require('moment')
numeral = require('numeral')
inflect = require('inflection')
deep = require('deep-diff')
module.exports = (grunt) ->
grunt.registerTask('devsync', '', ->
done = this.async()
process.maxTickDepth = 9999999
json_deserializer = (key, value) ->
if _.isString(value)
regexp = /^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d.\d\d\dZ$/.exec(value)
if regexp
return new Date(value)
return value
sync.fiber(->
conn = sync.await(r.connect({host: 'localhost', db: 'aamcrm'}, sync.defer()))
#Create Tables
table_list = sync.await(r.db('aamcrm').tableList().run(conn, sync.defer()))
table_list_dev = sync.await(r.db('aamcrm_dev').tableList().run(conn, sync.defer()))
for table in table_list
if _.contains(table_list_dev, table)
sync.await(r.db('aamcrm_dev').tableDrop(table).run(conn, sync.defer()))
table_list_dev = sync.await(r.db('aamcrm_dev').tableList().run(conn, sync.defer()))
console.log "REBUILDING TABLES..."
for table in table_list
unless _.contains(table_list_dev, table)
sync.await(r.db('aamcrm_dev').tableCreate(table).run(conn, sync.defer()))
console.log "REBUILDING INDEXES..."
for table in table_list
indexes = sync.await(r.db('aamcrm').table(table).indexList().run(conn, sync.defer()))
for index in indexes
indexes_dev = sync.await(r.db('aamcrm_dev').table(table).indexList().run(conn, sync.defer()))
unless _.contains(indexes_dev, index)
index_obj = _.first(sync.await(r.db('aamcrm').table(table).indexStatus(index).run(conn, sync.defer())))
sync.await(r.db('aamcrm_dev').table(table).indexCreate(
index_obj.index, index_obj.function, {geo: index_obj.geo, multi: index_obj.multi}
).run(conn, sync.defer()))
for table in table_list
console.log "CLONING DATA:", table
sync.await(r.db('aamcrm_dev').table(table).insert(r.db('aamcrm').table(table)).run(conn, sync.defer()))
sync.await(conn.close(sync.defer()))
done()
)
)
It died on cloning one of tables...I'm sure I just overloaded something. It had to flood the feed like crazy. |
|
Thank you for the script @internalfx . Even under the heaviest possible load, the server shouldn't crash. So this is definitely a bug and we'll try to fix it asap. |
|
agreed... I think I can at least get by for now by not doing dumb stuff like rewriting the WHOLE TABLE while listening on a change feed. Keep up the great work guys! |
|
Working on this now. Thanks for the bug report! |
|
Alright, this is a bug similar to other bugs we've had in the past: the What makes this bug different from the last few is that the lock we're acquiring is superfluous. It looks like we're correctly acquiring an auto drainer lock without crashing in (In other words, this bug can't just be solved by moving the mailbox below the The immediate solution is probably to make Longer-term, I think it might be nice to have an alternate interface to auto drainers. I understand why we don't let people acquire a lock on a draining drainer; the chance of infinitely delayed destruction is too high (and this is especially the case when the drainer is protecting a callback that can spawn jobs). But it would be really nice if I could write @larkost -- How hard would it be to write a test to duplicate this scenario (high write load, all of it going to a changefeed, and then the changefeed gets disconnected in a variety of different ways)? |
|
I would be OK with the thing you're proposing, although I almost never want that pattern in my own code. We should check if the C++ standard guarantees that it's legal to access an object during that object's destructor. |
|
Quoting the C++ standard:
Since the destructor of |
|
We violate that rule pretty freely, for the record. |
|
I looked into this more, and section 12.7 of https://isocpp.org/files/papers/N3690.pdf says:
Which seems to give us a bit more leeway. In practice, since the destructor itself needs to be able to manipulate the object freely, I have trouble imagining an optimization that would allow the destructor to behave correctly in all circumstances but prevent a function running in another coroutine from accessing the object safely. |
|
@mlucy I think that interface would be useful, though possibly a little unsafe. |
|
@danielmewes -- that's a good point, actually. I retract my proposed interface. |
|
The fix is in CR 2381 by @danielmewes, although we don't yet have a good way to test that it actually fixes the crashing bug in question. |
|
This is in |
|
Since 1.16 is at least a month away, I think we should do another point release. @AtnNn Can you start making preparations for a 1.15.3 point release? We would include this, #3346 , #3253 , and should also change libcurl to be linked statically ( |
|
@mlucy is this in v1.15.x? |
|
@danielmewes: merging this didn't work too well because |
|
@internalfx The fix has been release in RethinkDB 1.15.3 |
|
@danielmewes @mlucy @AtnNn You guys never cease to impress. |
Not sure what to make of it exactly, I do use change feeds....
Server was under high load when it happened.
The text was updated successfully, but these errors were encountered: