Test what happens when you mix sindex creation with rapid resharding.

[jdoliner]

I don't have a particular reason to think this but I suspect that there are a few bugs that will pop up if you create a sindex and then rapidly reshard the store. Mostly just because there's a lot going on there and that tests the most complicated parts of the code all at the same time.

[coffeemug]

@jdoliner -- how much work have we done towards this? If there are bugs here, we should find out earlier rather than later -- how much testing did we do for sindexes on a cluster? I can probably find the time to help here if I'm needed.

[coffeemug]

@jdoliner -- ping.

[jdoliner]

So I've been working on writing a test for this but our integration tests don't really support reql tests. And the python driver doesn't have support for secondary index gets yet so I'm sort of unsure of the right way to go about this right now. We can probably just hack something together though.

[coffeemug]

Well, I was actually thinking of doing by hand a few dozen times to see what happens. If it passes this dumb test, we can think of something more clever.

[jdoliner]

Seems reasonable as a start.

[jdoliner]

Alright I have uncovered 2 bugs using this #727 and #728. I recall seeing one other crash that looked very sindex related but attempts I stupidly didn't copy down the backtrace at the time and have completely failed to reproduce it since. Mostly it's hard to reproduce because #727 and #728 always come up instead. This issue is going to be inactive while I work on the other on those 2 issues since it's not telling me anything new as long as they exist.

[coffeemug]

@jdoliner -- could we move this out of this sprint and into 1.5? Unless you can fix this in the next day or so, we'll need another sprint to fix the issues, and we'll need to retest things. So we can't close this issue quite yet, but we shouldn't keep it either.

[jdoliner]

Yeah makes sense to me.

[coffeemug]

Done.

[jdoliner]

So this has been tested and several issues were found and fixed. We're not actively expanding these tests anymore so I don't see much point in keeping this issue open.