Running too many queries with runQueries at once will raise "exception: too few bytes." #9

Open
adimit opened this Issue Nov 23, 2012 · 4 comments

2 participants

@adimit
Collaborator

(I'm reporting this more as a reminder to myself and so that people can find it who might get bitten by this bug. I'll have to try and fix this later, but it might not be trivial.)

(There's also #4, which might be similar to this, but it's definitely a different bug.)

The title already says it all: if you run more than 32 queries at once with runQueries, you will get this nasty exception.

> let q = Query { queryString = "foo", queryIndexes = "*", queryComment = "" }
> runQueries searchConf (replicate 32 q)
-- … good stuff
> runQueries searchConf (replicate 33 q)
Error 1 "*** Exception: too few bytes. Failed reading at byte position 1650549796

This appears to happen before queries get sent to sphinx, so it seems a problem on our end. I won't have time to fix this immediately, but it's on my TODO-list. My own application frequently sends a bunch of queries like this, so I'll have to work around it (it's easy, just use Data.List.Split's chunksOf function.)

I'll probably have time to look at it, say, next weekend or so, hopefully earlier.

@gregwebs
Owner

Why do you send 32 queries at once? I don't know if sphinx is designed for that: I wouldn't be surprised if there was a limit in sphinx.

@adimit
Collaborator

Because I might be mistreating sphinx a bit, but, in essence, I'm sending a lot of very similar queries.

In any case, if it should turn out that Sphinx does not support so many queries, we should at least provide a more helpful error message.

@gregwebs
Owner

One option is just to add a note to the docs. I send a couple similar queries because it was quick to get working, but I think I could figure out how to get it down to one query: likely there is a way to restructure what you are doing

@adimit
Collaborator

It really depends on the problem with this one. Right now, I'm trying to research the effectiveness of different query types for certain tasks, so I send a bunch (couple hundred, actually) queries at once, noting and later evaluating their respective performance. So… I can't really restructure it, but few people will want to use sphinx the way I do :-P.

I also think that we should at least add a doc note and a more descriptive error message. Though I'm pretty sure that one should be able to fix this, if sphinx supports such abuse at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment