Crawler stop or idle after a while

[ValliMoceniche]

hello

i found that redis crawler will idle after a while,but there are a lot of item still in redis

strace show this:

strace -p 12879

gettimeofday({1421031574, 300673}, NULL) = 0
gettimeofday({1421031574, 301008}, NULL) = 0
epoll_wait(6, {}, 2, 31826) = 0
gettimeofday({1421031606, 142810}, NULL) = 0
gettimeofday({1421031606, 142931}, NULL) = 0
gettimeofday({1421031606, 143139}, NULL) = 0
gettimeofday({1421031606, 143261}, NULL) = 0
gettimeofday({1421031606, 143417}, NULL) = 0
epoll_wait(6, {}, 2, 356) = 0
gettimeofday({1421031606, 500132}, NULL) = 0
gettimeofday({1421031606, 500389}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1159, ...}) = 0
write(9, "2015-01-12 11:00:06+0800 [Level6"..., 107) = 107
gettimeofday({1421031606, 501373}, NULL) = 0
gettimeofday({1421031606, 501497}, NULL) = 0
gettimeofday({1421031606, 501725}, NULL) = 0

always epoll_wait

log show as:

INFO: Crawled 369 pages (at 0 pages/min), scraped 369 items (at 0 items/min)
INFO: Crawled 369 pages (at 0 pages/min), scraped 369 items (at 0 items/min)
INFO: Crawled 369 pages (at 0 pages/min), scraped 369 items (at 0 items/min)
INFO: Crawled 369 pages (at 0 pages/min), scraped 369 items (at 0 items/min)

server info:
server A: redis
server B: crawler 1
server C: crawler 2

spider is very simple code similar to example-project

[rmax]

Sorry for the very late response. What's your redis server version?

By default the RedisSpider waits infinitely for new requests, but if you say there are requests in the queue, then something is not right.

[sandeepsingh]

Hi, Even i am getting this problem, please post the solution for this issue.
Thanks