Handling lost connection error

[glebskalinskii]

Hi,

I'm trying to test what happens in case of Redis outage. If I shutdown redis server there's an error being thrown that you can see bellow, and being unhandled it breaks the app. Can you please advise on how to handle and recover from lost connection?

Basic setup:

import Redis from 'ioredis'
import { Queue, QueueScheduler, Worker } from 'bullmq'

const connection = new Redis({
  port: 6379,
  host: 'localhost'
})

const queueScheduler = new QueueScheduler('test')
const queue = new Queue('test', { connection })

  const worker = new Worker(
    'test',
    async job => {
        console.log(job.data)
    },
    {
      connection,
      limiter: {
        max: 1,
        duration: 1000
      }
    }
  )

[ERROR] 13:21:32 Error: connect ECONNREFUSED 127.0.0.1:6379
Error: connect ECONNREFUSED 127.0.0.1:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1128:14)
/Users/fo/Documents/_projects/bar/node_modules/winston/lib/winston/logger.js:307
      throw ex;
      ^

NodeError: write after end
    at writeAfterEnd (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:257:12)
    at PassThrough.Writable.write (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:301:21)
    at File.log (/Users/fo/Documents/_projects/bar/node_modules/winston/lib/winston/transports/file.js:185:34)
    at ExceptionStream._write (/Users/fo/Documents/_projects/bar/node_modules/winston/lib/winston/exception-stream.js:48:29)
    at doWrite (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:405:139)
    at writeOrBuffer (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:394:5)
    at ExceptionStream.Writable.write (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:303:11)
    at DerivedLogger.ondata (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:662:20)
    at DerivedLogger.emit (events.js:215:7)
    at DerivedLogger.EventEmitter.emit (domain.js:476:20)
    at addChunk (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:297:12)
    at readableAddChunk (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:279:11)
    at DerivedLogger.Readable.push (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:240:10)
    at DerivedLogger.Transform.push (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:139:32)
    at DerivedLogger._transform (/Users/fo/Documents/_projects/bar/node_modules/winston/lib/winston/logger.js:305:12)
    at DerivedLogger.Transform._read (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:177:10)
Emitted 'error' event on PassThrough instance at:
    at writeAfterEnd (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:259:10)
    at PassThrough.Writable.write (/Users/fo/Documents/_projects/bar/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:301:

[denvolok]

I've faced the same issue. I managed to stop crushes only after adding error handlers for each instance of Queue, Worker, QueueScheduler. Also, you can use QueueEvents instead of attaching an error handler for each Worker.

Example:

queue.on("error", (error) => console.log(error));
queueScheduler.on("error", (error) => console.log(error));
worker.on("error", (error) => console.log(error));

[manast]

@glebskalinskii is the error stack you are showing really coming from BullMQ? I have tested this several times recently and it works correctly, it will reconnect when the connection is re-stablished and continue processing jobs.

[manast]

(also we do not use winston in BullMQ, so it seems your code is triggering that exception).