Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If connection to db lost then deno process die and pup/sub not resuming. #83

Closed
devalexqt opened this issue May 29, 2020 · 7 comments · Fixed by #92
Closed

If connection to db lost then deno process die and pup/sub not resuming. #83

devalexqt opened this issue May 29, 2020 · 7 comments · Fixed by #92
Labels
bug Something isn't working enhancement New feature or request

Comments

@devalexqt
Copy link

devalexqt commented May 29, 2020

How to handle connection (and other) DB error?


import { connect } from "https://denopkg.com/keroxp/deno-redis/mod.ts"

const redis = await connect({
  hostname: "127.0.0.1",
  port: 6379,
})

const redisSub = await connect({
    hostname: "127.0.0.1",
    port: 6379,
  })

const redisPub = await connect({
    hostname: "127.0.0.1",
    port: 6379,
})



  setInterval(async ()=>{
      console.log(new Date())
      try{
            console.log("==>redis get:",await redis.get("test"))
            await redisPub.publish("test",JSON.stringify({time:new Date()}))
      }catch(err){
          console.log("==>error:",err)
      }   
  },3000)

  var sub = await redisSub.subscribe("test")

  for await (const { channel, message } of sub.receive()) {
    console.log(">>>>>>>>>>>message from redis sub, channel:",channel,", message",message)
  }

Then I kill redis db process on server (emulating of problem with connections) then this app just die...

error: Uncaught Error: Invalid state
  throw new Error("Invalid state");
        ^
    at readLine (https://raw.githubusercontent.com/keroxp/deno-redis/master/io.ts:102:9)
    at async readArrayReply (https://raw.githubusercontent.com/keroxp/deno-redis/master/io.ts:141:16)
    at async RedisSubscriptionImpl.receive (https://raw.githubusercontent.com/keroxp/deno-redis/master/pubsub.ts:64:20)
    at async file:///Users/alex/Documents/Material%20Design%20Web/webserver/test_deno.js:85:39

And if use try/catch, after I restart DB on server, redis.get, pub, sub not resume connection and not working.

  try{
  var sub = await redisSub.subscribe("test")

  for await (const { channel, message } of sub.receive()) {
    console.log(">>>>>>>>>>>message from redis sub, channel:",channel,", message",message)
  }
}catch(err){console.log("==>sub error:",err)}
==>error: BrokenPipe: Broken pipe (os error 32)   
    at unwrapResponse ($deno$/ops/dispatch_minimal.ts:63:11)
    at Object.sendAsyncMinimal ($deno$/ops/dispatch_minimal.ts:106:10)
    at async Object.write ($deno$/ops/io.ts:65:18)
    at async BufWriter.flush (https://deno.land/std@v0.51.0/io/bufio.ts:475:25)
    at async sendCommand (https://raw.githubusercontent.com/keroxp/deno-redis/master/io.ts:62:3)

So how to handle pub/sub error and resume connection and working state?

@devalexqt devalexqt changed the title If connection to db lost then deno process die. If connection to db lost then deno process die and pup/sub not resuming. May 29, 2020
@devalexqt
Copy link
Author

Any news?

@Terkwood
Copy link
Contributor

Nicely documented! I think the next step here is for another person to reproduce this(probably not so hard), and make a suggestion on how to improve it.

sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 16, 2020
@sebastienfilion
Copy link
Contributor

sebastienfilion commented Jun 16, 2020

As a first step, I created a test to reproduce the issue that @devalexqt reported. If it makes sense, I can look into suggesting a fix. #92

sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 16, 2020
@uki00a
Copy link
Member

uki00a commented Jun 21, 2020

Problem

I think the problem is that once connection is lost, the subscriber never resumes.

Solutions

To resolve the above problem, I think that deno-redis should do the following things:

  • provides a way to reconnect the lost connection.
  • resumes (all?) subscribers when reconnected.

What about the other libraries?

ioredis provides the auto-reconnect and auto-resubscribe features:

new Redis({
  maxRetriesPerRequest: 5,
  autoResubscribe: true, // When reconnected, the subscriber will also resume.
  retryStrategy: times => computeNextTimeout(times),
});

go-redis

PubSub automatically reconnects to Redis Server and resubscribes to the channels in case of network errors.

https://pkg.go.dev/github.com/go-redis/redis@v6.15.8+incompatible?tab=doc#PubSub

@uki00a
Copy link
Member

uki00a commented Jun 21, 2020

@devalexqt @Terkwood @sebastienfilion I'm sorry for the delay response.
I've organized my thoughts.
If you have any other ideas, please leave comments! 😄

@Terkwood
Copy link
Contributor

I vote to resume all subscribers when reconnected.

If others like the idea of auto-reconnect, that sounds good to me, too.

@uki00a uki00a added bug Something isn't working enhancement New feature or request labels Jun 21, 2020
@sebastienfilion
Copy link
Contributor

@Terkwood that was my idea. I’ve been working on a solution for the past few days; finishing testing. I hope I can push tomorrow for you all to see.

sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 23, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 24, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 24, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 25, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 25, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 26, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 28, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
sebastienfilion added a commit to sebastienfilion/deno-redis that referenced this issue Jun 28, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: denodrivers#83
@uki00a uki00a linked a pull request Jun 28, 2020 that will close this issue
uki00a pushed a commit that referenced this issue Jun 28, 2020
At the moment, a client will die if the server is killed or the connection is lost. This is a problem especially with clients subscribed to a channel.
Since the Writer/Reader buffers are passed around to I/O functions, it is impossible to reconnect a Redis client.
Having a connection object allows to easily replace the buffers.

Resolves: #83
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants