Database failover #203

Open
cirruspath opened this Issue Jun 11, 2012 · 20 comments

Comments

Projects
None yet
7 participants
@ghost

ghost commented Jun 11, 2012

Hi. I have a master/slave setup already working with my redis database, and instead of the "round-robin" approach that is currently implemented in the config file, do you have any suggestions for implementing failover such that if the connection to host #1 fails, it moves to host #2? Same/similar configuration envisioned.

Owner

michael-grunder commented Jun 28, 2012

Hey,

Are you talking about the RedisArray functionality, or just generally.

The way I usually handle this sort of problem, is by wrapping my phpredis calls on the PHP side. I don't wrap calls to actual data functions themselves because that introduces too much of a performance penalty, but rather to the process of connecting.

We actually use a seperate Redis instance that containes our nodes, and then attempt to get a connection, failing over if one is down.

@ghost

ghost commented Jun 28, 2012

Hi. Sorry, I should have been more specific. This is wrt using Redis as a session store and the session.save_path param in the php.ini file.

Today, you can pass multiple Redis instances in, but as far as I understand, phpredis uses those connections in a round robin fashion.

We need to be able to support failover on the session store.

Thanks,
Ryan

~ sent from the road ~

On Jun 27, 2012, at 5:39 PM, Michael Grunderreply@reply.github.com wrote:

Hey,

Are you talking about the RedisArray functionality, or just generally.

The way I usually handle this sort of problem, is by wrapping my phpredis calls on the PHP side. I don't wrap calls to actual data functions themselves because that introduces too much of a performance penalty, but rather to the process of connecting.

We actually use a seperate Redis instance that containes our nodes, and then attempt to get a connection, failing over if one is down.


Reply to this email directly or view it on GitHub:
nicolasff#203 (comment)

Owner

michael-grunder commented Jun 28, 2012

I don't think phpredis supports this at the present moment. Right now I think if the connection times out it will just fail.

@nicolasff If you think failover is a reasonable idea, I can look into what it would take to make that work.

Contributor

nicolasff commented Jun 28, 2012

@michael-grunder already looking into it, thanks!

Contributor

nicolasff commented Jun 28, 2012

@cirruspath the way sessions work is by calling a specific read function in the extension when session_start() is called in PHP, and another write function at the end of the PHP script.
So implementing a failover for reads is pretty straightforward, but I'm not sure about the behaviour for the session write since it is very likely that a failed read from the master would be followed by a failed write at the end of the PHP script.

Should you not expect the write to fail in that case, and just let it fail? I assume you don't want to write the session to the slave, do you? That could lead to inconsistencies when the master comes back...

@michael-grunder any suggestions? My changes are on the session-failover branch.

Owner

michael-grunder commented Jun 28, 2012

@nicolasff I think you're probably correct to let the session handler fail on write. The only real way I can see around this would add quite a bit of complexity to phpredis (e.g. keep track of session server states, etc). I don't even know how we could do this properly, and if the use case was using an actual master->slave configuration, any attempt on our end to keep such context could get nuked when the master came back up.

Truth be told, however, I haven't ever used PHP sessions at all, so I'll have to dig into it a little deeper to come up with any good ideas. :)

@ghost

ghost commented Jun 29, 2012

@nicolasff - I don't know the implementation of it in phpredis, but if a
read fails, can you attempt a connection to the secondary failover host and
do the read there? We have a master/slave configuration in place, so the
data SHOULD be the same - and honestly, our sessions don't change much once
they're logged in.

On Thu, Jun 28, 2012 at 3:06 PM, Michael Grunder <
reply@reply.github.com

wrote:

@nicolasff I think you're probably correct to let the session handler
fail on write. The only real way I can see around this would add quite a
bit of complexity to phpredis (e.g. keep track of session server states,
etc). I don't even know how we could do this properly, and if the use case
was using an actual master->slave configuration, any attempt on our end to
keep such context could get nuked when the master came back up.

Truth be told, however, I haven't ever used PHP sessions at all, so I'll
have to dig into it a little deeper to come up with any good ideas. :)


Reply to this email directly or view it on GitHub:
nicolasff#203 (comment)

Contributor

nicolasff commented Jun 29, 2012

@cirruspath yes, that's exactly what I did for the read path.
But at the end of a PHP script that uses sessions, the session data is written back to its data store which here would be a Redis write to the master. If we got the session data from the slave because the master was down, it is very likely that writing the data back would fail as well; if we write it to the slave we'll end up with inconsistencies. Does that make sense?

Consider the following script:

<?php

session_start(); /* this triggers a read on the master, which could fail and read from the slave instead. */

// do whatever with the session data

/** At the end of the script, the session data is written to Redis automatically,
  * whether $_SESSION has changed or not.
  * This will try to write to the master, and probably fail if the read had failed.
  * Writing the session to the slave leads to inconsistencies.
  */
?>

I'm keen on adding this feature, but I'm afraid it will still be kinda limited.

@ghost

ghost commented Jun 29, 2012

Got it.

You know, our sessions don't change much once they're established. We use
sessions to store information that we gather from an external service
primarily, so not being able to write it back isn't the end of the world.
In the case where we didn't have a session or it was incomplete, I'd want
that session data to be written to where we got it from -- and if there are
inconsistencies, it will basically mean that we have to refresh that info
-- which is likely what we'd need to do since it was missing/incomplete in
the first place.

So... I say write it back to the slave.

On Fri, Jun 29, 2012 at 7:55 AM, Nicolas Favre-Felix <
reply@reply.github.com

wrote:

@cirruspath yes, that's exactly what I did for the read path.
But at the end of a PHP script that uses sessions, the session data is
written back to its data store which here would be a Redis write to the
master. If we got the session data from the slave because the master was
down, it is very likely that writing the data back would fail as well; if
we write it to the slave we'll end up with inconsistencies. Does that make
sense?

Consider the following script:

I'm keen on adding this feature, but I'm afraid it will still be kinda
limited.


Reply to this email directly or view it on GitHub:
nicolasff#203 (comment)

@nicolasff nicolasff added a commit that referenced this issue Jul 4, 2012

@nicolasff nicolasff Write session data to master or slave (#203).
I had to introduce a "nothrow" flag in the RedisSock object in order
to avoid throwing exceptions in the context of a session write.
When PHP finishes and closes the session, it doesn't consider exceptions
to be valid and simply bails out without giving the caller a chance to
catch it. I replaced zend_throw_exception with redis_throw_exception which
checks for this flag and just returns if we are in a session close context.
e4ee953
Contributor

nicolasff commented Jul 4, 2012

@cirruspath the changes are in, you can try out the feature using the "session-failover" branch on GitHub.
@michael-grunder does the exception change in e4ee953 look ok to you?

@ghost

ghost commented Jul 4, 2012

Nicolas - Thank you!

Let me review and I'll let you know if I have any questions.

Thanks,
Ryan

Ryan Huff
CEO, Co-Founder
Cirruspath, LLC.
rhuff@cirruspath.com

(949) 354-4134

http://twitter.com/#%21/cirruscg
Check out our latest app: Cirrus Insight http://www.cirrusinsight.com/ --
Salesforce integration for Gmail users.

On Wed, Jul 4, 2012 at 2:24 PM, Nicolas Favre-Felix <
reply@reply.github.com

wrote:

@cirruspath the changes are in, you can try out the feature using the
"session-failover" branch on
GitHub.
@michael-grunder does the exception change in e4ee953 look ok to you?


Reply to this email directly or view it on GitHub:
nicolasff#203 (comment)

Owner

michael-grunder commented Jul 5, 2012

@nicolasff Looks good to me. Similar to the destructor issue.

It works great, but if I understand correct . The "failover=" param only works with the short "host:port" , and not with the "tcp://" scheme ?

There only seems to be a issue when "auth=" is used, in that case : auth must be the same on both servers. And when the failover happends, the connection goes to the slave. But only the AUTH and GET are issued, and no new keys are SET.

At the same time the following error is logged :
[error] [client 127.0.0.1] PHP Fatal error: Uncaught exception 'RedisException' with message 'Connection closed' in [no active file]:0\nStack trace:\n#0 {main}\n thrown in [no active file] on line 0

@nicolasff would this be available soon in a stable release?

Owner

michael-grunder commented Feb 14, 2013

Hey there,

There are a few feature branches that we want to get merged into master soon. I'm curious if you (or anyone else) has had success with the changes. @cirruspath, how did it work out for you?

@nicolasff You at all worried about these changes?

Cheers guys :)
Mike

snikkers commented Mar 8, 2013

Is it in a specific branch? I still have a test setup, so I can test it out .

Owner

michael-grunder commented Mar 8, 2013

I'm pretty sure it's in session-failover

Cheers,
Mike

Ivun commented May 5, 2014

Hello guys! Just reminding you about this feature. Any progress on that?

Hi
any news on this topic?
You're supporting cluster setup but not master/slave?

michael-grunder self-assigned this Mar 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment