Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Live Server Upgrades #106
So... Cloud Haskell isn't likely to support the same kind of hot code upgrade as Erlang any time soon, and for good reason. That model simply doesn't make sense for a strongly typed language like Haskell, and there's nothing in the RTS to support it either - one of our aims is to avoid changing the RTS and to keep CH as a library.
After looking at http://hackage.haskell.org/packages/archive/plugins/184.108.40.206/doc/html/System-Plugins-Load.html, I can see that it's possible to dynamically load new modules, but there's really nothing in that approach that fits the concept of upgrading an existing (i.e., already loaded) module cleanly.
I do however, think that we can approximate many of the benefits of Erlang's rolling upgrades without actually doing any runtime code changes. At a very high level, my idea is that we provide a mechanism for seamlessly migrating process state from one node to another, and enable many of the benefits of rolling upgrades by transparently moving managed processes from one node to another, where the latter node is running a different image to the former/original node.
First let's look at the mechanical aspects of this, just to see if it's plausible. Then we'll consider the pros and cons. From an implementation standpoint, I think we can achieve this via several steps:
Let's talk about (1) for now. Firstly, it's important to realise that for (1) and (2) I'm not proposing that the v1 node sends a
Consider, for example, the generic process implementation that we've started working on for -platform. In that module, we use the state monad to track the server state which is
type Process s = ST.StateT s BaseProcess.Process -- [snip for brevity] getState :: Process s s getState = ST.get putState :: s -> Process s () putState = ST.put modifyState :: (s -> s) -> Process s () modifyState = ST.modify -- [snip for brevity] loop :: Behaviour s -> Timeout -> Process s TerminateReason loop s t = do s' <- processReceive (dispatchers s) t nextAction s s' where nextAction :: Behaviour s -> ProcessAction -> Process s TerminateReason nextAction b ProcessContinue = loop b t nextAction b (ProcessTimeout t') = loop b t' nextAction _ (ProcessStop r) = return (TerminateReason r) processReceive :: [Dispatcher s] -> Timeout -> Process s ProcessAction processReceive ds timeout = do s <- getState let ms = map (matchMessage s) ds -- TODO: should we drain the message queue to avoid selective receive here? case timeout of Infinity -> do (s', r) <- ST.lift $ BaseProcess.receiveWait ms putState s' return r Timeout t -> do result <- ST.lift $ BaseProcess.receiveTimeout (intervalToMs t) ms case result of Just (s', r) -> do putState s' return r Nothing -> do return $ ProcessStop "timed out"
Now let us imagine that a new exception type is defined which can be thrown to a process (lightweight thread) - let's call it
-- assuming that we have something like data ForceUpgradeException = ForceUpgradeException !NodeId deriving (Typeable, Show) instance Exception ForceUpgradeException relocate :: (Serializable a) => NodeId -> a relocate nid state = mask $ sendCtrlMsg Nothing $ RelocateSignal nid state
Now looking at
So before we get into the muddy details of (2) I want to clarify that we're saying....
A corollary issue we ought to consider is that of maintaining state invariants whilst a process is transitioning from one state to another. Consider a process that, for example, is in the middle of writing to a file. The implementer may wish to handle normal state changes by reading the contents of the file and writing some changes out before replying (or continuing) but this will break in the face of asynchronous exceptions, such that if the
The obvious solution to this is to call
This use of a pointer is, of course, a very sensible optimisation. We wouldn't want to do away with that in the normal case, nor is it clear to me whether or not there is any other way to handle type identification on remote nodes without user intervention.
My proposal for handling this latter difficulty is very simple, and probably quite unpopular - make it the user's problem, at least to some extent. If the definition hasn't changed then they know this and can use an API call in
If the expected state has changed, then the user needs to deal with this themselves. If the type which the server loop is working with has changed, then they should simply write a transform from
This might sound complicated, but it's only a bit more complicated than what you've got to do in Erlang in practise. Erlang can do magical upgrades because modules can change on the fly, but more often than not record/tuple definitions have changed and the managed process APIs in OTP such as gen_server actually provide an explicit callback for code upgrades, just so that the developer can write these kinds of transformations themselves. Admittedly they've not got to bother with encoding/decoding, but I think that's a reasonable price to pay for a beautiful type system.
Before we can move on to 'addressability' and points (3) and (4), we have another set of issues to handle. When relocating a process, we have mandated that the
This is where it gets a bit more complicated....
All the messages in a process' mailbox are
GHC precomputes the MD5 hash for the TyCon so I'm guessing that we will know if a type is not recognisable when we try to examine its fingerprint. Quite how this works in practise is a little hazy for me - I've not delved into the GHC sources enough to fully understand it - I suspect that the location of the type's pointer in the data section of the image might be used along with the MD5, so we may need a little runtime support in order to identify that? Maybe not though....
Clearly functions like
I'm not 100% clear on how to do this, but it doesn't seem impossible anyway. In fact, one advantage of working via some kind of lookup table would be that we could handle transformations and straight decodes identically. The developer writing the upgrade will need to decide what to do with messages of types that no longer exist, as well as types for which the
Of course for now, I'm leaving all the lovely details of how to handle that to the reader! ;)
There is also the problem of moving the process' typed channels to the new node. We would need to re-establish the
Again, I'm going to leave this as another item for further discussion.
After worrying about how to transfer typed messages from one node to another, when both nodes are running different executables, this ought to be a breeze.
Processes are 'addressable' via their
At a minimum however, a process migration should generate a
data DiedReason = -- | Normal termination DiedNormal | DiedProcessRelocated !ProcessId | DiedNodeRelocated !NodeId | -- snip...
This at least allows remote processes to monitor the fact that the process/node they were interacting with has moved, so that code which holds on to one or more
Talking of monitors, when migrating a process to node-v2, we will need to re-establish all monitors and links, and any node controller that receives the
Dealing with synchronisation
While the -v1 node controller is merrily attempting to migrate a process' state and mailbox to another node, and dealing with incoming and outgoing links and monitors (!), it's highly likely that other nodes are attempting to interact with node-v1. Ignoring connection requests and other infrastructure level calls from
And.... Should we communicate something about this at the
What occurs to me about this, is that we do have to force the user to get involved in the migration. Not only is this normal even for Erlang release upgrades (when data types change for example), it is reasonable IMO and will still be simpler in practice than having to build some complex beast that handles down time. The architectural complexity would be primarily in CH, but the user would have to do some work to make the migration happen.
Another thing that we must consider here is that some kinds of process will not be able to magically migrate their state. A process which has, for example, opened a socket and started listening (or sending) will not be able to do this. The same would be true for a distributed erlang application that was failing-over to another node, though code upgrades would not cause that problem there. In practise, this is also not a huge difficulty, as a load balancer in front of the application isn't a hugely complex thing to have to configure. There is a bit of complexity leakage here, in that a process which is maintaining some kind of session, is going to have to make sure they can re-establish that state on node-v2. Again, this should be do-able because the load balancing mechanism should hide the fail-over from the client and the state migration can simply rely on the
Thoughts/Ideas on a postcard please!!! :)
Some quick thoughts.
First, it's an ambitious project. I rather think there may be other things that need doing first. But perhaps you are motivated by a particular application.
Second, the type of relocate looks very strange. Shouldn't it take the state as input (to be serialised) not return it? And what code should be run at the other end? You need some way to specify which function to call in the new image.
Third, if the old process (OldP) has state of type OldP.T, the new process (NewP) has to parse that state somehow. Imagine that we serialised it as a string with Show. Then OldP's 'show' would generate the string, and NewP's 'read' would parse it. They had better agree!
One possibility is to insist that OldP's state type and the type that NewP parses are identical. (After parsing, NewP might then translate that old state into a new richer state, but that's a separate matter.) What does "identical" mean? One simple possiblity would be "the same all the way down" which is what the MD5 finger print does.
And yet it may be too much to insist that the types are identical. An alternative would be use use a self-describing format, much as happens for web services. I'm out of my depth here, but I'd look at JSON and Protocol Buffers and that kind of stuff. No point in re-inventing the wheel.
HI Simon - thanks for the feedback!
I quite agree. This is very low priority for me, but I thought that others (such as Pankaj) might be interested, and I like to capture the conversation around these ideas on the issue tracker.
That's just me typing too late at night.
Yes indeed. I was thinking there would be a structured API for this, but you're right that this information needs to come from the 'user' as it were. Over email Jeff suggested that the code calling
Indeed this is one of the big challenges. Again over email, Jeff suggested solving this first:
"It might be nice to use a system to ensure that types with different fingerprints can be automatically converted. This is an issue that comes up a lot (also in sending messages between different versions, in the case of partial upgrades) and might be worth solving first." - Jeff Epstein
I might open a separate bug for that at some point, but I want to see if someone bites and takes an interest in owning this first. :)
@basvandijk that's completely awesome - I didn't know it existed. It looks very similar to what we need - if we adapt that approach to work with lazy bytestrings and the one or two other differences - the automatic derivation would need to be done a bit differently and so on - but yes, this looks like an ideal approach. Thanks for pointing it out!