callbacks are not retained after a zookeper restart

[mindreframer]

Setup:

• 3 zookeeper servers are started
• 2 ruby clients (from terminals)
• zookeeper ruby client (master branch, commit https://github.com/twitter/zookeeper/commit/3d5fdbd2f8ff4d37ae29f85c49a5bcd781468d5f)

client code:

 require 'zookeeper'

ZK = Zookeeper.new('192.168.2.93:2181, 192.168.2.94:2181, 192.168.2.95:2181')

def set_callback(path)
  wcb = Zookeeper::WatcherCallback.new do
    puts "callback fired for #{path}"
    new_value = ZK.get(:path => path)[:data]
    puts "value changed to #{new_value}"
    set_callback(path)
  end
  ZK.stat(:path => path, :watcher => wcb)
end

#ZK.set_debug_level(Zookeeper::ZOO_LOG_LEVEL_DEBUG)
set_callback('/test')
ZK.set(:path => '/test', :data => '60')

changing node '/test' in one irb console fires the callback for both clients

restarting a zookeeper server:

• all callbacks are fired (?> callback fired for /test)
• the node has no value at this point (no output like "value changed to ..." )

after a restart of one zookeeper server the cease to fire...

the gist with irb output with debugging is here:
https://gist.github.com/807539

[wickman]

if a zookeeper client loses its session, then all watches expire and the user of the client is responsible for re-registering them. same as the C client semantics.

[mindreframer]

that's strange...
http://kisalay.com/2010/12/27/zookeeper-overview-and-data-model/

Sessions enable a client to move transparently from one server to another within a ZooKeeper ensemble, and hence persist across ZooKeeper servers."
--> so, if one zookeeper server goes down, the session should still valid, right? Or am I mixing some things up?

It just doesn't make sense to set all the watches again, actually I would expect transparent handling as long as the zookeeper ensemble is partially up.
So, basically the workaround for now would be setting all the needed watches again, is this the suggested way?

Found something here:
http://www.jarvana.com/jarvana/view/org/apache/zookeeper/3.3.1/zookeeper-3.3.1-javadoc.jar!/org/apache/zookeeper/ZooKeeper.html

If the ZooKeeper server the client currently connects to fails or otherwise does not respond, the client will automatically try to connect to another server before its session ID expires. If successful, the application can continue to use the client.

A client needs an object of a class implementing Watcher interface for processing the events delivered to the client. When a client drops current connection and re-connects to a server, all the existing watches are considered as being triggered but the undelivered events are lost. To emulate this, the client will generate a special event to tell the event handler a connection has been dropped. This special event has type EventNone and state sKeeperStateDisconnected.

Seems, like you're right :) Thx for speedy feedback!