Struggling with watches

[laboo]

Hi there. I'm having some problems with surfacing watches in CuratorListener.

1. event.getData() always returns null.
2. WatchedEvent.NodeCreated never surfaces
3. Occasionally I get WATCHED type events and event.getPath() returns null

I'll work on creating unit tests that demonstrates this, but would be grateful if you could point me at code that shows me I'm wrong.

Thanks.

[Randgalt]

CuratorListener is a catch-all event queue. event.getPath() is only non-null when you called watched() on one of the builders. i.e.

client.getChildren().watched().forPath(path);

[laboo]

Not quite sure what you're saying there, Jordan. Not sure why my CuratorListener should ever get a CuratorEventType.WATCHED event with a null path. But it's not important, I can ignore it.

My #2 is wrong. If I correctly set a checkExists().watched(), then I do in fact see the WatchedEvent.NodeCreated ZK event. My mistake.

So, my only remaining question is why I never see anything but null from getData(). When a new node is created, or its data is updated, shouldn't getData() be returning the data for that node?

Thanks.

[Randgalt]

Not sure why my CuratorListener should ever get
a CuratorEventType.WATCHED event with a null path

It shouldn't. Maybe there's a bug. What operation were you watching when it happened?

[Randgalt]

why I never see anything but null from getData()

You will only ever get a value getData() if you make a background getData call:

client.getData().inBackground().forPath(path)

[laboo]

Yep, that's what I was missing. Thanks so much, Jordan.

[laboo]

Sorry for the reopen, but event.getStat() always returns null for me, with or without inBackground().

[Randgalt]

I'll try to document CuratorEvent better. I get the idea that you're expecting all the fields to always return a value. That's not how it works. The CuratorEvent is a Composite of all possible events and callbacks that ZooKeeper offers. Only the methods that correspond to a particular callback or event are active. i.e., for a background getData call, CuratorEvent.getData() and CuratorEvent.getPath() will have values. For a watched getChildren call, CuratorEvent.getChildren() will have a value, etc.

You might be better off setting explicit watchers, background handlers instead of using the CuratorListener. If you explain your use-case I can provide more details.

[laboo]

Yes, I'll switch to watcher with callbacks -- which I have from my own implementation with the ZK client. Was hoping I could simplify things a bit with the CuratorListener. Thanks for the help.

[Randgalt]

If you describe what you're trying to do I might be able to suggest something.

[Randgalt]

FYI - if you want an example of how to use CuratorListener, have a look at PathChildrenCache.java

[laboo]

Looking at PathChildrenCache.java, I think you're right, that's not what I want. So let me describe the simplest use case.

I start with this: client.getData().watched().inBackground().forPath("/abc");

znode "/abc" exists. I go into the ZK client (zkCli.sh) and set the data for "/abc" to some new value.

When the watch triggers in my curator client I want to see (1) the new data and (2) the Stat object for "/abc".

How do I do that without making a subsequent get call on "/abc"?

Using the CuratorListener, all I seem to get in the CuratorEvent object is the data, not the Stat object (which is always null).

Thanks again.

[Randgalt]

Unfortunately, ZooKeeper doesn't support that behavior. When the watch gets triggered, you only know that data has changed but not what the data is. ZooKeeper requires an additional call to get the data.

[laboo]

Right. I was hoping/dreaming that Curator was in some way abstracting that away for me.

So, I've got a znode I want to continuously watch, resetting the watch every time it triggers. With the ZK native client, I do a get/watch, receive the data/Stat object in an async callback (processResult()) and get notified of the triggered watch in a Watcher's process() callback, where I do another get/watch...and it all repeats.

In curator then, I'd reproduce this with a BackgroundCallback (processResult) and a Watcher that I register with the client?

[Randgalt]

You should use PathChildrenCache. It was written to solve the exact scenario you describe.

[Randgalt]

Right. I was hoping/dreaming that Curator was in some way abstracting that away for me.

It does :) Curator is focused on providing recipes. PathChildrenCache does exactly what you want.

[laboo]

D'oh. Don't I feel stupid circling around again to PathChildrenCache. I obviously didn't look at it very deeply previously. Sorry about that.

Can you explain a little about the "false positives" and "false negatives" comments on that class?

[Randgalt]

Can you explain a little about the "false positives" and "false negatives" comments on that class?

It's a warning that the PathChildrenCache is never transactionally consistent - only because ZooKeeper doesn't provide this guarantee. It's only eventually consistent.

[Randgalt]

P.S. My goal is that most users never have to look beyond the recipes.

[laboo]

I've spent a lot of time on the wiki and somehow never clicked on the Utils link on the recipes page. All this time we've been discussing PathChildrenCache, I was in the source. I'll learn, eventually.

This probably goes without saying, but Curator and its recipes are a distributed developer's dream come true. I'm grateful for the work you've put in on it, and for Netflix's making it Apache licensed. It's a thing of beauty.

[Randgalt]

I've spent a lot of time on the wiki and somehow never clicked on the Utils link on the recipes page

Hmm - I'll add a link to it on the recipes pages.

[Randgalt]

This probably goes without saying, but Curator and its recipes are a distributed developer's dream come true.

blush thanks

[laboo]

Using a PathChilidrenCache, CACHE_DATA mode...

My ZK tree looks like this: /base/a, base/b, base/c

When I delete /base/a I get a CHILD_UPDATED for /base/b and base/c (in addition to the CHILD_REMOVED for /base/a).

Same with create. When I recreate /base/a, I get a CHILD_UPDATED on all the other children.

Intentional?

[Randgalt]

Probably not - if you don't mind, please open another issue with this.

[itissid]

So this might be a noob question. But I wrote a small test here: https://gist.github.com/1886760. The thing I realized that the Assert statements always fails is because the events are only ordered, but there delivery might not always be. Which leads me to believe we should progarm around the idea of eventual consistency. Because I always will receive the last event.
This was loosely related to what this issue was which was how to program to the zookeeper paradigm

[Randgalt]

ZooKeeper watchers are single fire. (see http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#sc_WatchGuarantees). So, I assume that your actual count is 1. Once a watcher is called, you must re-set the watch to have it called again. The only way to re-set a watch is to make another ZooKeeper call (checkExists(), etc.).

[umabisht]

Hi,
Inspite of using a background call i.e. CuratorFramework.getData().inBackground().forPath("/abc"), i am getting a null.
Can you please help?

[Randgalt]

umabisht - can you give more context please? If the node doesn't exist you get null. You will also get an error code in your callback.

[umabisht]

Ok, Actually i have a structure built at Zookeeper i.e /SecurityTest_NA/...../last_node(data=2)

Now inmy code, where i have initialized framework like this.

CuratorFramework framework =CuratorFrameworkFactory.newClient("localhost:2181",30000,10000,new RetryOneTime(10));

1- framework.getData().inBackground().forPath(pathtilllastnode).toString(); gives null
2- framework.getData().watched().forPath(pathtilllastnode).toString(); gives some value but not 2

[umabisht]

just to add to it , i am using Curator version 1.0.9 with ZooKeeper 3.3.1.

[Randgalt]

ZooKeeper APIs have two modes, foreground/synchronous and background/asynchronous. In your example #1 you're using the background mode so the forPath() method returns null. The result comes through the default listener. You can also, optionally, pass a callback to receive the result.

[Randgalt]

just to add to it , i am using Curator version 1.0.9 with ZooKeeper 3.3.1.

Why?

[itissid]

Yeah use the latest versions, atleast use 3.3.4 which is compatible. It had lot of important bug fixes.

[umabisht]

Migrating our existing system from 3.3.1 to higher versions is in planning phase and hopefully will be achieved soon.Meanwhile we chose to work with curator api and had to select version 1.0.9 to overcome compatibility issues.
I am sure that curator latest verrsion must be better and enhanced in every aspect but assume that it wont differ much from 1.0.9 in terms of implementation, so that we can anytime upgrade it with not much pains.. ??

[itissid]

I dont understand. "Curator 1.0.x will stay compatible with ZooKeeper 3.3.x - NOTE: Now end of life and no longer maintained"
Thats on the curator framework front page. You can easily go to 3.3.4 min. There are multiple nasty bugs like ephemeral nodes not getting deleted and race conditions which are pretty serious and have come to bite us. I am not saying 3.3.1 is broken beyond repair. But the rule is to be upto date as much as possible.

[Randgalt]

Curator 1.0.x is at 3.3.5 now. I added the note because I'd like folks to move to the main branch. But, I'll be releasing 1.0.x versions for another month or so.