DefaultPromise may notify Listeners in wrong order

[normanmaurer]

This pull-req is not complete yet but show a race in DefaultPromise which can lead to have FutureListener notified in a wrong order.

One way to fix this is to enlarge the scope of the synchronization but I would like to think about some better way to fix it.

This can happen if addListener is called from a different thread then the EventExecutor that is used by the DefaultPromise itself and the DefaultPromise is notified while adding these listeners.

[normanmaurer]

@trustin any good idea ?

[ghost]

Build result for #2157 at f60eeac: Success

[ghost]

Build result for #2157 at 6ca8e87b461260f37f4ada9a4b48aac5d398c6ea: Failure

[normanmaurer]

@trustin please review... we could also make use of Unsafe (if present) to replace Atomic_FieldUpdater as Unsafe is faster because it does not do any validation (which Atomic_FieldUpdater) does.

[normanmaurer]

Here they also tell about the performance difference:
https://groups.google.com/forum/#!msg/mechanical-sympathy/X-GtLuG0ETo/_KcpM7T8ZX4J

[ghost]

Build result for #2157 at 60489c1940fe3ff2e308bda4412c97efb51d993d: Success

[normanmaurer]

@daschl please also review

[ghost]

Build result for #2157 at 6ccad82e3d8de2a3b0ae49bfa3706b73a074707c: Success

[ghost]

Build result for #2157 at 72482f0858673a4432a28e8bee8552bf4b3e105a: Success

[normanmaurer]

Also about performance. With this branch I get slightly better performance (able to reproduce every time) , most likely because of the Atomic*FieldUpdater changes:

This branch:

[nmaurer@ear]~/wrk% ./wrk  -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 8 --pipeline 16 http://localhost:8080/plaintext
Running 2m test @ http://localhost:8080/plaintext
  8 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.09ms    2.96ms  55.53ms   89.68%
    Req/Sec   223.13k    77.56k  403.55k    64.58%
  199582048 requests in 2.00m, 26.95GB read
Requests/sec: 1663182.82
Transfer/sec:    229.99MB

4.0 branch:

[nmaurer@ear]~/wrk% ./wrk  -H 'Host: localhost' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -d 120 -c 256 -t 8 --pipeline 16 http://localhost:8080/plaintext
Running 2m test @ http://localhost:8080/plaintext
  8 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.08ms    2.86ms  52.51ms   89.43%
    Req/Sec   222.70k    77.23k  410.67k    64.37%
  199282272 requests in 2.00m, 26.91GB read
Requests/sec: 1660680.58
Transfer/sec:    229.64MB

[trustin]

UnsafeAtomic*FieldUpdater could be extracted out to separate .java files?

[trustin]

You could use GlobalEventExecutor instead.

[trustin]

I wish it does not create a new entry when there's only one listener.

[normanmaurer]

Will see what I can do... Rest looks good beside your comments and ready to pull in after addressing them?

Am 05.02.2014 um 00:36 schrieb Trustin Lee notifications@github.com:

I wish it does not create a new entry when there's only one listener.

—
Reply to this email directly or view it on GitHub.

[ghost]

Build result for #2157 at 91d413322f407e9c49556289ce2a2d9d4f2f7c62: Success

[ghost]

Build result for #2157 at 85a6b905f3589ac334257929e1b779e4f8ca57c7: Success

[normanmaurer]

@trustin addressed all your comments. Please review again and let me know what you think now ;)

[ghost]

Build result for #2157 at 7de37ca4c29d4ed8bdf59d2d7cc9403167bb3abb: Success

[normanmaurer]

@trustin ok to merge?

[ghost]

Build result for #2157 at a8e84db1eabcb2e6559a23c4f655ce411f163afc: Success

[ghost]

Build result for #2157 at 2fbc42a: Success

[normanmaurer]

Squashed into 2 commits

[trustin]

No need to wrap with if (entry == null) { } because compareAndSet() already does the same check?

[trustin]

This loop looks weird to me. The first if (entry == null || ...) block and the second if (...) { if (...) { block are pretty much same except that the second one decreases the counter, which means the second block will never be evaluated.

[trustin]

• This loop is vulnerable to the race where more than on thread attempt to append a listener. When one thread succeeds to append a listener, the other thread can dereference it and thus we lose it.
• Also, you are traversing the linked list sequentially, which is potentially slow. Perhaps we could maintain the reference to the last listener?

[normanmaurer]

Yes... but this will make it even more hard to not use synchronized and I though it should not be a big problem. Maybe I could also just use synchronized and be happy ;)

[trustin]

How about this:

1. Add a volatile field that tells if the primary notification is actually made. It will be set from the executor thread when all listeners added so far were notified (see notifyListeners()).
2. In addListener() we check if the volatile field is set. If set, it is safe to use the current execution path. If not set, always use executor.execute() for notification.
3. In the notification task we created in addListener(), we always check if the primary notification is made by checking the volatile field, and then re-schedule the task instead of making a notification. If the volatile field is set, we are safe to notify immediately here.

This basically uses executor's task queue for ordering.

[ghost]

Build result for #2157 at d3a1274: Success

[normanmaurer]

@trustin tried this and did not work out so far. Not sure if we are just trying to be too smart..

[normanmaurer]

I will check again this evening about this issue..

[trustin]

#2186 shows an alternative fix

[normanmaurer]

Was fixed by 309ee68