cells: Suppress RejectedExecutionException on shutdown

Motivation:

The following was observed during shutdown:

30 May 2016 09:04:35 (System) [] Background exception was not retry-able or retry gave up
java.util.concurrent.RejectedExecutionException: Executor has been shut down.
        at org.dcache.util.BoundedExecutor.execute(BoundedExecutor.java:140) ~[dcache-common-2.16.0-SNAPSHOT.jar:2.16.0-SNAPSHOT]
        at org.dcache.util.BoundedExecutor.execute(BoundedExecutor.java:147) ~[dcache-common-2.16.0-SNAPSHOT.jar:2.16.0-SNAPSHOT]
        at org.apache.curator.framework.imps.Backgrounding$1.processResult(Backgrounding.java:102) ~[curator-framework-2.10.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:749) [curator-framework-2.10.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:522) [curator-framework-2.10.0.jar:na]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166) [curator-framework-2.10.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) [zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [zookeeper-3.4.6.jar:3.4.6-1569965]

The problem is that we cannot unregister watchers from ZooKeeper and thus
ZooKeeper may try to call a watcher or callback after a cell has shut down.

Modification:

Catch the exception and discard it if the executor was shut down. The executor
involved is only used for curator watchers and callbacks and thus tasks are
safe to discard when the receiver has shut down.

Result:

No exception.

Target: trunk
Request: 2.16
Require-notes: no
Require-book: no
Fixes: #2475
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>

Reviewed at https://rb.dcache.org/r/9353/

(cherry picked from commit 2072541)

modules/cells/src/main/java/dmg/cells/zookeeper/CellCuratorFramework.java

@@ -70,6 +70,7 @@
 import java.util.Collection;
 import java.util.List;
 import java.util.concurrent.Executor;
+import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.TimeUnit;
 
 import dmg.cells.nucleus.CDC;
@@ -88,7 +89,23 @@ public class CellCuratorFramework implements CuratorFramework
     public CellCuratorFramework(CuratorFramework inner, Executor executor)
     {
         this.inner = inner;
-        this.executor = new SequentialExecutor(executor);
+        this.executor = new SequentialExecutor(executor) {
+            @Override
+            public void execute(Runnable task)
+            {
+                try {
+                    super.execute(task);
+                } catch (RejectedExecutionException e) {
+                    /* There is no way to unregister watchers from ZooKeeper. Thus
+                     * it is possible for ZooKeeper to try to call a watcher after a
+                     * cell shut down, resulting in a RejectedExecutionException.
+                     */
+                    if (!isShutdown()) {
+                        throw e;
+                    }
+                }
+            }
+        };
     }
 
     protected static BackgroundCallback wrap(BackgroundCallback callback)