Adaptive does not receive periodic callbacks in distributed 1.21.x, tornado 4.x

[azjps]

In #1172, the syntax for PeriodicCallback in Adaptive (and so forth) were updated for tornado 5.x to remove the ioloop argument. Unfortunately, this has been causing Adaptive to stop functioning for me with tornado 4.x. Here's a MCVE:

In [1]: from distributed.deploy import Adaptive

In [2]: from distributed import LocalCluster, Client, Scheduler

In [3]: cluster = LocalCluster(n_workers=0)

In [4]: adaptive = Adaptive(cluster.scheduler, cluster)

In [5]: client = Client(cluster)

In [6]: fut = client.submit(lambda x: x, 1)

In [7]: fut
# With following versions installed, future completes pretty much immediately as expected:
# python 2.7.6
# dask==0.17.1
# dask-drmaa==0.1.0
# distributed==1.20.2
# tornado==4.5.3
Out[7]: <Future: status: finished, type: int, key: <lambda>-b3d74994205badad27bd7f7ba7512ee1>

----------

In [7]: fut
# With following versions installed, future *never* completes:
# python 2.7.6
# dask==0.17.1
# dask-drmaa==0.1.0
# distributed==1.21.3
# tornado==4.5.3
Out[7]: <Future: status: pending, key: <lambda>-399a830e5fd9ea6b19d00bcf0bfefc24>

In [8]:                                                                                             
Do you really want to exit ([y]/n)? y  # backtrace truncated
distributed.scheduler - ERROR - <Task 'f-32394be4ee42451cc0cb97b2cdaf9c2f' no-worker>
Traceback (most recent call last):
  File "/apps/stonefs1/stonepy/venv/20180222.el6-stonelib/lib/python2.7/site-packages/distributed/scheduler.py", line 3615, in transition_no_worker_released
    self.unrunnable.remove(ts)
KeyError: <Task 'f-32394be4ee42451cc0cb97b2cdaf9c2f' no-worker>
distributed.scheduler - ERROR - Error transitioning 'f-32394be4ee42451cc0cb97b2cdaf9c2f' from 'no-worker' to 'released'

I don't know enough about tornado to suggest a fix (or know if I am doing something wrong), but in my own derived class from Adaptive, I've restored the old PeriodicCallback syntax in its constructor and everything works as expected. (cc @pitrou)

[mrocklin]

Thank you @azjps for the excellent reproducible example. I was able to easily replicate this error and have resolved it in #1841 .

Also I apologize for the delay in response.