development docs for tests

[dazza-codes]

Remaining issue after #3398

While using the example unit tests from the dev-doc, ran into some problems that might find a way to get updated in another PR. The following was not addressed in #3398 because it's not entirely clear how it should work, i.e. the working code with comments on failures is like:

# import all the dask distributed pytest fixtures
from distributed import Future
from distributed.utils_test import *  # pylint: disable=wildcard-import


def test_sync_submit(client):
    future = client.submit(inc, 10)
    assert isinstance(future, Future)
    assert future.result() == 11  # use the synchronous/blocking API here


def test_sync_submit_full(client, s, a, b):
    """
    In this style of test you do not have access to the scheduler or workers.
    The variables s, a, b are now dictionaries holding a multiprocessing.Process
    object and a port integer. However, you can now use the normal synchronous
    API (never use yield in this style of test) and you can close processes
    easily by terminating them.
    """
    assert isinstance(client, Client)
    assert isinstance(s, dict)
    assert client.scheduler.address == s['address']
    assert isinstance(a, dict)  # worker-a
    assert isinstance(b, dict)  # worker-b

    future = client.submit(inc, 10)
    assert isinstance(future, Future)
    assert future.result() == 11  # use the synchronous/blocking API here

    fixed_test = False
    if fixed_test:
        # killing a worker fails the test cleanup checks:
        # Failed: some RPCs left active by test

        worker_process = a['proc']()  # call weakref to get worker process
        worker_process.terminate()  # kill one of the workers
        # client.retire_workers()  # also not a clean test teardown

        result = future.result()  # test that future remains valid
        assert isinstance(future, Future)
        assert result == 11