API that expose the list of active Run UIDs to the client

[dmgav]

The API that expose the list of active Run UIDs were implemented. The list of active runs include all the runs that were opened by the currently running plan. The contents of the list may change as the plan is executed. The runs that are closed by the plan are marked as closed and remain in the list. API allow obtain the full list of active runs or subsets of currently open or already closed runs. The list is cleared once the plan execution is completed. The full list of Run UIDs for the completed plans could be found in the plan history.

In case of simple single-run plans, the list of Run UIDs will contain a single item. The client will probably never see this run (or the last run of a multi-run plan) reported as closed, since the list will be cleared once the plan execution is completed.

The items of the list are dictionaries with the following keys: uid - Run UID, is_open - boolean value that indicates if the run is currently open (set to False when the 'stop' document is received), exit_status - run exit status extracted from the 'stop' document.

To keep track of active Run UIDs, Run Engine is subscribed to a custom callback, which extracts UIDs from each emitted 'start' document and creates new items of the list. The callback also captures the emitted 'stop' documents, extracts 'run_start' UID and 'exit_status' values and modifies the respective list entry. Each time the list is modified, it is propagated from RE Worker to RE Manager process and becomes available to client via re_runs ZMQ API. To make tracking of the list more convenient, each state of the list is assigned a uid (same format as the other uids). The uid is passed as run_list_uid field of RE Manager status (status request). A new list uid is generated each time the list is updated. The client application will keep track of the list uid and download the updated list whenever the run_list_uid changes. Downloading the list only when it is changed is expected to minimize the number of requests to the server. Clearing the list at the end of the plan execution is also considered an update and a new value of run_list_uid will be generated.

New features:

• re_state status field ('status' request) - current state of the Run Engine. The state of Run Engine is related, but different from the state of RE Manager. The field may be used for monitoring of Run Engine and detection of transition states (e.g. pausing or resuming) or states that are not handled by RE Manager (such as suspended).

• run_list_uid status field (status request) - UID generated for the current state of the list of active runs. New UID is generated each time the list is changed. Monitoring this field allows the client to know when the list was updated at the server and should be loaded.

• re_runs ZMQ API - load the list of active runs. Optional parameter option may have values active (default), open, closed that allow to choose between the full list, or subsets of open or closed runs.

• /re/runs/active, /re/runs/open, /re/runs/closed HTTP API (GET requests).

• CLI support:

  qserver re runs            # Download the list of active runs (runs generated by the currently running plans)
  qserver re runs active     # Download the list of active runs
  qserver re runs open       # Download the list of open runs (subset of active runs)
  qserver re runs closed     # Download the list of closed runs (subset of active runs)

• permissions_reload ZMQ API - reloads the list of available plans and devices and permssions for user groups from the files on disk. Calling this function makes sense if the files were changed on disk, otherwise reloading will have no effect. The API was necessary to write the unit tests for this PR, but it may be also useful for debugging of profile collection. Support for HTTP and CLI will be implemented as a separate PR.

The PR also includes additional minor tweaks to the code:

• configuration of logging so that it works on OSX;

• PipeJsonRpcReceive class is now removing all messages from the pipe before starting the processing thread. Clearing the pipe from outdated messages makes code more stable. The change was made to get rid of error messages on OSX.

Addresses the issue #111

[gwbischof]

Nice work! I tested this and it is working as expected.

[mrakitin]

I am in the process of reviewing it now, please don't merge - I have a few suggestions.

[dmgav]

I think my next step will be to deal with randomly failing HTTP server tests, because it is getting really annoying.

[dmgav]

BTW, after the modifications I was able to successfully run the unit tests on Mac OS. So in one of the future PRs I will probably add OSX to CI, may be allowing it to fail.

[mrakitin]

I did it! That's a very important and useful functionality, already relied on the webclient side. Here are my suggestions.

[mrakitin]

Thank you for addressing almost all comments/suggestions, @dmgav! I spotted a couple of more places which can reuse the _generate_run_list_uid helper function. For a few of them I made my suggestions, but some were in the code before. Want to address them here too?

$ git grep 'str(u'
bluesky_queueserver/manager/comms.py:        msg["id"] = str(uuid.uuid4())
bluesky_queueserver/manager/manager.py:    return str(uuid.uuid4())
bluesky_queueserver/manager/plan_queue_ops.py:        return str(uuid.uuid4())
bluesky_queueserver/manager/tests/test_plan_monitoring.py:    uids = [str(uuid.uuid4()) for _ in range(3)]
bluesky_queueserver/manager/tests/test_plan_monitoring.py:    uid = str(uuid.uuid4())
bluesky_queueserver/tests/common.py:        msg["id"] = str(uuid.uuid4())

[mrakitin]

Thank you, @dmgav.