Add support for reading plugin cached readings (#213)

docs/api/index.html.md

@@ -656,6 +656,59 @@ These values can be found via the [scan](#scan) command.
 | *{unit}.symbol* | The symbol (or short name) of the unit. *(e.g. "m/s^2")* |
 
 
+## Read Cached
+
+> The response for the `readcached` endpoint is streamed JSON. 
+
+```shell
+curl "http://host:5000/synse/v2/readcached"
+```
+
+```python
+import requests
+import json
+
+response = requests.get('http://host:5000/synse/v2/readcached', stream=True)
+
+# get each line of the streamed response as json
+for chunk in response.iter_lines():
+    data = json.loads(chunk) 
+```
+
+> A single line of the streamed response JSON would be structured as:
+
+```json
+{"location":{"rack":"rack-1","board":"vec","device":"34c226b1afadaae5f172a4e1763fd1a6"},"kind":"humidity","value":31,"timestamp":"2018-10-19T19:13:00.9184028Z","unit":{"symbol":"C","name":"celsius"},"type":"temperature","info":""}
+```
+
+Stream reading data from all configured plugins.
+
+All plugins have the capability of caching their readings locally in order to maintain a higher
+resolution of state beyond the poll frequency which Synse Server may request at. This is particularly
+useful for push-based plugins, where we would lose the pushed reading if it were not cached. 
+
+At the plugin level, caching read data can be enabled, but is disabled by default. Even if disabled,
+this route will still return data for every device that supports reading on each of the configured
+plugins. When read caching is disabled, this will just return a dump of the "current" reading state
+that is maintained by the plugin.
+
+### HTTP Request
+
+`GET http://host:5000/synse/v2/readcached`
+
+### Query Parameters
+
+| Parameter | Description |
+| --------- | ----------- |
+| *start*   | An RFC3339 or RFC3339Nano formatted timestamp which specifies a starting bound on the cache data to return. If no timestamp is specified, there will not be a starting bound. |
+| *end*     | An RFC3339 or RFC3339Nano formatted timestamp which specifies an ending bound on the cache data to return. If no timestamp is specified, there will not be an ending bound. |
+
+### Response Fields
+
+See the responses for [read](#read). The data here is the same, with the addition of a *kind* field, which
+specifies the device kind and *location* object, which provides the routing info for the corresponding device.
+
+
 ## Write
 
 ```shell

docs/index.html

@@ -252,6 +252,9 @@
                   <li>
                     <a href="#read" class="toc-h2 toc-link" data-title="Read">Read</a>
                   </li>
+                  <li>
+                    <a href="#read-cached" class="toc-h2 toc-link" data-title="Read Cached">Read Cached</a>
+                  </li>
                   <li>
                     <a href="#write" class="toc-h2 toc-link" data-title="Write">Write</a>
                   </li>
@@ -1119,6 +1122,56 @@ <h3 id='response-fields-5'>Response Fields</h3>
 <td>The symbol (or short name) of the unit. <em>(e.g. &quot;m/s^2&quot;)</em></td>
 </tr>
 </tbody></table>
+<h2 id='read-cached'>Read Cached</h2>
+<blockquote>
+<p>The response for the <code>readcached</code> endpoint is streamed JSON. </p>
+</blockquote>
+<pre class="highlight shell tab-shell"><code>curl <span class="s2">"http://host:5000/synse/v2/readcached"</span>
+</code></pre><pre class="highlight python tab-python"><code><span class="kn">import</span> <span class="nn">requests</span>
+<span class="kn">import</span> <span class="nn">json</span>
+
+<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'http://host:5000/synse/v2/readcached'</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+
+<span class="c"># get each line of the streamed response as json</span>
+<span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">iter_lines</span><span class="p">():</span>
+    <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span> 
+</code></pre>
+<blockquote>
+<p>A single line of the streamed response JSON would be structured as:</p>
+</blockquote>
+<pre class="highlight json tab-json"><code><span class="p">{</span><span class="s2">"location"</span><span class="p">:{</span><span class="s2">"rack"</span><span class="p">:</span><span class="s2">"rack-1"</span><span class="p">,</span><span class="s2">"board"</span><span class="p">:</span><span class="s2">"vec"</span><span class="p">,</span><span class="s2">"device"</span><span class="p">:</span><span class="s2">"34c226b1afadaae5f172a4e1763fd1a6"</span><span class="p">},</span><span class="s2">"kind"</span><span class="p">:</span><span class="s2">"humidity"</span><span class="p">,</span><span class="s2">"value"</span><span class="p">:</span><span class="mi">31</span><span class="p">,</span><span class="s2">"timestamp"</span><span class="p">:</span><span class="s2">"2018-10-19T19:13:00.9184028Z"</span><span class="p">,</span><span class="s2">"unit"</span><span class="p">:{</span><span class="s2">"symbol"</span><span class="p">:</span><span class="s2">"C"</span><span class="p">,</span><span class="s2">"name"</span><span class="p">:</span><span class="s2">"celsius"</span><span class="p">},</span><span class="s2">"type"</span><span class="p">:</span><span class="s2">"temperature"</span><span class="p">,</span><span class="s2">"info"</span><span class="p">:</span><span class="s2">""</span><span class="p">}</span><span class="w">
+</span></code></pre>
+<p>Stream reading data from all configured plugins.</p>
+
+<p>All plugins have the capability of caching their readings locally in order to maintain a higher
+resolution of state beyond the poll frequency which Synse Server may request at. This is particularly
+useful for push-based plugins, where we would lose the pushed reading if it were not cached. </p>
+
+<p>At the plugin level, caching read data can be enabled, but is disabled by default. Even if disabled,
+this route will still return data for every device that supports reading on each of the configured
+plugins. When read caching is disabled, this will just return a dump of the &quot;current&quot; reading state
+that is maintained by the plugin.</p>
+<h3 id='http-request-8'>HTTP Request</h3>
+<p><code>GET http://host:5000/synse/v2/readcached</code></p>
+<h3 id='query-parameters-2'>Query Parameters</h3>
+<table><thead>
+<tr>
+<th>Parameter</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td><em>start</em></td>
+<td>An RFC3339 or RFC3339Nano formatted timestamp which specifies a starting bound on the cache data to return. If no timestamp is specified, there will not be a starting bound.</td>
+</tr>
+<tr>
+<td><em>end</em></td>
+<td>An RFC3339 or RFC3339Nano formatted timestamp which specifies an ending bound on the cache data to return. If no timestamp is specified, there will not be an ending bound.</td>
+</tr>
+</tbody></table>
+<h3 id='response-fields-6'>Response Fields</h3>
+<p>See the responses for <a href="#read">read</a>. The data here is the same, with the addition of a <em>kind</em> field, which
+specifies the device kind and <em>location</em> object, which provides the routing info for the corresponding device.</p>
 <h2 id='write'>Write</h2><pre class="highlight shell tab-shell"><code>curl <span class="se">\</span>
   -H <span class="s2">"Content-Type: application/json"</span> <span class="se">\</span>
   -X POST <span class="se">\</span>
@@ -1171,7 +1224,7 @@ <h2 id='write'>Write</h2><pre class="highlight shell tab-shell"><code>curl <span
  In Synse Server v2, the <i>data</i> field was called <i>raw</i>. For backwards compatibility,
  <i>raw</i> is still allowed, but will be phased out in the future.
 </aside>
-<h3 id='http-request-8'>HTTP Request</h3>
+<h3 id='http-request-9'>HTTP Request</h3>
 <p><code>POST http://host:5000/synse/v2/write/{rack}/{board}/{device}</code></p>
 <h3 id='uri-parameters-3'>URI Parameters</h3>
 <table><thead>
@@ -1233,7 +1286,7 @@ <h3 id='post-body'>POST Body</h3>
 <p>Some devices may only need an <code>action</code> specified. Some may need both <code>action</code> and <code>data</code> specified.
 While it is up to the underlying plugin to determine what are valid values for a device, generally,
 the <code>action</code> should be the attribute to set and <code>data</code> should be the value to set it to.</p>
-<h3 id='response-fields-6'>Response Fields</h3>
+<h3 id='response-fields-7'>Response Fields</h3>
 <table><thead>
 <tr>
 <th>Field</th>
@@ -1292,7 +1345,7 @@ <h2 id='transaction'>Transaction</h2><pre class="highlight shell tab-shell"><cod
 of time that a transaction is cached for is configurable. See the Synse Server configuration
 <a href="http://synse-server.readthedocs.io/en/latest/user/configuration.html">Configuration Documentation</a>
 for more.</p>
-<h3 id='http-request-9'>HTTP Request</h3>
+<h3 id='http-request-10'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/transaction[/{transaction id}]</code></p>
 <h3 id='uri-parameters-4'>URI Parameters</h3>
 <table><thead>
@@ -1308,7 +1361,7 @@ <h3 id='uri-parameters-4'>URI Parameters</h3>
 <td>The ID of the write transaction to get the status of. This is given by the corresponding <a href="#write">write</a>.</td>
 </tr>
 </tbody></table>
-<h3 id='response-fields-7'>Response Fields</h3>
+<h3 id='response-fields-8'>Response Fields</h3>
 <table><thead>
 <tr>
 <th>Field</th>
@@ -1441,7 +1494,7 @@ <h2 id='info'>Info</h2>
 </span><span class="p">}</span><span class="w">
 </span></code></pre>
 <p>Get the available information for the specified resource.</p>
-<h3 id='http-request-10'>HTTP Request</h3>
+<h3 id='http-request-11'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/info/{rack}[/{board}[/{device}]]</code></p>
 <h3 id='uri-parameters-5'>URI Parameters</h3>
 <table><thead>
@@ -1467,7 +1520,7 @@ <h3 id='uri-parameters-5'>URI Parameters</h3>
 <td>The id of the device to get info for.</td>
 </tr>
 </tbody></table>
-<h3 id='response-fields-8'>Response Fields</h3>
+<h3 id='response-fields-9'>Response Fields</h3>
 <p><strong><em>Rack Level Response</em></strong></p>
 
 <table><thead>
@@ -1620,7 +1673,7 @@ <h2 id='led'>LED</h2>
 of valid query parameters are specified, the endpoint will write to the specified device.</p>
 
 <p>Invalid query parameters will result in a 400 Invalid Arguments error.</p>
-<h3 id='http-request-11'>HTTP Request</h3>
+<h3 id='http-request-12'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/led/{rack}/{board}/{device}</code></p>
 <h3 id='uri-parameters-6'>URI Parameters</h3>
 <table><thead>
@@ -1646,7 +1699,7 @@ <h3 id='uri-parameters-6'>URI Parameters</h3>
 <td>The id of the LED device to read from/write to.</td>
 </tr>
 </tbody></table>
-<h3 id='query-parameters-2'>Query Parameters</h3>
+<h3 id='query-parameters-3'>Query Parameters</h3>
 <table><thead>
 <tr>
 <th>Parameter</th>
@@ -1667,7 +1720,7 @@ <h3 id='query-parameters-2'>Query Parameters</h3>
  While Synse Server supports the listed Query Parameters, not all devices will support the 
  corresponding actions. As a result, writing to some <i>LED</i> instances may result in error.
 </aside>
-<h3 id='response-fields-9'>Response Fields</h3>
+<h3 id='response-fields-10'>Response Fields</h3>
 <p>See the responses for <a href="#read">read</a> and <a href="#write">write</a>.</p>
 <h2 id='fan'>Fan</h2>
 <blockquote>
@@ -1729,7 +1782,7 @@ <h2 id='fan'>Fan</h2>
 of valid query parameters are specified, the endpoint will write to the specified device.</p>
 
 <p>Invalid query parameters will result in a 400 Invalid Arguments error.</p>
-<h3 id='http-request-12'>HTTP Request</h3>
+<h3 id='http-request-13'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/fan/{rack}/{board}/{device}</code></p>
 <h3 id='uri-parameters-7'>URI Parameters</h3>
 <table><thead>
@@ -1755,7 +1808,7 @@ <h3 id='uri-parameters-7'>URI Parameters</h3>
 <td>The id of the fan device to read from/write to.</td>
 </tr>
 </tbody></table>
-<h3 id='query-parameters-3'>Query Parameters</h3>
+<h3 id='query-parameters-4'>Query Parameters</h3>
 <table><thead>
 <tr>
 <th>Parameter</th>
@@ -1776,7 +1829,7 @@ <h3 id='query-parameters-3'>Query Parameters</h3>
  While Synse Server supports the listed Query Parameters, not all devices will support the 
  corresponding actions. As a result, writing to some <i>fan</i> instances may result in error.
 </aside>
-<h3 id='response-fields-10'>Response Fields</h3>
+<h3 id='response-fields-11'>Response Fields</h3>
 <p>See the responses for <a href="#read">read</a> and <a href="#write">write</a>.</p>
 <h2 id='power'>Power</h2>
 <blockquote>
@@ -1835,7 +1888,7 @@ <h2 id='power'>Power</h2>
 of valid query parameters are specified, the endpoint will write to the specified device.</p>
 
 <p>Invalid query parameters will result in a 400 Invalid Arguments error.</p>
-<h3 id='http-request-13'>HTTP Request</h3>
+<h3 id='http-request-14'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/power/{rack}/{board}/{device}</code></p>
 <h3 id='uri-parameters-8'>URI Parameters</h3>
 <table><thead>
@@ -1861,7 +1914,7 @@ <h3 id='uri-parameters-8'>URI Parameters</h3>
 <td>The id of the power device to read from/write to.</td>
 </tr>
 </tbody></table>
-<h3 id='query-parameters-4'>Query Parameters</h3>
+<h3 id='query-parameters-5'>Query Parameters</h3>
 <table><thead>
 <tr>
 <th>Parameter</th>
@@ -1882,7 +1935,7 @@ <h3 id='query-parameters-4'>Query Parameters</h3>
 <p>The power <code>state</code>, commonly <code>on</code>/<code>off</code>, is not bound to those values. It is up to the underlying
 plugin what power actions are available. For example, the IPMI plugin supports <code>on</code>, <code>off</code>, <code>reset</code>,
 and <code>cycle</code>.</p>
-<h3 id='response-fields-11'>Response Fields</h3>
+<h3 id='response-fields-12'>Response Fields</h3>
 <p>See the responses for <a href="#read">read</a> and <a href="#write">write</a>.</p>
 <h2 id='boot-target'>Boot Target</h2>
 <blockquote>
@@ -1941,7 +1994,7 @@ <h2 id='boot-target'>Boot Target</h2>
 of valid query parameters are specified, the endpoint will write to the specified device.</p>
 
 <p>Invalid query parameters will result in a 400 Invalid Arguments error.</p>
-<h3 id='http-request-14'>HTTP Request</h3>
+<h3 id='http-request-15'>HTTP Request</h3>
 <p><code>GET http://host:5000/synse/v2/boot_target/{rack}/{board}/{device}</code></p>
 <h3 id='uri-parameters-9'>URI Parameters</h3>
 <table><thead>
@@ -1967,7 +2020,7 @@ <h3 id='uri-parameters-9'>URI Parameters</h3>
 <td>The id of the boot_target device to read from/write to.</td>
 </tr>
 </tbody></table>
-<h3 id='query-parameters-5'>Query Parameters</h3>
+<h3 id='query-parameters-6'>Query Parameters</h3>
 <table><thead>
 <tr>
 <th>Parameter</th>
@@ -1984,7 +2037,7 @@ <h3 id='query-parameters-5'>Query Parameters</h3>
  While Synse Server supports the listed Query Parameters, not all devices will support the 
  corresponding actions. As a result, writing to some <i>boot_target</i> instances may result in error.
 </aside>
-<h3 id='response-fields-12'>Response Fields</h3>
+<h3 id='response-fields-13'>Response Fields</h3>
 <p>See the responses for <a href="#read">read</a> and <a href="#write">write</a>.</p>
 
       </div>

requirements.txt

@@ -28,7 +28,7 @@ requests==2.19.1          # via kubernetes, requests-oauthlib
 rsa==3.4.2                # via google-auth
 sanic==0.8.3
 six==1.11.0               # via google-auth, grpcio, kubernetes, protobuf, python-dateutil, websocket-client
-synse-grpc==1.0.1
+synse-grpc==1.1.0
 ujson==1.35               # via sanic
 urllib3==1.23             # via kubernetes, requests
 uvloop==0.9.1             # via sanic

setup.py

@@ -42,7 +42,7 @@
         'kubernetes',
         'pyyaml',
         'sanic>=0.8.0',
-        'synse-grpc>=1.0.1',
+        'synse-grpc>=1.1.0',
     ],
     tests_require=[
         'aiohttp',

synse/__init__.py

@@ -3,7 +3,7 @@
 """
 
 __title__ = 'synse'
-__version__ = '2.1.2'
+__version__ = '2.2.0'
 __description__ = 'Synse Server'
 __author__ = 'Vapor IO'
 __author_email__ = 'vapor@vapor.io'

synse/commands/__init__.py

@@ -16,6 +16,7 @@
 from .info import info
 from .plugins import get_plugins
 from .read import read
+from .read_cached import read_cached
 from .scan import scan
 from .test import test
 from .transaction import check_transaction

synse/commands/read_cached.py

@@ -0,0 +1,65 @@
+"""Command handler for the `readcached` route."""
+
+import grpc
+
+from synse import cache, errors, plugin
+from synse.i18n import _
+from synse.log import logger
+from synse.scheme import ReadCachedResponse
+
+
+async def read_cached(start=None, end=None):
+    """The handler for the Synse Server "readcached" API command.
+
+    Args:
+        start (str): An RFC3339 or RFC3339Nano formatted timestamp
+            which defines a starting bound on the cache data to
+            return. If no timestamp is specified, there will not
+            be a starting bound. (default: None)
+        end (str): An RFC3339 or RFC3339Nano formatted timestamp
+            which defines an ending bound on the cache data to
+            return. If no timestamp is specified, there will not
+            be an ending bound. (default: None)
+
+    Yields:
+        ReadCachedResponse: The cached reading from the plugin.
+    """
+    start, end = start or '', end or ''
+    logger.debug(_('Read Cached command (start: {}, end: {})').format(start, end))
+
+    # If the plugins have not yet been registered, register them now.
+    if len(plugin.Plugin.manager.plugins) == 0:
+        logger.debug(_('Re-registering plugins'))
+        plugin.register_plugins()
+
+    # For each plugin, we'll want to request a dump of its readings cache.
+    async for plugin_name, plugin_handler in plugin.get_plugins():  # pylint: disable=not-an-iterable
+        logger.debug(_('Getting readings cache for plugin: {}').format(plugin_name))
+
+        # Get the cached data from the plugin
+        try:
+            for reading in plugin_handler.client.read_cached(start, end):
+                # If there is no reading, we're done iterating
+                if reading is None:
+                    return
+
+                try:
+                    __, device = await cache.get_device_info(  # pylint: disable=unused-variable
+                        reading.rack,
+                        reading.board,
+                        reading.device
+                    )
+                except errors.DeviceNotFoundError:
+                    logger.info(_(
+                        'Did not find device {}-{}-{} locally. Skipping device; '
+                        'server cache may be out of sync.'
+                    ))
+                    continue
+
+                yield ReadCachedResponse(
+                    device=device,
+                    device_reading=reading,
+                )
+
+        except grpc.RpcError as ex:
+            raise errors.FailedReadCachedCommandError(str(ex)) from ex