Improve sandbox environment setup and documentation

CHANGES.rst

@@ -8,6 +8,7 @@ development
 - Add example for annotating phenology data within Grafana
 - Change license to AGPL-3.0
 - Modernize dependency versions. Drop support for Python 3.6.
+- Improve sandbox environment setup and documentation
 
 
 2020-12-27 0.1.2

README.rst

@@ -1,37 +1,124 @@
+.. image:: https://github.com/panodata/grafana-pandas-datasource/workflows/Tests/badge.svg
+    :target: https://github.com/panodata/grafana-pandas-datasource/actions?workflow=Tests
+
+.. image:: https://img.shields.io/pypi/pyversions/grafana-pandas-datasource.svg
+    :target: https://pypi.org/project/grafana-pandas-datasource/
+
+.. image:: https://img.shields.io/pypi/status/grafana-pandas-datasource.svg
+    :target: https://pypi.org/project/grafana-pandas-datasource/
+
+.. image:: https://img.shields.io/pypi/v/grafana-pandas-datasource.svg
+    :target: https://pypi.org/project/grafana-pandas-datasource/
+
+.. image:: https://img.shields.io/pypi/dm/grafana-pandas-datasource.svg
+    :target: https://pypi.org/project/grafana-pandas-datasource/
+
+.. image:: https://img.shields.io/pypi/l/grafana-pandas-datasource.svg
+    :target: https://github.com/panodata/grafana-pandas-datasource/blob/main/LICENSE
+
+.. image:: https://img.shields.io/badge/Grafana-6.x%20--%208.x-blue.svg
+    :target: https://github.com/grafana/grafana
+    :alt: Supported Grafana versions
+
+|
+
 #########################
-Grafana Pandas Datasource
+Grafana pandas datasource
 #########################
 
 
 *****
 About
 *****
-A REST API based on Flask for serving Pandas Dataframes to Grafana.
 
-This way, a native Python application can be used to directly supply
-data to Grafana both easily and powerfully.
+A HTTP API based on Flask_ for serving pandas_ data frames to Grafana_,
+generated by NumPy_. The `Grafana Simple JSON Datasource`_ is used to interface
+Grafana with the HTTP API.
 
-It was inspired by and is compatible with the simple json datasource.
+This way, a native Python application can be used to directly supply data to
+Grafana easily and powerfully.
 
-https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850
+The framework supports feeding both timeseries data as well as annotations
+through corresponding ``/query`` and ``/annotations`` endpoints and also
+provides ``/search`` and ``/panels`` endpoints.
 
-Setup
-=====
-::
 
-    pip install grafana-pandas-datasource
+*******************
+Sandbox environment
+*******************
+
+In order to work efficiently with the resources provided by this repository, we
+recommend to install some programs upfront. This will optimally work on Linux
+and macOS. Windows users might use the WSL subsystem.
+
+Install prerequisites::
+
+    # Debian Linux
+    apt install git python3 python3-pip httpie docker.io
+    pip install poetry
+
+    # macOS / Homebrew
+    brew install git python3 poetry httpie docker
+
+Acquire sources and bootstrap sandbox environment::
+
+    git clone https://github.com/panodata/grafana-pandas-datasource
+    cd grafana-pandas-datasource
+    poetry install
+    poetry shell
+
+Test drive::
 
+    # Run Grafana pandas datasource demo.
+    python examples/sinewave-midnights/demo.py
 
-Resources
-=========
-- https://github.com/grafana/grafana
-- https://grafana.com/grafana/plugins/grafana-simple-json-datasource
+    # Submit a timeseries data request.
+    echo '{"targets": [{"target": "sine_wave:24"}], "range": {"from": "2022-02-22T15", "to": "2022-02-22T20"}}' | http http://127.0.0.1:3003/query
+
+    # Submit an annotation data request.
+    echo '{"annotation": {"query": "midnights:xx"}, "range": {"from": "2022-02-20", "to": "2022-02-22"}}' | http http://127.0.0.1:3003/annotations
+
+When the environment has been properly configured, both requests above will
+yield appropriate responses.
 
 
 ********
 Examples
 ********
-There are different demo programs accompanied with Grafana datasource
-and dashboard definition files.
 
-https://github.com/panodata/grafana-pandas-datasource/tree/main/examples
+There are `different demo programs`_ accompanied with Grafana datasource and
+dashboard definition files. 
+
+After confirming the sandbox environment has been installed successfully,
+please head over to the `Sinewave/Midnights example`_ page in order to learn
+how to provision Grafana with corresponding resources.
+
+
+*****
+Setup
+*****
+
+When aiming to run a dedicated service, without needing to invoke the examples,
+you can add the package ``grafana-pandas-datasource`` to the list of your
+project requirements.
+
+::
+
+    pip install grafana-pandas-datasource
+
+
+*******
+Credits
+*******
+
+Kudos to Linar, who conceived the initial version of this software the other
+day at https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850.
+
+
+.. _different demo programs: https://github.com/panodata/grafana-pandas-datasource/tree/main/examples
+.. _Flask: https://github.com/pallets/flask
+.. _Grafana: https://github.com/grafana/grafana
+.. _Grafana Simple JSON Datasource: https://grafana.com/grafana/plugins/grafana-simple-json-datasource/
+.. _NumPy: https://numpy.org/
+.. _pandas: https://github.com/pandas-dev/pandas
+.. _Sinewave/Midnights example: https://github.com/panodata/grafana-pandas-datasource/tree/main/examples/sinewave-midnights

docs/backlog.rst

@@ -0,0 +1,7 @@
+#################################
+Grafana pandas Datasource backlog
+#################################
+
+- [x] Adjust HTML index page
+- [o] Blackify
+- [o] Upgrade to Flask 2

examples/phenodata-mellifera/demo.py

@@ -82,7 +82,7 @@ def phenodata_mellifera(dataset: str, years: tuple[int], phases: tuple[str], opt
 
     data = pd.concat(data_total)
 
-    # Create Pandas Series from Dataframe.
+    # Create pandas Series from DataFrame.
     index = data.Datum.astype('datetime64')
     values = data.Spezies.str.cat(data.Phase, sep=" - ").str.cat(data.Station, sep=" - ")
     series = pd.Series(data=values.tolist(), index=index)
@@ -99,7 +99,7 @@ def main():
     # Create Flask application.
     app = create_app()
 
-    # Register Pandas component.
+    # Register pandas component.
     app.register_blueprint(pandas_component, url_prefix="/")
 
     # Invoke Flask application.

examples/phenodata-mellifera/readme.rst

@@ -43,7 +43,7 @@ Invoke
         --env='GF_SECURITY_ADMIN_PASSWORD=admin' --env='GF_INSTALL_PLUGINS=grafana-simple-json-datasource' \
         grafana/grafana:7.3.6
 
-    # Run Grafana Pandas Datasource demo.
+    # Run Grafana pandas Datasource demo.
     python phenodata-mellifera-demo.py
 
 

examples/sinewave-midnights/demo.py

@@ -13,7 +13,7 @@
 """
 Demo for grafana-pandas-datasource.
 
-This is a demo program which generates data using NumPy and Pandas.
+This is a demo program which generates data using NumPy and pandas.
 It creates
 - a sine wave for data and
 - midnight times for annotations
@@ -25,6 +25,13 @@
 To query the reader, use ``<reader_name>:<query_string>``, e.g.
 - ``sine_wave:24``
 - ``midnights:xx``
+
+Test drive::
+
+    python examples/sinewave-midnights/demo.py
+    echo '{"targets": [{"target": "sine_wave:24"}], "range": {"from": "2022-02-22T15", "to": "2022-02-22T20"}}' | http http://127.0.0.1:3003/query
+    echo '{"annotation": {"query": "midnights:xx"}, "range": {"from": "2022-02-20", "to": "2022-02-22"}}' | http http://127.0.0.1:3003/annotations
+
 """
 
 
@@ -53,7 +60,7 @@ def main():
     # Create Flask application.
     app = create_app()
 
-    # Register Pandas component.
+    # Register pandas component.
     app.register_blueprint(pandas_component, url_prefix="/")
 
     # Invoke Flask application.

examples/sinewave-midnights/readme.rst

@@ -1,71 +1,99 @@
 ##########################
 Sinewave/Midnights example
 ##########################
-This is a demo program which generates a sine wave for data and
-annotations for designating midnight times. For both, we are using NumPy.
-
-.. figure:: https://user-images.githubusercontent.com/453543/103137119-78dab480-46c6-11eb-829f-6aa957239804.png
-
-    Image: Sinewave data and midnights annotations, both generated using NumPy.
 
 
-Setup
-=====
-::
-
-    pip install grafana-pandas-datasource
+*****
+About
+*****
 
+This is a demo program which generates a sine wave for data and
+annotations for designating midnight times.
 
-Acquire example files
-=====================
-::
+.. figure:: https://user-images.githubusercontent.com/453543/103137119-78dab480-46c6-11eb-829f-6aa957239804.png
 
-    export EXAMPLES_BASEURL=https://raw.githubusercontent.com/panodata/grafana-pandas-datasource/0.1.0/examples
+    Image: Sinewave data and midnights annotations, both generated using NumPy_.
 
-    wget ${EXAMPLES_BASEURL}/sinewave-midnights/demo.py \
-        --output-document=sinewave-midnights-demo.py
 
-    wget ${EXAMPLES_BASEURL}/sinewave-midnights/datasource.json \
-        --output-document=sinewave-midnights-datasource.json
+*****
+Setup
+*****
 
-    wget ${EXAMPLES_BASEURL}/sinewave-midnights/dashboard.json \
-        --output-document=sinewave-midnights-dashboard.json
+For general installation instructions, see `setup sandbox environment`_.
 
 
-Invoke
-======
+**************
+Start services
+**************
 ::
 
     # Run Grafana.
     docker run --rm -it \
         --publish=3000:3000 --volume="$(pwd)/var/lib/grafana":/var/lib/grafana \
         --env='GF_SECURITY_ADMIN_PASSWORD=admin' --env='GF_INSTALL_PLUGINS=grafana-simple-json-datasource' \
-        grafana/grafana:7.3.6
+        grafana/grafana:8.3.4
 
-    # Run Grafana Pandas Datasource demo.
-    python sinewave-midnights-demo.py
+    # Run Grafana pandas datasource demo.
+    python examples/sinewave-midnights/demo.py
 
 
-Configure
-=========
-.. note::
+*****************
+Configure Grafana
+*****************
 
-    The host where the datasource service is running can be accessed from the
-    Grafana Docker container using the hostname ``host.docker.internal``.
+Command line
+============
 
-You can have a quickstart by putting ``examples/sinewave-midnights/datasource.json``
-and ``examples/sinewave-midnights/dashboard.json`` into Grafana::
+You can have a quickstart by putting those two JSON definition files into
+Grafana::
 
     # Login to Grafana.
     export GRAFANA_URL=http://localhost:3000
     http --session=grafana ${GRAFANA_URL} --auth=admin:admin
 
     # Create datasource.
-    cat sinewave-midnights-datasource.json | \
+    cat examples/sinewave-midnights/datasource.json | \
         http --session=grafana POST ${GRAFANA_URL}/api/datasources
 
     # Create dashboard.
-    cat sinewave-midnights-dashboard.json | \
+    cat examples/sinewave-midnights/dashboard.json | \
         http --session=grafana POST ${GRAFANA_URL}/api/dashboards/db
 
-    open ${GRAFANA_URL}
+Then, visit the dashboard at::
+
+    open http://localhost:3000/d/xNbUrobGz/sine-24
+
+GUI
+===
+
+This section walks you through setting up a data source and dashboard in
+Grafana manually, step by step with screenshots. Please follow the guidelines
+carefully.
+
+.. figure:: https://user-images.githubusercontent.com/453543/150621604-f9b4664c-493a-4a9d-bd46-cf59da175438.png
+
+    Install "SimpleJson" plugin at http://localhost:3000/plugins/grafana-simple-json-datasource.
+
+.. figure:: https://user-images.githubusercontent.com/453543/150621516-cb8b24fa-46ee-4515-b66e-81f79a046912.png
+
+    Add new data source of "SimpleJson" type at http://localhost:3000/datasources/new.
+    Configure the URL to the Flask service serving pandas data frames.
+    When running Grafana in Docker, use ``host.docker.internal`` to address the
+    Docker host.
+
+.. figure:: https://user-images.githubusercontent.com/453543/150621520-f0eeb740-2c12-4a8b-908c-50893a8bd583.png
+
+    Create dashboard with Timeseries or Graph panel at http://localhost:3000/dashboard/new,
+    adjust "Data source" and "metric" values.
+
+.. figure:: https://user-images.githubusercontent.com/453543/150621869-5d226582-886c-41c4-a446-d8d75685f9d2.png
+
+    At the time picker, choose an interval of "Last 2 days".
+
+.. figure:: https://user-images.githubusercontent.com/453543/150621970-3d20f11c-007a-4e6e-ad8f-abf1f3e02ed0.png
+
+    Save your dashboard.
+
+
+.. _NumPy: https://numpy.org/
+.. _setup sandbox environment: https://github.com/panodata/grafana-pandas-datasource/blob/main/README.rst#sandbox-environment

grafana_pandas_datasource/registry.py

@@ -12,7 +12,7 @@
 class DataGenerators:
     """
     Store references to data generator functions
-    yielding Pandas data frames.
+    yielding pandas data frames.
     """
 
     metric_readers: Dict[str, Callable] = field(default_factory=dict)

grafana_pandas_datasource/service.py

@@ -19,7 +19,8 @@
 @cross_origin()
 def hello_world():
     print(request.headers, request.get_json())
-    return 'Jether\'s Grafana Pandas Datasource, used for rendering HTML panels and timeseries data.'
+    return 'Grafana pandas datasource: Serve NumPy data via pandas data frames to Grafana. ' \
+           'For documentation, see <a href="https://github.com/panodata/grafana-pandas-datasource">https://github.com/panodata/grafana-pandas-datasource</a>.'
 
 
 @pandas_component.route('/search', methods=methods)