Merge pull request #318 from boblemaire/Add-integrations

Add integrations

Docs/index.rst

@@ -14,10 +14,12 @@ IoTaWatt documentation
     CT Configuration <CTconfig>
     Status Display <status>
     Outputs <outputs.rst>
+    Integrators <integrators>
     Uploaders <uploaders>
     Configure PVoutput <PVoutput>
     Configure Emoncms <Emoncms>
     Configure influxDB <influxDB>
+    Home Assistant <homeAssistant>
     Setting up Passwords <passConfig>
     Three-phase power <threePhase>
     CT Basics <CTbasics>

Docs/integrators.rst

@@ -0,0 +1,228 @@
+============
+Integrators
+============
+
+What are integrators?
+----------------------
+
+The datalog provides high speed access to all monotonic data derived 
+directly from the datalog or indirectly with output Scripts.  That is to say, if
+the total value only increases, or only decreases, everything works just fine.
+
+The exception is where a CT is configured with "allow negative values" as with 
+a mains CT where solar or other alternative energy sources can send excess
+energy to the grid.  In those cases, the total energy represented is the net 
+energy, not a monotonically increasing value. Additionally, Scripts that do 
+not result in monotonically increasing values may not produce the intended result.
+
+The individual five-second data in the datalog is signed is the net energy
+imported (+) or exported (-) during each five second interval. When that data is viewed in a broader context 
+of hours, the net does not provide accurate individual import and export values.
+
+To develop the individual total imported and exported energy values,
+IoTaWatt needs to process the five-second detail measurements 
+to accumulate the sums of the individual positive and negative values,
+and to save them as monotonic values that can
+be used in output Scripts.  That's what integrators do.
+
+How they work
+-------------
+
+As previously mentioned, the detail data exists in the five-second current datalog.
+When you create an integrator, an additional datalog is created to
+contain the integrated values. These are called integrations - in the mathematical sense, 
+not to be confused with combining as with integrations in Home Assistant terminology.
+
+A background Service is created to process the five-second datalog, by default 
+beginning with the previous 24 hours. This takes about ten minutes. The Integrator accumulates
+the positive and negative five-second values in the new integration datalog.
+
+When the background process has processed the historical data, it is *synchronized*
+with the current log. At this point the Service terminates and the
+datalog Service, which maintains the current log, takes on the task of updating
+the integration log as an atomic operation when writing new datalog entries.
+
+When the log is synchronized, 
+the integrated results are available for use in output Scripts (calculator)
+
+Creating an Integrator
+-----------------------
+
+Hover over |Setup| and click |Integrations| in the dropdown buttons.
+
+.. image:: pics/integrators/integratorsDisplay.png
+    :scale: 60 %
+    :align: center
+    :alt: Outputs Display
+
+
+This will list any integrations that you have already configured, 
+and allow you to click |add| to create a new one.  
+You can click |edit| on existing integrations to change or delete them.
+
+.. image:: pics/integrators/configureIntegrator.png
+    :scale: 60 %
+    :align: center
+    :alt: Configure Integrator
+
+From here, the process is identical to adding any other output.
+For a refresher, see `Adding a new Output <outputs.html>`_ 
+Integrations must be uniquely named using alpha-numeric characters and the
+underscore (_) character.
+
+The Script that is specified will be integrated to sum the positive and negative components.
+
+Using the data
+--------------
+
+Once the Integration is *synchronized*, it should remain so going forward.
+When creating a new output Script, there will be three new selections
+available in the inputs list.  
+
+.. image:: pics/integrators/inputsList.png
+    :scale: 60 %
+    :align: right
+    :alt: Inputs List
+
+They will all begin with the name of the
+integration with these appendages:
+
+  .net
+    Returns the net value of the Integration script, just as it
+    would if the script were used outside of an integration, except that
+    the result will be based only on data within the time period defined
+    by the Integration log.
+
+  .pos
+    Returns the sum of the positive five-second intervals within the requested period.
+
+  .neg
+    Returns the sum of the negative five-second intervals within the requested period.
+
+
+Note that for any period,  .net = .pos + .neg
+
+Tracking Grid import/export with solar
+--------------------------------------
+
+To track energy import and export to the grid, use an integration
+of a Script that calculates the power flow from the grid (+) and
+to the grid (-).
+
+Set "allow negative values" in the config for all mains.
+Insure that they indicate positive when importing power (at night).
+If they indicate negative at night, reverse the CT or check the
+"reverse" box in the config.
+
+Solar inverter connects after mains (most common)
+""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The most common service has the solar inverter connected to a breaker
+in the service panel, or to the mains cable(s) after the mains CT.
+With this configuration, the solar reduces the energy flow in the mains
+and will drive the mains negative when generation exceeds energy use.  
+
+.. figure:: pics/integrators/configureMainIntegrator.png
+    :scale: 60 %
+    :align: right
+    :alt: Configure Integrator
+
+    Split-phase mains example
+
+Create an output that is the sum of all of the mains.
+Typically a 230V single-phase service will have one main,
+a split-phase North-American servive will have two mains and
+a three-phase service will have three:
+
+  (single-phase)
+    grid = main_1
+
+  (split-phase) 
+    grid = main_1 + main_2 
+
+  (three-phase)
+    grid = main_1 + main_2 + main_3 
+
+Solar inverter connects before mains
+""""""""""""""""""""""""""""""""""""
+
+In this configuration, the solar inverter is connected to the mains feeder
+before (on meter side) the mains CT. Solar generation does not effect the energy measured
+by the mains CTs.
+
+.. figure:: pics/integrators/configureType1Solar.png
+    :scale: 60 %
+    :align: right
+    :alt: Configure Integrator
+
+    Single-phase solar connected before mains.
+
+You must subtract the solar from the grid total:
+
+  (single-phase)
+    grid = main_1 - (solar max 0)
+
+  (split-phase) 
+    grid = main_1 + main_2 - (solar max 0)
+
+  (three-phase)
+    grid = main_1 + main_2 + main_3 - (solar max 0)
+
+Now plot the grid and solar output using graph+ for yesterday.
+Hopefully the sun was shining and your plot looks something like this
+where you can see the solar directly reduce the grid power.
+
+.. image:: pics/integrators/gridoutput.png
+    :scale: 60 %
+    :align: right
+    :alt: grid plot
+
+When you are satisfied the grid output correctly reflects your grid power flow,
+delete the grid output and add an integrator named grid with that same Script.
+
+You will see the integration "grid" listed under the Data Logs tab in the Status Display.
+Wait about 10-15 minutes for the integration to process the last 24 hours of data.
+When the end time is the same as the current log, the integration is "synchronized"
+and you can use it to add import and export outputs.
+You can reference or query the import and export outputs to get accurate Wh. 
+
+.. image:: pics/integrators/addExport.png
+    :scale: 50 %
+    :align: left
+    :alt: add output "export"
+
+.. image:: pics/integrators/addImport.png
+    :scale: 50 %
+    :align: center
+    :alt: Add output "import"
+
+Home Assistant Energy
+---------------------
+
+If you followed the examples above, configuration of Home Assistant Energy is
+simple and easy.  Once you install a Home Assistant IoTaWatt Integration,
+configure Energy like this.
+
+.. image:: pics/integrators/homeAssistantEnergyConfig.png
+    :scale: 80 %
+    :align: center
+    :alt: Home Assistant Energy Config
+
+.. |Setup| image:: pics/SetupButton.png
+    :scale: 60 %
+    :alt: **Setup button**
+
+.. |Integrations| image:: pics/integrators/integratorsButton.png
+    :scale: 60 %
+    :alt: **Integrations button**
+
+.. |add| image:: pics/addButton.png
+    :scale: 60 %
+    :alt: **Add button**
+
+.. |edit| image:: pics/editButton.png
+    :scale: 70 %
+    :alt: **edit button**
+
+
+

Docs/uploaders.rst

@@ -75,7 +75,7 @@ Both solar generation and energy consumption data can be
 combined to provide a live 'Net' view of energy being  
 generated and consumed.
 
-.. figure:: pics/PVoutputDisplay.png
+.. figure:: pics/PVoutput/PVoutputDisplay.png
     :scale: 50 %
     :align: center
     :alt: **PVoutput Display**

Firmware/IotaWatt/CSVquery.cpp

@@ -514,13 +514,7 @@ void CSVquery::buildLine(){
         else {
             trace(T_CSVquery,64);
             double value = 0.0;
-            if(col->source == 'N'){
-                integrator *integration = (integrator *)col->script->getParm();
-                value = integration->run(_oldRec, _newRec, elapsedHours);
-            }
-            else {
-                value = col->script->run(_oldRec, _newRec, elapsedHours, col->unit);
-            }
+            value = col->script->run(_oldRec, _newRec, col->unit);
             trace(T_CSVquery,65);
             printValue(value, col->decimals);
         }

Firmware/IotaWatt/Emoncms_uploader.cpp

@@ -152,7 +152,7 @@ uint32_t emoncms_uploader::handle_write_s(){
             int index=1;
             while(script){
                 while(index++ < String(script->name()).toInt()) reqData.write(",null");
-                double value1 = script->run(oldRecord, newRecord, elapsedHours);
+                double value1 = script->run(oldRecord, newRecord);
                 if(value1 == value1){
                     if(script->precision()){
                         char valstr[20];

Firmware/IotaWatt/GetFeedData.cpp

@@ -217,16 +217,16 @@ uint32_t getFeedData(){ //(struct serviceBlock* _serviceBlock){
               *replyData += "null";
             }
             else if(reqPtr->queryType == 'V'){
-              *replyData += String(reqPtr->output->run(lastRecord, logRecord, elapsedHours), 1);
+              *replyData += String(reqPtr->output->run(lastRecord, logRecord, Volts), 1);
             }
             else if(reqPtr->queryType == 'P'){
-              *replyData += String(reqPtr->output->run(lastRecord, logRecord, elapsedHours), 1);
+              *replyData += String(reqPtr->output->run(lastRecord, logRecord, Watts), 1);
             }
             else if(reqPtr->queryType == 'E'){
-                *replyData += String(reqPtr->output->run((IotaLogRecord*) nullptr, logRecord, 1000.0), 3);
+                *replyData += String(reqPtr->output->run(nullptr, logRecord, kWh), 3);
             }
             else if(reqPtr->queryType == 'O'){
-              *replyData += String(reqPtr->output->run(lastRecord, logRecord, elapsedHours), reqPtr->output->precision());
+              *replyData += String(reqPtr->output->run(lastRecord, logRecord), reqPtr->output->precision());
             }
             else {
               *replyData += "null";