IoT-Energy-Logger

---

Prototype ESP8266 Modbus MQTT Google Cloud Energy Logger.

Overview

Google Cloud platform setup

1. Activate APIs

   • The very first step is to activate the APIs by going to APIs and Services and enabling the following:
     • PubSub (Required)
     • IoT Core (Required)
     • Cloud Functions (Optional but this is the best(cheapest) way to go)
     • BigQuery (Optional but this makes the most sense for datastorage)
     • Cloud Storage (Optional - Application dependant)
     • DataFlow (Optional - Application dependant)
     • Compute Engine (Optional - Application dependant)

   If forget one, don't worry, you'll receive helpful debug messages from the API down the line.

2. Set up Cloud Core Now that the APIs are enabled, set up the Cloud Core Iot functionality which is responsible for handling the incoming MQTT messages.

   • To set up Cloud Core you need to:
     • Create a new Registry
       • Choose an appropriate name
       • Select a region. I chose europe-west1 since it's the closest to South Africa. Low latency.
       • Create a PubSub topic for the registry.
     • Open your new Registry and create a new Device
       • Give it a name
       • Expand the drop down and go to Authentication
         • Select ES256
         • Now open Generate_ec_device_keys.py and run the script. This will produce a Private and Public elipitcal device key or EC Key. Paste the Public key value in the Authentication tab. The Private will be inserted into the ESP firmware. The EC Keys are also saved to a text file if you happen to forget to copy them.
         • Finalise by clicking Create
   • IMPORTANT - These details must appear as they were created here in the ESP firmware or else it will not work. The ESP will report via its serial monitor that it failed to get a JWT.
   • Alternatively, you can make use to the Cloud Device Manager.py which will do all of this for you. All you have to do is state the details and copy the Private key. This feature is coming soon and will be used to automate adding new devices.

3. Select your desired Pipeline

   • DataFlow - a codeless pipeline setup. It's very expensive($300/pm). Just learn the API.
   • CloudFunctions
     • PubSub to Cloud Storage. Saves data as a text file. Code and instructions here. CloudStorage is quite expensive if you start scaling up the size of your project. You are not only charged for storing your data but also retrieving that data from nearline storage. Since this function has to read all the data from the last CSV, append one line to it and rewrite the file, this is very inefficient and expensive over time($30/pm).
     • PubSub to BigQuery. Appends an SQL database. By far the best choice. Code and instructions here. For one device, posting every 10s, it's basically free. ie it costs $0.00. You get 10Gb storage and 1T processing per month for FREE. Read more here

• Setup DataFlow pipeline from MQTT payload to Cloud Storage. For those how are interested.
  • "Create a new job from Template"
  • Give it a name, select a region, select the template Pub/Sub to Text Files on Cloud Storage
  • Required parameters:
    • The Pub/Sub topic name can be found under the Pub/Sub tab in the left-hand panel. Click on the topic to open it, and copy the content of the Topic name
    • Assuming you've already created a bucket in Cloud Storage, the "output file directory in Cloud storage" must look something like gs://<yourBucketName>
    • "Output filename prefix" is the name of the text file
    • For "Temporary Location", just create a folder in your bucket called temp. Your location then becomes gs://<yourBucketName>/temp
    • Run Job, navigate to the Jobs tab in Dataflow and check if the status says Running

ESP 8266 - Node MCU Setup

Setup instructions and code found at https://github.com/GoogleCloudPlatform/google-cloud-iot-arduino

Code for Node MCU ESP8266 12E is found in Esp8266-lwmqtt.

Firmware will not compile unless you have the following Arduino libraries installed: Search in Library manager
"Google Cloud IoT Core JWT" by Gus Class
"ESPSoftwareSerial" by Dirk Kaar (may be removed in future versions of this project)
"MQTT" by Joel Gaehwiler

NODE MCU1.0 12E Pinout

Because the pin numbers on the board ARE NOT THE SAME as the internal pin numbers, please take note of the pinout below

Internal pin number referencing is used in Arduino script. See pinout.
Pin name   Board number   Internal pin number
RE      16         D0
DE      5         D1
RX      4         D2
TX       0         D3

Energy Meter

My project was completed using the SDM230 however I would rather recommend using a current transformer based meter such as the SDM120 CT. Inline meters such as the SDM230 requires some rewiring of the distribution board using 16mm^2 wire if you intend on measuring the main incoming power. This can be a timely exercise and if the terminals are not tight as hell, you can cause a fire as a result of the poor connection.

The SMD120 CT

• requires half the space
• no extra cabling to be purchase
• is much safer as the full load of the house/office doesn't go through it
• is much quicker to install
• you do not have to disconnect the power to the whole building to install it

SMD230 Modbus made by Eastron

Over The Air(OTA) updates

Firmware can be updated without a wired connection by flashing new firmware via Wifi. First:

• Convert the firmware to a binary file. This is done in the Arduino IDE under Sketch->Export compiled Binary.
• Put the ESP in flashing mode by connecting PIN D5 (pin 14 internal) to GROUND(0V).
• The ESP will create an access point with SSID AutoConnectAP and password password.
• Connect to the access point, go to Update and upload the binary file.

Current Issues

• DATA corruption
  • I suspect that there is an Interrupt Service Routine(ISR) from the Wifi operations that interrupts serial processes which leads to data loss. This is an intermittent issue but one that requires attention. Disabling the global ISR during serial operations causes the ESP to hang. Thinking about moving away from SoftwareSerial.
  • #Update - The issue has concerningly, "gone away by itself" after "fiddling" with the wires. Now its has been operating without fault for 2 weeks straight. With that being said, the wired connections are made with hobbies jumper cables not known for its signal integrity.
  • #Update - I am starting to believe that I have no idea what the actual problem is. The single phase meter works fine, the 3 phase meter seems to have TX and RX issues. It appears that the energy meter doesnt receive a(correct) command. As per the datasheet, "The slave will not respond at all if there is an error with the parity or CRC of the query." This provides some indication that a bit wise error may have occurred. On the very odd occasion, a very small or very large value is received from the energy meter. 1.035x10^-36 or 2.58x10^38. This gives more indication of a bit wise error. Running a byte count on the RX values will confirm this.
• Google is discontinuing Cloud Core from August 2023 and provide NO alternavive service in their announcement. Thanks. It is unclear whether they have another service that achieves the same goals. If not, then AWS here I come.
• The MODBUS CRC16(16 bit Cyclic Redundancy Check) calculation in firmware still need to be fixed. Modbus commands are hardcode with CRC already calculated using this online calculator. Its fine for my purposes, but if there any prospects of more functionality, this will have to be fixed. As this issue stands now, it requires a quirky serial print of the modbus command in hex format for it to work???... why the on gods green earth that makes it work is beyond me.
• When there is a power interruption and your router reboots, the ESP will not be able to find the SSID of your network and it will cause the ESP to hang and not join your network when the router has rebooted. Pressing issue if you're in SA

Future Work

• Python data visualisation
• Add local MQTT Server option by means of Raspberry Pi, with either local or cloud storage
• Add implementation for Raspberry Pi - To replace ESP. ESPs are dirt cheap and easy to get compared to Pi's but you end up paying for it in development time... a lot of it. I want to do remote firmware deployment but it will take time. It's very easy to do remote firmware deployment on a Pi...comparatively.
• Add a web server/web page that displays current and past energy usage with some analytics and trends.
• Add Wifi manager so that you dont have to hardcode wifi credintials
• Auto set up a new MQTT device. Must be scalable.
  • Will have to:
    • Modify Wifi manager sketch to give the option to specify details about the device
    • Add another cloud function triggered by HTTP, which creates a new registry(if device is in a different location)
    • Adds a new device to the registry(with given name)
    • Creates private and public keys. Saves public key to server and sends private key over HTTPS to ESP. Save private key to EPROM/flash
    • Cloud function:
      • Creates a new bucket for new device
      • Creates new PubSub topic
      • Creates new cloud function that acts as a pipeline between the PubSub data and cloud storage. OORRRR use the same cloud function pipeline and pass an extra argument to it(the topic name? device name?) that directs the output to the correct bucket.
• Add PubSub commands to select which energy parameters to read.
• Custom PCB that contains
  • 230V->3.3V
  • MAX458
  • ESP 32 or 8266
  • Option for LoRa or ZigBee