Real-time energy monitoring tweak

[hepcat72]

For some reason, I can't copy message links from Discord, so to summarize what leads me to opening this discussion...

I am trying to use wattage changes as automation triggers. My goal is to flip a dumb switch on a device plugged into a sonoff and make other things happen. I have tried both PowerDelta and a combination of PowerLow/PowerHigh rules, but there's a delay of around 2 seconds regardless of the method for detecting the wattage change. I have a Sonoff S31 15A.

[For further background, I have also tried a Currant brand outlet and I could set the same threshold on it and it detected the wattage change with almost no delay. The only problem with that was they didn't have a means of getting the triggers into Node RED.]

So I've been studying the code to see if I can tweak anything to make detection of wattage changes closer to real-time (both via PowerDelta and low/high rules thresholds - which, BTW appear to be only negligibly different), ideally a fraction of a second. I found where Energy.active_power is calculated and set and the code looks like it should report power changes within 1 second. My initial hunch was that Energy.power_history might have been mis-handled (since my observation:

https://drive.google.com/open?id=1_C4OFqSyKh6MxeL2yJqyaPd11vWmHUYI

was that changes were detected in about 2 seconds and power_history contains readings from 2 seconds before "now" [note, I move the mouse when I flip the switch in the above video: up=on, down=off])... but it all looks right.

My other hunch is that the power readings are averaged over a window, sampled once a second, thus causing an immediate change to be caught on a delay (>1s). My oberservations in the web GUI support this. When I flip the switch, it goes from 1W to 6W (1s later), then to the target 11W (1s after that).

I have a question. Power readings are obtained from the sensor and active_power is calculated in a loop here:

Tasmota/tasmota/xnrg_02_cse7766.ino

Line 145 in dcbb3f1

while (CseSerial->available()) {

It must be those values that possibly have the windowed-average I'm hoping to find. However, I can't determine how frequently that while loop cycles. Is it potentially looping faster than once a second and is that looping independently of the loop that checks the margins:

Tasmota/tasmota/tasmota.ino

Line 379 in 628f17d

void loop(void) {

I guess even if that loop runs once a second, it's the sender that would be implementing the windowed average. Otherwise, the while loop would catch the change immediately...

I guess I'm just thinking out loud here.

Oh, BTW, the link to the PDF in the comments is stale:

Tasmota/tasmota/xnrg_02_cse7766.ino

Line 28 in dcbb3f1

* Based on datasheet from http://www.chipsea.com/UploadFiles/2017/08/11144342F01B5662.pdf

I found the doc here:

https://dl.itead.cc/S31/CSE7766.pdf

[Frogmore42]

You might want to back up and explain more about why you are choosing this method. What are you really trying to do?
There are reasons that using power utilization might be the best solution, but normally it is better to use an actual switch to a device. Some of the Shelly devices can take a high voltage switch input and can run Tasmota. You will get very quick response from that.

When you put power measuring circuitry in between, you get the delay of measuring power accurately (averaging is usually involved), and the delay of reporting, which is not instant.

Looking at the code:

1. CseSerialInput gets called whenever there is serial data available from the chip. The datasheet is not really clear on how often this will be, but it does have this:

Note: Maximum measurement is about 16s for current cycle, voltage cycle and active
power cycle. It will prompt that the measuring range of corresponding cycle is exceeded
after exceeded, as shown in Table 3-3.

Later it has this:

The unit for voltage cycle, current cycle and power cycle is us. When signal cycle
is larger than 1s, the signal cycle will be outputted to the user immediately, and it will
prompt through Adj.4(5 or 6) that it is an incomplete cycle. (For example, if signal
cycle is 3.5s, 1s signal cycle will be outputted to the user when the accumulated cycle
counts to 1s, and 1.5s signal cycle will be outputted to the user when the accumulated
cycle counts to 1.5s instead the message is received by the user after reaching 3.5s so
that the response speed is improved. When the signal amplitude to be measured
declines, the signal cycle will increase.)

That says to me that, at best you are going to get a few samples per second for normal values of power.
2. Once CseSerialInput gets a complete transmission from the chip, it calls CseReceived()
3. CseReceived() interprets the data and sets some variables with the high-level information
4. CseEverySecond() gets called (every second) to validate the data and then calls EnergyUpdateToday()
5. EnergyUpdateToday() takes the data and updates its variables
6. EnergyEverySecond() takes the variables and does some checks and then calls EnergyMarginCheck()
7. EnergyMarginCheck() does the actual check you want

I am not really sure the order of the two modules (Energy (xdrv03) and CSE7766 (xnrg_02)). But, there is a lot of code there. It is NOT designed to be used for things that need to have a near instant response (as you have discovered, it can take some seconds for the action to occur).

The WebUI updates 1/sec. When there is 0/low power utilization the CSE7766 takes more than a second to calculate it. So, when the power changes it will take it some time to figure this out. Power measurement takes time, you probably won't be able to make it work much faster. If you need real time support, use an actual switch input.

[hepcat72]

I fail to understand what your resistance is to this endeavor, but thanks for your words of caution. Like I said, there is at least 1 smart outlet out there that can detect wattage changes in a fraction of a second (near instant). And given the poor wording in the documentation, and language that suggests mechanisms for improved response time, and the fact that tasmota was not designed with fast energy change response times, it seems entirely possible that a minor change in the timing of the computation of the calibration value could lead to higher resolution of wattage values.

There's a significant difference between the solution you propose and the one I'm advocating. Your solution involves non-trivial effort if you want to change the controlling device. With my solution, you can swap out any device with a power switch and use it as a (semi-)smart switch (i.e. "reporting" switch).

Unless you can prove that the S31 hardware is sending window-averaged values (i.e. the calculation is hard-wired into the board), I believe that it's entirely possible to implement this change, though it's also entirely possible that I'm not up to the task.

Regarding the risks, any mistake is possible, but the only change I'm thinking about is including a re-calculation of the calibration value when ADJ.4 shows that an adjustment is indicated. I will follow the code to see what side-effects that could have, but currently, it seems like it's just for reporting, and does not affect what happens electrically. Again, I could be wrong, so I appreciate the words of caution.

[hepcat72]

Actually, I don't even need to alter the saved calibration value. I can just put in an else that uses a temporary value.

[Frogmore42]

I am happy that you are looking at this. I hope that you are successful. If you are happy spending your time investigating this, that is great. That is what makes open source so great. You are interested in something that is important to you. You share that passion with the world and probably someone will benefit from your work.

Most people just want a solution to their problem. Energy monitoring circuits are not designed for controlling things, but that doesn't mean they can't be used for it. I am looking at the Shelly 2.5. It has a real energy monitoring IC with a real datasheet and supports 2 channels for energy values. Looking at the datasheet, it should be possible to query the IC frequently (maybe even every 50ms). The IC in the S31 does not have this capability. That is what my comment about polishing a turd is about. The S31 is fine for energy monitoring, but, as you have discovered, it is not great for getting information quickly. The IC in the POW 1 is much worse with measuring low power levels and takes several seconds to get an answer at low wattage utilization.

[Frogmore42]

My comment about the risks is during debugging, it is generally necessary to have a serial connection to the device to see what it is doing. It also typically requires downloading new code to the device multiple times. Doing this safely is important.

[hepcat72]

I saw that there is a commented out debug statement in the energy monitoring code. I wondered if I would see that in the console if I uncommented it... Maybe it goes to the serial connection you're referring to(?).

But theoretically, I should be able to see if any change works just by watching the energy values in the web interface, since (when I flip the switch on the device plugged into the S31) it goes from 1 to 6 to 11 watts in 0 to 1 to 2 seconds of elapsed time respectively.

And again, it's entirely possible (perhaps even likely) that I'm not interpreting the verbiage in the doc correctly and that no change I make will have the desired affect. It all hinges on whether it reports a signal early, as opposed to just late (which is how I interpret the code was written to consume).

Anyway, I have so many personal projects going, I expect my progress to be really slow - like measured in months. So far, I've just been looking into it on weekend mornings while having coffee.

[arendst]

If you want to see quick response from a power switch flick you should start reading the response from the power management chip.

In the case of your S31 enable weblog 4 and study the data received from the chip. In there you would like to see rapid changes if you flick the switch. If no rapid changes are reported all other software changes are useless.

[hepcat72]

Wow, thanks. That's great! I'll definitely put this on the top of my stack. Might save me from a bunch of wasted effort.

[hepcat72]

I played around with weblog 4 today. Definitely helpful!

I spent the day plotting at this data (normalized) while flipping the switch. The coefficient doesn't change as I had inferred, however, looking at the other data and reading page 20 of the docs a few times, I believe I see a solution to the smoothed wattage data. I believe the solution lies in the V1R and V2R values when adj indicates the energy data is "invalid". It looks like 2 or 3 bits are flipping in ck that can be used to determine V1R and V2R. I did some manual adjustments based on the formulas and it looks like the wattage can be closer to "real" when V1R and V2R are either 2 or 1, as suggested in the doc on page 20. So far, it's just a hunch, but there are definite jumps in the value of ck that could suggest a bit flip at just the times when the V1R and V2R adjustments are needed. The doc suggests that V1R and V2R are not needed when the calibration is good, and if I infer that ADJ indicating an "invalid" energy reading to mean that the calibration is not good, and I figure out which bits in ck are the important ones, I think that using V1R/V2R as each either 1 or 2 in the calculation, the wattage values will instantly be more accurate.

However, regardless of what I do, there's nothing in the data that suggests I can get the switch flip event to consistently under a second (unless I change the reporting frequency). All the data stays level for a little over a second, regardless of what I do... I've been using a low powerhigh, and high powerlow, and a low powerdelta, which seems to make a quantum difference in the response time. However, if I can figure out V1R and V2R, that issue would go away.

Here's the data that I think is relevant. Everything is normalized via min/max from 0 to 100:

The purple line shows seconds step-wise. I added 3 type of spikes for events in the log. The yellow is a nonsense command I entered in the log at the exact time I flipped the switch. The navy and red spikes are the power threshold rule and the powerdelta events respectively. Green is the power cycle value from the mailbox. Black is my calculated active power. Light blue is the ADJ value (up=valid, down=invalid). And the brown line is the ck value, which I think may be used to extract values for V1R and V2R via bit flip.

Anyway, that's enough for this weekend. I'll play some more with it next weekend.