Polling rates causing server issues #213
-
|
Hello there Anyone who is a member of the LuxPower UK Facebook group may have seen a post recently where the recent issues with LuxPower servers going down has been blamed on people using APIs to poll servers too frequently: https://www.facebook.com/groups/438696100079367/permalink/1388372131778421/ I use lxp-bridge with node-red to create a local dashboard (I don't use HA). Sorry if I get my terminology wrong here, but I call each input group (1, 2 and 3) every 60s through MQTT to get the data I want for my dashboard. Based on the discussion in that post I've changed it to every 3 minutes until I better understand if I'm part of the problem. Some people have replied and said the issue relates mostly to use of a REST API rather than the kind of direct calls made by lxp-bridge and LuxPython but I am not clear on this. I know in your documentation you mention how every time you call for data through lxp-bridge the Chinese servers see the request, but someone said this may no longer be the case in newer firmwares (at least for inputs 1 & 2). Could you please clarify this situation as I certainly don't want to have my access to the Chinese servers blocked (I haven't replicated all the functionality locally) but equally I'd rather not be restricted to 3-5 minute poll rate. Thanks for any advice you can give! |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
Hello :) Not a member of the Facebook group but the Octopus Energy forum has a few posts about slow server performance too. All the more reason to avoid them entirely and have full local control of your inverter in my view ;) If there's any particular functionality we can help with let us know. Although I want to keep lxp-bridge usage-agnostic, might be things that can be done, depends :) So to answer your points, first off obviously lxp-bridge itself never ever makes any direct requests to China (or anywhere else!), neither to their REST API or to the servers that the inverter talks to. The only device lxp-bridge talks to is the inverter itself. So what happens when we send requests to the inverter is generally the answers are broadcast out to all connected clients. Since by default that will include Luxpower themselves, if lxp-bridge, for example, request inputs/1 (that's the name I gave it in lxp-bridge, technically speaking it means input registers 0 to 39), then when the inverter replies, it gets sent to lxp-bridge and also to China. As far as I know the only way to prevent this is basically to disconnect from China. That may mean firewalling, removing default route, or trying to remove the IP from the dongle's web interface, that sort of thing. What I have noticed in the past is that Luxpower need to see all the input registers in order to log a database row in their server. So that's inputs/1, inputs/2 and inputs/3 (which again, technically, translates to registers 0-39, 40-79, and 80-119). If you only request the first 2, then they won't log a database row, but I do think they still see the data for those two requests. Couldn't comment on whether newer firmwares have changed this though, I've never had mine updated. Hpoe that helps clear things up a bit! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Chris Thank you for your comprehensive response. I don't think there's anything missing from lxp-bridge that would prevent me from replicating the external functionality on my local dashboard, it's more of a motivation issue :) The only reason I use node-red is that I have some previous experience with it, plus an RPi 2 I can run it on easily. If I was starting from scratch I'm sure HA would be a better option. Below is the message posted on the FB group by (I believe) someone from infinity:
There is much discussion after this trying to get clarification of exactly what this means, but I agree that the only way to be safe is to disconnect from China. I still use the LuxPower app for the graphs (I only show instantaneous data on my dashboard, databases and graphing are beyond my very limited capabilities) plus I use the system status page frequently because my batteries have a habit of falling over from time to time. The only data I use from inputs/3 are the battery charge/discharge limit so I can probably limit that to a less frequent interval if it means less database traffic for LuxPower. Hopefully all will become clear in the near future on that front. Thanks again |
Beta Was this translation helpful? Give feedback.
-
|
Chris,
The impression I get is that lux logs the regular status and only does a
pass through recording of the setting when the user requests then on the
front end. There is no evidence that all of the changeable settings are in
the statistics pages, such as charge times etc.
I think you are right to say the the transmission of data via lxp bridge
will cause a broadcast to the china servers and I am aware that overtime
lux have reduced the frequency. A couple of years ago the update around 1
minute in frequency, now it is longer.
I still think lxp bridge is a great project and the right way to monitor
the unit. The fact people ( I did at one point and soon figured it was a
bad idea) have started to directly go to the lux servers shows that there
is a need for more integration. Maybe the ask for lux is to support local
integration with a small update to the dongles so users can officially work
locally. A bit like Victron etc al.
As long as lxp bridge will keep doing it's good work, I think the load can
be kept away from the lux servers. Please remember that lux and infinity
solutions are businesses and need to pay bills. Hosting a web site and a
monitoring platform (on AWS from what I can see) is not free. The more it
is abused, the more a subscription model might be needed. Or maybe the web
interface needs securing and a formal rate limiting API interface is
needed. Using a combo of security objects for web site with short lived jwt
tokens might make it possible to limit load. If the token need to be
refreshed regularly, you can block the python mechanism. Then provide a
formal API method which is rate limited with a bunch of caching servers in
front of the main servers . The API servers respond with a configured
refresh interval.
From a sizing point of view, maybe lux need to limit the amount of
historical data they keep?
Myenegi had the same problem and ended up.limitjng the functionality
massively.
Sorry this email is a bit of a ramble, but the short version is local data
capture is best.
John
…On Sat, Oct 21, 2023, 17:41 Chris Elsworth ***@***.***> wrote:
Hello :)
Not a member of the Facebook group but the Octopus Energy forum has a few
posts about slow server performance too. All the more reason to avoid them
entirely and have full local control of your inverter in my view ;) If
there's any particular functionality we can help with let us know. Although
I want to keep lxp-bridge usage-agnostic, might be things that can be done,
depends :)
So to answer your points, first off obviously lxp-bridge itself never ever
makes any direct requests to China (or anywhere else!), neither to their
REST API or to the servers that the inverter talks to. The only device
lxp-bridge talks to is the inverter itself.
So what happens when we send requests to the inverter is generally the
answers are broadcast out to all connected clients. Since by default that
will include Luxpower themselves, if lxp-bridge, for example, request
inputs/1 (that's the name I gave it in lxp-bridge, technically speaking it
means input registers 0 to 39), then when the inverter replies, it gets
sent to lxp-bridge and also to China.
As far as I know the only way to prevent this is basically to disconnect
from China. That may mean firewalling, removing default route, or trying to
remove the IP from the dongle's web interface, that sort of thing.
What I have noticed in the past is that Luxpower need to see all the input
registers in order to log a database row in their server. So that's
inputs/1, inputs/2 and inputs/3 (which again, technically, translates to
registers 0-39, 40-79, and 80-119). If you only request the first 2, then
they won't log a database row, but I do think they still see the data for
those two requests. Couldn't comment on whether newer firmwares have
changed this though, I've never had mine updated.
Hpoe that helps clear things up a bit!
—
Reply to this email directly, view it on GitHub
<#213 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AN6SNQLJ7SEYRH5WDVQZGXLYAO3XTAVCNFSM6AAAAAA6KBUQUSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNBWGE2TA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
Chris,
I just reviewed the data logged on the lux server site and the data is
showing updates every 5 minutes at a minimum. There are lots of gaps in the
data, so it looks like the servers are not keeping up at all. Sometimes
there is only data for 3 data points in an hour.
Reviewing what I can see from lxp bridge shows a regular update every 5
minutes. My config pushes the data to a mysql database that I then render
on Grafana. I use Home Assistant as a method of updating the parameters for
charging etc. For interest, I have two ac units which sometimes charge off
each other, standby mode is useful for stopping that. But for long term I
am likely to shift the units on different phases.
This backs up my previous email, local recording is best. 5 minute data
points are enough to monitor and reconfigure units based on circumstance.
To be honest, once the number of sun hours drops below 13 hours the
batteries are being charged on cheap rate and not not completely on solar.
The solar yield is too low.
Again, local is best and I believe lux basically has a loading and sizing
problem.
John.
…On Sat, 21 Oct 2023 at 13:49, John RUSH ***@***.***> wrote:
Chris,
The impression I get is that lux logs the regular status and only does a
pass through recording of the setting when the user requests then on the
front end. There is no evidence that all of the changeable settings are in
the statistics pages, such as charge times etc.
I think you are right to say the the transmission of data via lxp bridge
will cause a broadcast to the china servers and I am aware that overtime
lux have reduced the frequency. A couple of years ago the update around 1
minute in frequency, now it is longer.
I still think lxp bridge is a great project and the right way to monitor
the unit. The fact people ( I did at one point and soon figured it was a
bad idea) have started to directly go to the lux servers shows that there
is a need for more integration. Maybe the ask for lux is to support local
integration with a small update to the dongles so users can officially work
locally. A bit like Victron etc al.
As long as lxp bridge will keep doing it's good work, I think the load can
be kept away from the lux servers. Please remember that lux and infinity
solutions are businesses and need to pay bills. Hosting a web site and a
monitoring platform (on AWS from what I can see) is not free. The more it
is abused, the more a subscription model might be needed. Or maybe the web
interface needs securing and a formal rate limiting API interface is
needed. Using a combo of security objects for web site with short lived jwt
tokens might make it possible to limit load. If the token need to be
refreshed regularly, you can block the python mechanism. Then provide a
formal API method which is rate limited with a bunch of caching servers in
front of the main servers . The API servers respond with a configured
refresh interval.
From a sizing point of view, maybe lux need to limit the amount of
historical data they keep?
Myenegi had the same problem and ended up.limitjng the functionality
massively.
Sorry this email is a bit of a ramble, but the short version is local data
capture is best.
John
On Sat, Oct 21, 2023, 17:41 Chris Elsworth ***@***.***>
wrote:
> Hello :)
>
> Not a member of the Facebook group but the Octopus Energy forum has a few
> posts about slow server performance too. All the more reason to avoid them
> entirely and have full local control of your inverter in my view ;) If
> there's any particular functionality we can help with let us know. Although
> I want to keep lxp-bridge usage-agnostic, might be things that can be done,
> depends :)
>
> So to answer your points, first off obviously lxp-bridge itself never
> ever makes any direct requests to China (or anywhere else!), neither to
> their REST API or to the servers that the inverter talks to. The only
> device lxp-bridge talks to is the inverter itself.
>
> So what happens when we send requests to the inverter is generally the
> answers are broadcast out to all connected clients. Since by default that
> will include Luxpower themselves, if lxp-bridge, for example, request
> inputs/1 (that's the name I gave it in lxp-bridge, technically speaking it
> means input registers 0 to 39), then when the inverter replies, it gets
> sent to lxp-bridge and also to China.
>
> As far as I know the only way to prevent this is basically to disconnect
> from China. That may mean firewalling, removing default route, or trying to
> remove the IP from the dongle's web interface, that sort of thing.
>
> What I have noticed in the past is that Luxpower need to see all the
> input registers in order to log a database row in their server. So that's
> inputs/1, inputs/2 and inputs/3 (which again, technically, translates to
> registers 0-39, 40-79, and 80-119). If you only request the first 2, then
> they won't log a database row, but I do think they still see the data for
> those two requests. Couldn't comment on whether newer firmwares have
> changed this though, I've never had mine updated.
>
> Hpoe that helps clear things up a bit!
>
> —
> Reply to this email directly, view it on GitHub
> <#213 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN6SNQLJ7SEYRH5WDVQZGXLYAO3XTAVCNFSM6AAAAAA6KBUQUSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNBWGE2TA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***
> com>
>
--
Regards
John RUSH
|
Beta Was this translation helpful? Give feedback.
Hello :)
Not a member of the Facebook group but the Octopus Energy forum has a few posts about slow server performance too. All the more reason to avoid them entirely and have full local control of your inverter in my view ;) If there's any particular functionality we can help with let us know. Although I want to keep lxp-bridge usage-agnostic, might be things that can be done, depends :)
So to answer your points, first off obviously lxp-bridge itself never ever makes any direct requests to China (or anywhere else!), neither to their REST API or to the servers that the inverter talks to. The only device lxp-bridge talks to is the inverter itself.
So what happens when we send requests to t…