v0.8.1 - Set battery reserve, operation mode #78

emptywee · 2024-03-26T23:17:02Z

Added get_mode() function.
Added set_battery_op_reserve() function to set battery operation mode and/or reserve level. Likely won't work in the local mode.
Added basic validation for main class __init__() parameters (a.k.a. user input).

* Added `get_mode()` function. * Added `set_battery_op_reserve()` function to set battery operation mode and/or reserve level. Likely won't work in the local mode. * Added basic validation for main class `__init__()` parameters (a.k.a. user input).

emptywee · 2024-03-26T23:19:30Z

pypowerwall/local/pypowerwall_local.py

+        if r.status_code == 404:
+            log.debug('404 Powerwall API not found at %s' % url)
+            return None
+        if 400 <= r.status_code < 500:


In local mode any attempt to send payload to /api/operation results in 403. Should we treat it here as Unauthorized instead of re-attempting to re-login?

Yes! Good idea.

emptywee · 2024-03-26T23:20:50Z

pypowerwall/__init__.py

+            self._check_if_dir_is_writable(dirname)
+
+    @staticmethod
+    def _check_if_dir_is_writable(dirpath):


This needs to be tested on Windows, if it causes issues we can fix it or remove the whole thing or the particular piece of it which causes issues.

emptywee · 2024-03-26T23:21:19Z

pypowerwall/__init__.py

+            return data['real_mode']
+        return None
+
+    def set_battery_op_reserve(self, level: Optional[float] = None, mode: Optional[str] = None,


Open to suggestions if the name of the function is too long/odd.

It is fine, but we could abbreviate as set_operation() since that is the Tesla API and it describes what it does, but I would like to see two interfaces to this that align to their "get" equivalents:

set_reserve(level) set_mode(mode)

jasonacox · 2024-03-27T03:43:52Z

pypowerwall/__init__.py

+            try:
+                return json.dumps(response)
+            except JSONDecodeError:
+                log.error(f"Unable to dump response '{response}' as JSON. I know you asked for it, sorry.")


You're funny. 😂

jasonacox · 2024-03-27T03:50:40Z

This is great @emptywee !!!!

Added set_battery_op_reserve() function to set battery operation mode and/or reserve level. Likely won't work in the local mode.

This still allows us to run this as a standalone script (in cloud mode) to change these settings (e.g. cron jobs or embedded in other scrips). Technically (read: Jason hasn't tried) a user of the library can instantiate two instances, one for local, one for cloud and use that one for control commands.

Will a local mode user get enough of an error to indicate they need to swtich to cloud mode? We could add that as a log error.

emptywee · 2024-03-27T04:40:09Z

Will a local mode user get enough of an error to indicate they need to swtich to cloud mode? We could add that as a log error.

With some modifications to error handling, it can produce:

Testing set_battery_op_reserve()...
DEBUG:https://192.168.1.1:443 "GET /api/operation HTTP/1.1" 200 117
DEBUG:https://192.168.1.1:443 "POST /api/operation HTTP/1.1" 403 115
ERROR:403 Unauthorized by Powerwall API at https://192.168.1.1/api/operation - Endpoint disabled in this firmware or user lacks permission
Set battery operation mode to backup failed.

* Added `set_mode()` function. * Added `set_reserve()` function. * Handle 401/403 errors from Powerwall separately in local mode. * Handle 50x errors from Powerwall in local mode.

emptywee · 2024-03-27T04:52:02Z

@jasonacox ok see 8d100fa

emptywee · 2024-03-27T04:54:27Z

Added set_mode() function.
Added set_reserve() function.
Handle 401/403 errors from Powerwall separately in local mode.
Handle 50x errors from Powerwall in local mode.

emptywee · 2024-03-27T05:06:20Z

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

jasonacox · 2024-03-27T06:51:31Z

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

jasonacox · 2024-03-27T07:05:01Z

pypowerwall/local/pypowerwall_local.py

                # Rate limited - Switch to cooldown mode for 5 minutes
                self.pwcooldown = time.perf_counter() + 300
                log.error('429 Rate limited by Powerwall API at %s - Activating 5 minute cooldown' % url)
-                # Serve up cached data if it exists
+                # Serve up cached data if it exists (@emptywee: this doesn't look like we serve up cached data here?)


This comment needs to be removed. If we hit this, we are at the TTL for the cached objects which is what the user expects us to do. After that TTL we should send fresh data or return None indicating something bad happened or no connection.

emptywee · 2024-03-27T14:28:23Z

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

I'll test with my pw today, see how it behaves.

emptywee · 2024-03-27T19:24:09Z

Well, in case of token expiration, they respond with 401:

< HTTP/2 401 
< content-type: application/json
< set-cookie: AuthCookie=; Path=/; Max-Age=0
< set-cookie: UserRecord=; Path=/; Max-Age=0
< x-content-type-options: nosniff
< content-length: 38
< date: Wed, 27 Mar 2024 19:22:39 GMT
< 
* Connection #0 to host 192.168.102.64 left intact
{"code":401,"message":"Token Expired"}

Should we adjust the simulation server to mirror this behavior if we want to test how the library re-authenticates?

jasonacox · 2024-03-28T03:07:04Z

Should we adjust the simulation server to mirror this behavior if we want to test how the library re-authenticates?

Yes, I'm good with that.

emptywee · 2024-03-28T04:12:14Z

Yes, I'm good with that.

Would you be able to build a new docker image for the pwsim? It is used in PR checks as image: jasonacox/pwsimulator and I cannot update the image in Docker Hub. Unless we want to run the pwsim from the source code of the PR? For that we'll need to change the GitHub Actions workflow, I think.

P.S. I updated the pwsim stub.py to respond with 401 to simulate an expired token.

emptywee · 2024-03-28T04:15:11Z

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

…into pr/78

jasonacox · 2024-03-28T04:48:32Z

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

I pushed the update for pwsimulator and it passes now. I've always wanted to investigate how to do the automated docker push, but haven't. I have a build script that does it with little effort and typically would only run this when we cut a release.

docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm/v7 --push -t jasonacox/pwsimulator:${VER} .

emptywee · 2024-03-28T04:52:19Z

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

I'll test with my pw today, see how it behaves.

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

I pushed the update for pwsimulator and it passes now. I've always wanted to investigate how to do the automated docker push, but haven't. I have a build script that does it with little effort and typically would only run this when we cut a release.
docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm/v7 --push -t jasonacox/pwsimulator:${VER} .

https://docs.github.com/en/actions/publishing-packages/publishing-docker-images this?

emptywee · 2024-03-28T13:53:40Z

Nice. I can make some little change to pwsim to test the new gh workflow :)

..and test the new gh workflow

emptywee · 2024-03-28T17:35:53Z

@jasonacox hmm, do you need to enable the new workflow somewhere? I didn't see it kicked off on my push to pwsimulator... or it has to be in the main branch?

emptywee · 2024-03-28T17:36:49Z

Oh, it ran in my/forked repo =)

Weird.

emptywee · 2024-03-31T14:47:35Z

Ah, glad that you find it helpful and not irritating :) was a bit afraid of the latter!

emptywee · 2024-04-02T02:17:12Z

@jasonacox How's testing going? Ready to merge or not yet?

jasonacox · 2024-04-02T07:51:01Z

I haven't had time to troubleshoot why the force option is not working with the new functions. It may not be passing it along to the poll(). Do you have time to check?

>>> import pypowerwall
>>> pypowerwall.set_debug(True)
>>> pw = pypowerwall.Powerwall(host="",email=email)
>>> pw.get_reserve(force=True)
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
21.0

emptywee · 2024-04-02T13:39:03Z

Oh, absolutely. Thought you figured it out already and I was sitting duck waiting for you to merge it :P

emptywee · 2024-04-02T17:45:40Z

@jasonacox so, I looked at it and it's coming from the part which I didn't really touch, if you check the original code from 0.7.x, e.g.:

pypowerwall/pypowerwall/cloud.py

Lines 202 to 248 in 7b1fc45

    
               def _site_api(self, name, ttl, **kwargs): 
        
                   """ 
        
                   Private function to get site data from Tesla Cloud using 
        
                   TeslaPy API.  This function uses a lock to prevent threads 
        
                   from sending multiple requests to Tesla Cloud at the same time. 
        
                   It also caches the data for ttl seconds. 
        
                   Arguments: 
        
                       name - TeslaPy API name 
        
                       ttl - Cache expiration time in seconds 
        
                       kwargs - Variable arguments to pass to API call 
        
                   Returns (response, cached) 
        
                       response - TeslaPy API response 
        
                       cached - True if cached data was returned 
        
                   """ 
        
                   if self.tesla is None: 
        
                       log.debug(f" -- cloud: No connection to Tesla Cloud") 
        
                       return (None, False) 
        
                   # Check for lock and wait if api request already sent 
        
                   if name in self.apilock: 
        
                       locktime = time.perf_counter() 
        
                       while self.apilock[name]: 
        
                           time.sleep(0.2) 
        
                           if time.perf_counter() >= locktime + self.timeout: 
        
                               log.debug(f" -- cloud: Timeout waiting for {name}") 
        
                               return (None, False) 
        
                   # Check to see if we have cached data 
        
                   if name in self.pwcache: 
        
                       if self.pwcachetime[name] > time.perf_counter() - ttl: 
        
                           log.debug(f" -- cloud: Returning cached {name} data") 
        
                           return (self.pwcache[name], True) 
        
                   try: 
        
                       # Set lock 
        
                       self.apilock[name] = True 
        
                       response = self.site.api(name, **kwargs) 
        
                   except Exception as err: 
        
                       log.error(f"Failed to retrieve {name} - {repr(err)}") 
        
                       response = None 
        
                   else: 
        
                       log.debug(f" -- cloud: Retrieved {name} data") 
        
                       self.pwcache[name] = response 
        
                       self.pwcachetime[name] = time.perf_counter() 
        
                   finally: 
        
                       # Release lock 
        
                       self.apilock[name] = False 
        
                       return (response, False)

it doesn't honor any kind of force flags there. I can add it there, though. What would you say?

jasonacox · 2024-04-03T03:09:29Z

You're amazing! Great find. Yes, the main cloud use up to this point was "read only" to get the metric data which is delayed in the cloud anyway, so it hasn't surfaced as an issue. However, with your great work here, we now have "write" functions that could change state that would not show as changed until the cache TTL expires. That doesn't seem ideal. In fact, I wonder if get_reserve() and get_mode() should always be forced by default?

I am still good with merging this but will wait if you want to try to add the force feature first.

jasonacox · 2024-03-28T04:53:37Z

pypowerwall/__init__.py

+                alerts.append(alert)
+            if grid_status.get('grid_services_active'):
+                alerts.append('GridServicesActive')
+


@emptywee - I'm slipped this in for 0.8.1 instead of cutting a new PR. 😁

@mcbirse - The data from grid_status represents some of what we lost but also adds other alerts showing power transition and grid services state. There are a few other booleans that indicate alert states that we may want to use (e.g. updating which could be inverse as old FWUpdateSucceeded).

jasonacox · 2024-03-31T08:25:44Z

proxy/server.py

@@ -148,7 +152,8 @@ def get_value(a, key):
 try:
    pw = pypowerwall.Powerwall(host, password, email, timezone, cache_expire,
                               timeout, pool_maxsize, siteid=siteid,
-                               authpath=authpath, authmode=authmode)
+                               authpath=authpath, authmode=authmode,
+                               cachefile=cachefile)


This was the issue. The proxy was using the cachefile default (local directory .cachefile) which would not be writable. This has been an issue for a long time! It wouldn't surface because the service would just use the password to log in at restart. However, that creates more load on the Powerall.

@emptywee your init check code caught this! Love that! ❤️

I added some additional details to the init messages to help troubleshoot and identify. I kept hunting the authpath one but it was really the cachefile one.

emptywee · 2024-04-03T03:15:00Z

Sure, adding the force flag should be trivial at this point. We can clear certain cache entries once "write" operation is complete.

…le, invalidate respective cache data on write operations

emptywee · 2024-04-03T04:52:06Z

@jasonacox I was thinking something like this c46b479

jasonacox · 2024-04-04T04:31:26Z

Nice @emptywee ! I'm testing...

jasonacox · 2024-04-04T05:21:31Z

Ok, this is odd. When you set_reserve(), and then try get_reserve() it will return None from cache. Something is wrong with the cache logic. But, if you force=True, it does work.

>>> pw.set_reserve(20)
DEBUG: -- cloud: Request for /api/operation
DEBUG:Invoking 0 protected resource request hooks.
... truncated ...
>>> pw.get_reserve()
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
>>>
>>> pw.get_reserve(force=True)
DEBUG: -- cloud: Request for /api/operation
DEBUG:Invoking 0 protected resource request hooks.
[... trim ...]
DEBUG: -- cloud: Retrieved SITE_CONFIG data
20.0
>>>
>>> pw.get_reserve()
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
20.0
>>>

emptywee · 2024-04-04T05:27:59Z

Weird, I'll double check the caching condition there. Must be something odd there.

emptywee · 2024-04-04T05:48:08Z

@jasonacox yeah it was a silly thing. Can you check now?

Sorry it took that long, I was taking a shower :)

# Conflicts: # pypowerwall/local/pypowerwall_local.py

emptywee · 2024-04-04T05:52:20Z

Calling it a night now, but it should work now as intended.

jasonacox · 2024-04-05T03:10:27Z

It works!! Thanks @emptywee !

I'm testing jasonacox/pypowerwall:0.8.1t52-beta4 - should be ready for merge soon.

jasonacox · 2024-04-05T04:04:18Z

✅ Tests look great. Thanks @emptywee !!! Great enhancements.

jasonacox · 2024-04-05T04:17:07Z

Library: https://pypi.org/project/pypowerwall/0.8.1/

Proxy Container: jasonacox/pypowerwall:0.8.1t52

emptywee · 2024-04-05T04:19:22Z

Woot woot!

mcbirse · 2024-04-05T07:43:35Z

Brilliant work guys! Looks like you've both got all the testing covered... I'll test the new container at some point.

Sorry I haven't had a chance to be involved and help with this - got a work project on that is currently sapping all my time and energy.

jasonacox · 2024-04-06T03:05:22Z

No worries @mcbirse ! I understand that. :-)

I'm going to cut a PR for Powerwall-Dashboard that includes our latest pypowerwall container. It adds a handful of new alerts and seems to be performing well. A few in the community have already started running the beta.

A surprise to me was how much GridServicesActive flapps. I don't know what that may indicate but find it interesting.

emptywee · 2024-04-06T23:58:01Z

My stackstorm actions now got simplified :) woohoo!

mcbirse · 2024-04-07T12:31:44Z

A surprise to me was how much GridServicesActive flapps. I don't know what that may indicate but find it interesting.

I'm not 100% certain, however I was under the impression that GridServicesActive is supposed to show when your Powerwall is participating in a VPP event...

For it to be toggling so frequently however, could it be used to indicate something else as well, perhaps?

emptywee commented Mar 26, 2024

View reviewed changes

jasonacox reviewed Mar 27, 2024

View reviewed changes

## v0.8.1 - Set battery reserve, operation mode

8d100fa

* Added `set_mode()` function. * Added `set_reserve()` function. * Handle 401/403 errors from Powerwall separately in local mode. * Handle 50x errors from Powerwall in local mode.

rename set_battery_op_reserve() to set_operation()

49923fc

jasonacox reviewed Mar 27, 2024

View reviewed changes

pwsim respond with 401 to simulate an expired token

055de19

jasonacox added 2 commits March 27, 2024 21:29

Add grid_status alerts

64e6a5d

Merge branch '0.8.x/post' of https://github.com/emptywee/pypowerwall …

3e11809

…into pr/78

emptywee changed the title ~~## v0.8.1 - Set battery reserve, operation mode~~ v0.8.1 - Set battery reserve, operation mode Mar 28, 2024

Sim docker build-push action

507f538

Add version/author information to pwsim

f66a8e6

..and test the new gh workflow

jasonacox approved these changes Apr 3, 2024

View reviewed changes

pass force flag all the way down to _site_api() of the Cloud modu…

c46b479

…le, invalidate respective cache data on write operations

Add debug for local polling

16b2c92

properly check if we have a cached response

03dfa1b

igcherkaev added 2 commits April 4, 2024 00:49

properly check if we have a cached response

b85fc6f

Merge remote-tracking branch 'origin/0.8.x/post' into 0.8.x/post

5823d5e

# Conflicts: # pypowerwall/local/pypowerwall_local.py

Doc updates v0.8.1

c7252ac

jasonacox merged commit 00f823f into jasonacox:main Apr 5, 2024
12 checks passed

jasonacox mentioned this pull request Apr 6, 2024

v4.3.0 jasonacox/Powerwall-Dashboard#462

Merged

v0.8.1 - Set battery reserve, operation mode #78

v0.8.1 - Set battery reserve, operation mode #78

Conversation

emptywee commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonacox commented Mar 27, 2024

emptywee commented Mar 27, 2024

emptywee commented Mar 27, 2024

emptywee commented Mar 27, 2024

emptywee commented Mar 27, 2024

jasonacox commented Mar 27, 2024

Choose a reason for hiding this comment

emptywee commented Mar 27, 2024

emptywee commented Mar 27, 2024

jasonacox commented Mar 28, 2024

emptywee commented Mar 28, 2024 • edited Loading

emptywee commented Mar 28, 2024

jasonacox commented Mar 28, 2024

emptywee commented Mar 28, 2024

emptywee commented Mar 28, 2024

emptywee commented Mar 28, 2024

emptywee commented Mar 28, 2024 • edited Loading

emptywee commented Mar 31, 2024

emptywee commented Apr 2, 2024

jasonacox commented Apr 2, 2024

emptywee commented Apr 2, 2024

emptywee commented Apr 2, 2024

jasonacox commented Apr 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emptywee commented Apr 3, 2024

emptywee commented Apr 3, 2024

jasonacox commented Apr 4, 2024

jasonacox commented Apr 4, 2024

emptywee commented Apr 4, 2024

emptywee commented Apr 4, 2024

emptywee commented Apr 4, 2024

jasonacox commented Apr 5, 2024

jasonacox commented Apr 5, 2024

jasonacox commented Apr 5, 2024

emptywee commented Apr 5, 2024

mcbirse commented Apr 5, 2024

jasonacox commented Apr 6, 2024

emptywee commented Apr 6, 2024

mcbirse commented Apr 7, 2024

emptywee commented Mar 28, 2024 •

edited

Loading

emptywee commented Mar 28, 2024 •

edited

Loading