Exposing more data from the api-calls

[tobbensoft]

During debugging to fix #51 and seeing the available data from the api..

Is it a big thing to expose this in HA without any "processing" just so we could start using it to display information?

That could probably bring more interest and contributors to the project.

[alandtse]

Exposing anything in HA would be difficult because of the review process in pushing code. If you describe what you want to specifically have access to in HA that would be easier. There are a bunch of sites that allow you to browser the known Tesla API. The question is to consider how to represent data in HA.

[tobbensoft]

Listing a few from the top of my mind that could be useful

Last successful api-call (timestamp) in combination with state: online/asleep
charge_limit_soc_std:
charge_state:
car_version:

Managing online/asleep would be a good thing (that I do not know enough about yet)
The services @llluis was mentioning he implemented in #24 is probably in the right direction.

[alandtse]

Thanks. Please think about how they would be displayed in HA. We can easily expose the data you want through the API as it's all collected but the question will be how to properly display it in HA. I have a charging rate sensor PR to try to get more info for charging so I'm not sure if that's what you needed.

[tobbensoft]

@alandtse your PR looks great..

awake/sleep could be represented by a binary_sensor.tesla_awake ? (I could not find any of the device classes the seems to fit awake/sleep)

Is there any guidelines when sensors don't fit devices classes? Like car_version: for instance?

[magnussand]

I just started out using HA and are currently setting up the development environment for it.

I'm also interested in this as well as have the ability to read remaining charge time, charge current and turn on or off individual seat heaters.

I guess things like on/off of each seat heater should be switches, but everything that is read only seems logical to have registered as sensors, for instance, current location, speed, cabin temperature etc.

[tobbensoft]

@alandtse your latest PR works fine in the pre-release!

Wondering why I always have 2Volts charger voltage even when disconnected :)

[alandtse]

@magnussand We did expose a bunch of the charger related stuff. We don't have the seat heaters yet, but do check the component as some of the other temperature stuff and gps stuff is already exposed.

@alandtse Glad to hear. Waiting for it to go live so I can benefit from the work on my production system. ;)

[alandtse]

@tobbensoft Online binary sensor can be easily added by replacing two files in teslajsonpy.
See https://github.com/zabuldon/teslajsonpy/compare/master...alandtse:online_sensor. HA 0.104.0 will automatically populate it once teslajsonpy is updated. Also, no idea why you have 2 volts when disconnected, that's what Tesla is reporting 🤷‍♂.

EDIT: Btw shift_state is basically in the parking brake sensor. That only is On when shift_state is P. I honestly don't know if there's another state like Neutral or Reverse, but for purposes of detecting driving that's probably the right sensor.

[tobbensoft]

Upgraded to HA 0.104.0 and replaced the files.. loads and populates as expected.

Is binary_sensor correct for the online status?
Is there a default naming for the status connected/disconnected

To my knowledge it has at least 3 states. online/asleep/offline

[llluis]

Thank guys for the effort.
To allow a state machine through automations as proposed by @tobbensoft, the online binary sensor shall have a separate behaviour from the rest of the API: it shall poll /api/1/vehicles/{id} at short intervals (like, 60s, configurable maybe). This call never wakes the car and would allow the automations to change poll interval of the full data dynamically.

Problem is: I never succeeded in changing the poll interval in HA during runtime. As far as I saw, it's always 30s but with throttle constraint.

Or we can implement this state machine in teslajsonpy. Would be a lot easier and more manageable, avoiding HA approval cycles as mentionned by @alandtse.

[tobbensoft]

@llluis by the looks of the logs in "teslafi" it seems to be 60s for the /api/1/vehicles/{id} polling.

And I assume you mean 300s :)

[alandtse]

Is binary_sensor correct for the online status?
Is there a default naming for the status connected/disconnected

To my knowledge it has at least 3 states. online/asleep/offline

Yes, a connector switch has those default naming. Templates will respond with On and Off though. Given we aren't exposing an explicit wake command, what value are you getting if you know it's asleep vs offline? For example, what would you do differently? Please remember commands automatically will wake up the car while updates will not.

To allow a state machine through automations as proposed by @tobbensoft, the online binary sensor shall have a separate behaviour from the rest of the API: it shall poll /api/1/vehicles/{id} at short intervals (like, 60s, configurable maybe). This call never wakes the car and would allow the automations to change poll interval of the full data dynamically.

This is the behavior more or less (but using 300 instead of 60). The tesljsonpy controller updates the online state every scan_interval seconds. After the online check, it then checks the Update switch to decide if it'll directly ping the car for any cars that are determined to be online.

So to changes in HA, if you have a scan_interval of 300 (which is currently the hardcoded floor), you could decide to have HA skip updates for a vehicle some period after you detect the parking brake is engaged. For example, assume you'll be in a location for an hour after driving. You then can enable the update switch when you want to start getting updates again (but it has to be engaged for at least scan_interval so it's picked up during one of the controller update cycles).

Problem is: I never succeeded in changing the poll interval in HA during runtime. As far as I saw, it's always 30s but with throttle constraint.

Ignore the HA logs for updates. That 30s polling is coming from HA and will get the cached value. The controller operates at scan_interval to update the cache per my explanation above.

Now there could be a question about letting the scan_interval go even lower by removing the 300 floor. We have always been conservative to avoid getting people locked out of their car or having Tesla decide to lock down their API.

[tobbensoft]

@alandtse will try to catch the offline mode and match it with the sensor state. (only seen it a few times)

Is it possible to get a timestamp when last successful API-call was made? (I like to know that everything is ok and polling when car is asleep)
I can't think of how to use the information present in HA for that right now.

[alandtse]

Is it possible to get a timestamp when last successful API-call was made? (I like to know that everything is ok and polling when car is asleep)

No. That data isn't saved anywhere. You'll only see it in teslajsonpy logs. I'm not aware of any HA property or attribute where we'd save that info as the philosophy is the api is supposed to handle things like that. You can assume it's polling per my description above and you can check the logs.

[llluis]

This is the behavior more or less (but using 300 instead of 60). The tesljsonpy controller updates the online state every scan_interval seconds. After the online check, it then checks the Update switch to decide if it'll directly ping the car for any cars that are determined to be online.

I thought that the online status were polled at the same time as everything else. That means that if the car is sleeping, independent of the online switch, it will detect it became online? Because what we need is to detect the car sucessfully slept to reduce polling time and detect a drive or charge right away. I need to make a few tests.

So to changes in HA, if you have a scan_interval of 300 (which is currently the hardcoded floor), you could decide to have HA skip updates for a vehicle some period after you detect the parking brake is engaged. For example, assume you'll be in a location for an hour after driving. You then can enable the update switch when you want to start getting updates again (but it has to be engaged for at least scan_interval so it's picked up during one of the controller update cycles).

Online status could always be 60s. Only the car requests would be afftected by scan_interval. But I believe we can live with as is.
My idea was to remove the floor and set it to 60s. Then, the only situation it would increase (15 min) is when you want to let the car sleep (parking break engaged for 15min). At 15min interval, car will sleep inbetween. After that you can return to 60s and will catch the wakeup.

Ignore the HA logs for updates. That 30s polling is coming from HA and will get the cached value. The controller operates at scan_interval to update the cache per my explanation above.

Yeah, I forgot the throttle is done at the controller and not in the update method in the HA integration. Good, then we can expose the "scan_interval" to dynamically change it through a service call.

Now there could be a question about letting the scan_interval go even lower by removing the 300 floor. We have always been conservative to avoid getting people locked out of their car or having Tesla decide to lock down their API.

My TeslaFi polls it every 60s, and so far so good, no problem of being kicked out. But it's dynamic. If it detects a situation of "can sleep", it changes to 15 min. Once the car slept, it goes back to 60s to catch the wake up. If driving or charing, it kleeps the 60s.

[llluis]

@llluis by the looks of the logs in "teslafi" it seems to be 60s for the /api/1/vehicles/{id} polling.

@tobbensoft, Yes, Teslafi is always checking online status every 60s.

And I assume you mean 300s :)

HA calls the controller method every 30s. But the controller throttles this request to scan_interval, which is 300s. So the car is only called every 300s.

[alandtse]

I thought that the online status were polled at the same time as everything else. That means that if the car is sleeping, independent of the online switch, it will detect it became online?

Yes, it will detect if online if Tesla says it's online. The Update switch does not do anything to the online check.

Good, then we can expose the "scan_interval" to dynamically change it through a service call.

No need to expose a new service, just change the config flow option. However, no idea if the config_flow options are changeable programmatically and that would be a HA feature request if it isn't. That said, you can just turn off the Update switch for periods you want the car to be sleeping. You don't have to change the scan_interval.

[tobbensoft]

@alandtse I think I got the update/sleeping working as it should by using the update switch and automations. Will test for a little longer. (Maybe write a little FAQ for the docs?)

2 things are not perfect yet though :)

• Detection of the online state is limited to the 300s interval. That could almost be a whole drive to the kindergarten for me.
  As discussed above.. why don't we make the online check 60s and keep the other check configurable at 300s or above? (Teslafi is using this polling rate and works fine) (It seems that the polling limit is per token and not per car)

• With a restart of HA and the car is in sleep state, a lot of data is "unavailable" and screws up the log/graphs.. This is not the integrations fault but if we wakeup the car after a reboot, the data would be correct... Are there any other solutions? (Template seems wrong here I think)
  What are your thoughts here?

[alandtse]

Not opposed to allowing someone to set the online check to 60 but should keep the default. Please note there are people with multiple cars so we can't set the default too low.

There is a forced wake up on restart so assuming the cars respond, it will populate. Of course we can't force cars to respond so templates will need to wait for data to populate. Some templates use a uptime sensor and require at least 5 minutes of HA uptime before they activate.

[tobbensoft]

• Sounds good, how is the online check configurable?

• I have done multiple restarts when car is asleep and never seen that it wakes up or the car is sent any wakeup command. Any thoughts how to check whats wrong?

[alandtse]

• Sounds good, how is the online check configurable?

It's the scan_interval. We need to up the floor, but that can be 60. Your template can then manage how often any vehicles are polled. If we really want to expose to scan_intervals, see if the HA constants has more than one else we'll have to create a new variable which will need to be approved by HA.

• I have done multiple restarts when car is asleep and never seen that it wakes up or the car is sent any wakeup command. Any thoughts how to check whats wrong?

teslajsonpy logs will show you the API calls its making. On restart it will update all data with a forced wakeup and if the car is asleep it will retry the wakeup/data pull 5 times with an exponential delay until it gives up. Look for the initial online check https://owner-api.teslamotors.com/api/1/vehicles and then the json response to see whether the car is online. Next you'll see the individual car pull. https://owner-api.teslamotors.com/api/1/vehicles/XXX/data. The response will tell you if it responded right away or had to try a wakeup routine.

[tobbensoft]

Can I hack some file to change scan_interval right now?

There is a forced wake up on restart so assuming the cars respond, it will populate. Of course we can't force cars to respond so templates will need to wait for data to populate. Some templates use a uptime sensor and require at least 5 minutes of HA uptime before they activate.

What I see in the logs is that the https://owner-api.teslamotors.com/api/1/vehicles call is made, but I can't see any wakeup command initiated? (If i toggle something, everything works as expected)

So the https://owner-api.teslamotors.com/api/1/vehicles/XXX/data is never made.

I can try this again tonight to make sure I'm not missing anything.

[alandtse]

Probably makes sense to message me on discord instead of trying to chat in github. alandtse#5464

Can I hack some file to change scan_interval right now?

1. Change the clamp value here and restart HA.
   https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/tesla/config_flow.py#L104

2. And then update the options in the integration to whatever you want.

So the https://owner-api.teslamotors.com/api/1/vehicles/XXX/data is never made.

Ok, there's a bug in the code.
https://github.com/zabuldon/teslajsonpy/blob/master/teslajsonpy/controller.py#L481

            async with self.__lock[vin]:
                if (
                    online
                    and (  # pylint: disable=too-many-boolean-expressions
                        self.__update.get(vin)
                    )

should be:

            async with self.__lock[vin]:
                if (
                    (online or wake_if_asleep)
                    and (  # pylint: disable=too-many-boolean-expressions
                        self.__update.get(vin)
                    )

Try that and confirm the behavior for me.

[alandtse]

Added some additional fixes in master...alandtse:online_sensor. Will probably push that as a new version over the weekend. If you could let me know if it breaks anything, that would be great.

[tobbensoft]

Installed... looks good so far...

[llluis]

Everything all good so far. I'm experimenting a few other changes and your branch is great.

I've changed the online check to 60s and independent of the main polling:

https://github.com/llluis/teslajsonpy/blob/aeada5cca1c322d029c2a034eacecd8ca5fc36a9/teslajsonpy/controller.py#L472

Seems to be working pretty good. I put some additional logging to verify.
Creating a few automations now to enable and disable the update switch.

Few changes so far: alandtse/teslajsonpy@online_sensor...llluis:master

[alandtse]

Please test this version. #57. Websockets should connect and automatically update various drive states every few seconds if driving is ever detected (during regular scan_interval check). Unfortunately, HA will still only update the device tracker at 30 second intervals as the websocket update is transparent and I didn't figure out how to increase poll rate for the device tracker only.

[llluis]

I merged #57 and added control to switch from 60 to 600 polling along with automations to control it. I believe it will be the way to do it.
Problem with turning off update switch is that until the car sleeps, you will loose everything inbetween. With a 10min polling, you give the car 10 min slices to try to sleep.

Also, I loved the attributes in the controls. I added a few more, awesome!!! Gives all the info without creating additional entities. Even if needed, templates will solve it.

[alandtse]

Closing this since we have the component now to talk about changes there. https://github.com/alandtse/tesla/issues

If we're missing api calls, then we should open a new issue here.