Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tesla-history.py not pulling all data #455

Open
lukemcguire opened this issue Mar 21, 2024 · 16 comments
Open

tesla-history.py not pulling all data #455

lukemcguire opened this issue Mar 21, 2024 · 16 comments
Labels
bug Something isn't working

Comments

@lukemcguire
Copy link

Problem
When running the tesla-history import tool not all of the data is being imported to the database. Specifically, the end of each day is missing data. When running the tool it'll find the data gap but it won't

To Reproduce
Please paste a screenshot or steps to reproduce the behavior:

  1. Find time period with data gap (e.g. 2023-06-26 17:00:00 - 2023-06-26 23:59:59)
  2. Run command 'python3 tesla-history.py --start "2023-06-26 00:00:00" --end "2023-06-27 23:59:59"'
  3. Receive error 'ERROR: No data returned for this date/time range'

Here's the output of the same command with --debug flag - debug.txt

Screenshots
image

Host System

  • Hardware - Synology DS920+
  • OS - DSM 7.2.1-69057 Update 3
  • Powerwall-Dashboard stack is running in Container Manager (Synology's docker tool)

Additional context
I think it may be something to do with time zones since it appears to not pull the last 7 hours of each day and I'm in tz America/Los_Angeles. Looking through the debug output it seems that it finds the gap but when it tries to pull the data it's pulling from the wrong time window.
@jasonacox
Copy link
Owner

Hi @lukemcguire - Does the host your are running have your timezone set correctly? There is code in the script to do the timezone calculation but it does use your localhost settings. It should be something like:

$ timedatectl
                      Local time: Thu 2024-03-21 19:48:56 PDT
                  Universal time: Fri 2024-03-22 02:48:56 UTC
                        RTC time: Fri 2024-03-22 02:48:56
                       Time zone: America/Los_Angeles (PDT, -0700)

@lukemcguire
Copy link
Author

Well, add timedatectl to the list of useful things that Synology neglected to include in their distro. Anyways...

$ date ; date -u
Fri Mar 22 09:26:35 AM PDT 2024
Fri Mar 22 04:26:35 PM UTC 20244

image

So that looks like the system time is ok. Is it possible that one of the containers isn't in the right time zone? I don't know if that's being handled in the script or not.

Looking at the screenshot I provided, the data gap would be from 2023-06-26 17:00-23:59 PDT. However, based on the debug output that I provided, it says * Found data gap: [2023-06-26 00:00:00+00:00] - [2023-06-26 07:00:00+00:00] (7:00:00s). This wouldn't seem to be a timezone related issue as the amount of the time error isn't a factor of 7. Maybe I'm grasping at straws here.

Is there something else I can check?

@lukemcguire
Copy link
Author

I guess I should also note that I've run the weather-history tool and it is working just fine and is not returning gaps.

@jasonacox
Copy link
Owner

These aren't running in containers, just scripts on your localhost. The weather history tool isn't using the Tesla Cloud API so it wouldn't be impacted by the timezone translation issues. One final thought, can you confirm your Tesla App shows the correct timezone selected?

I'll flag this as a bug to look at. If you want to debug the code is here: https://github.com/jasonacox/Powerwall-Dashboard/blob/main/tools/tesla-history/tesla-history.py

@jasonacox jasonacox added the bug Something isn't working label Mar 23, 2024
@lukemcguire
Copy link
Author

lukemcguire commented Mar 23, 2024 via email

@lukemcguire
Copy link
Author

Screenshot attached.

Screenshot_20240323-144727

@mcbirse
Copy link
Collaborator

mcbirse commented Mar 24, 2024

Hi @lukemcguire - there seems to be some issue detecting your local timezone correctly when interpreting the --start and --end times.

The giveaway is below - for the period of history data being retrieved, it should show correctly your timezone offset in the timestamps (i.e -07:00), but it does not.

Running for period: [2023-06-26 00:00:00+00:00] - [2023-06-27 23:59:59+00:00] (1 day, 23:59:59s)

It's possibly more of a system compatibility issue rather than a bug in the script. Synology NAS (and NAS systems in general) seem to be a problem.

There's an easy workaround. You can actually specify full formed timestamps including the offset with the --start and --end arguments.

i.e. try this instead (add your timezone offset after the times):

python3 tesla-history.py --start "2023-06-26 00:00:00-07:00" --end "2023-06-27 23:59:59-07:00"

Hopefully that would work for you. Let us know how it goes.

@lukemcguire
Copy link
Author

Interesting idea, but unfortunately still no joy.

$ python3 tesla-history.py --start "2023-06-26 00:00:00-07:00" --end "2023-06-27 23:59:59-07:00"
----------------------------------------
Tesla account: XXXX
----------------------------------------
      Site ID: XXXX
    Site type: Powerwall x1
    Site name: My Home
     Timezone: America/Los_Angeles
    Installed: 2023-05-15 14:11:56-07:00
  System time: 2024-03-25 10:49:16-07:00
----------------------------------------
Running for period: [2023-06-26 07:00:00+00:00] - [2023-06-28 06:59:59+00:00] (1 day, 23:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2023-06-26 23:55:00+00:00] - [2023-06-27 07:00:00+00:00] (7:05:00s)
* Found data gap: [2023-06-27 23:55:00+00:00] - [2023-06-28 06:59:59+00:00] (7:04:59s)

Searching InfluxDB for data gaps (grid status)
* None found

Retrieving data for gap: [2023-06-26 23:56:00+00:00] - [2023-06-27 06:59:59+00:00] (7:03:59s)
* Loading daily history: [2023-06-26] (America/Los_Angeles)
* Loading daily history: [2023-06-27] (America/Los_Angeles)
Retrieving data for gap: [2023-06-27 23:56:00+00:00] - [2023-06-28 06:59:59+00:00] (7:03:59s)
* Loading daily history: [2023-06-28] (America/Los_Angeles)

ERROR: No data returned for this date/time range

@mcbirse
Copy link
Collaborator

mcbirse commented Mar 26, 2024

What is your TZ value set to in your tesla-history.conf file? Can you post a copy of the config file.

@lukemcguire
Copy link
Author

Here you go. Renamed so that Github allows posting.

tesla-history.conf.txt

@jasonacox
Copy link
Owner

This is so odd. I have a test system that has a lot of gaps. I found a good 5 day gap:

  • Loading daily history: [2024-02-26] (America/Los_Angeles)
    Retrieving data for gap: [2024-02-26 02:14:00-08:00] - [2024-03-02 22:56:59-08:00] (5 days, 20:42:59s)
  • Loading daily history: [2024-02-27] (America/Los_Angeles)

It filled in the full days... here is where it filled in the gap to the recorded data (the weather data shows the gap):

image

@lukemcguire do you have another system that could be used to run the script? You would just edit the conf file to point to the IP address of your Dashboard host (for Influxdb).

@mcbirse
Copy link
Collaborator

mcbirse commented Mar 26, 2024

This is so odd.

Agreed - really odd.

There has to be in issue getting timezones correctly on this system. The start/end period shown makes no sense to me. The times should be shown in the configured InfluxDB timezone, with an offset - like "2023-06-26 00:00:00-07:00" - not +00 as below.

Running for period: [2023-06-26 07:00:00+00:00] - [2023-06-28 06:59:59+00:00] (1 day, 23:59:59s)

Also, it was mentioned the weather-history tool was fine, but that script must show the same problem. It uses the same python modules to get timezones etc.

@lukemcguire - can you run the weather-history script and show the output from the first few lines? Does it output exactly like below when using command python3 weather-history.py --start "2023-06-26 00:00:00" --end "2023-06-27 23:59:59"?

---------------------------------------------------
Weather History Import Tool for Powerwall-Dashboard
---------------------------------------------------
Running for period: [2023-06-26 00:00:00-07:00] - [2023-06-27 23:59:59-07:00] (1 day, 23:59:59s)

Not sure how else to work out where the issue is.... could the host system be missing tzdata?

Perhaps try this please. Run python and paste all of the below commands in the python shell.

from dateutil.parser import isoparse
from dateutil import tz

start = isoparse("2023-06-26 00:00:00")
start
print(start)
type(start)
type(start.utcoffset())

influxtz = tz.gettz("America/Los_Angeles")
influxtz
print(influxtz)
type(influxtz)

start = start.replace(tzinfo=influxtz)
start
print(start)
type(start)
type(start.utcoffset())

utctz = tz.tzutc()
utctz
print(utctz)
type(utctz)

startutc = start.astimezone(utctz)
startutc
print(startutc)
type(startutc)
type(startutc.utcoffset())

startlocal = start.astimezone(influxtz)
startlocal
print(startlocal)
type(startlocal)
type(startlocal.utcoffset())

Then, what does the output show? Something must be off with one of these. i.e. I get below:

$ python3
Python 3.11.4 (main, Jun  9 2023, 02:29:05) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from dateutil.parser import isoparse
>>> from dateutil import tz
>>>
>>> start = isoparse("2023-06-26 00:00:00")
>>> start
datetime.datetime(2023, 6, 26, 0, 0)
>>> print(start)
2023-06-26 00:00:00
>>> type(start)
<class 'datetime.datetime'>
>>> type(start.utcoffset())
<class 'NoneType'>
>>>
>>> influxtz = tz.gettz("America/Los_Angeles")
>>> influxtz
tzfile('US/Pacific')
>>> print(influxtz)
tzfile('US/Pacific')
>>> type(influxtz)
<class 'dateutil.zoneinfo.tzfile'>
>>>
>>> start = start.replace(tzinfo=influxtz)
>>> start
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('US/Pacific'))
>>> print(start)
2023-06-26 00:00:00-07:00
>>> type(start)
<class 'datetime.datetime'>
>>> type(start.utcoffset())
<class 'datetime.timedelta'>
>>>
>>> utctz = tz.tzutc()
>>> utctz
tzutc()
>>> print(utctz)
tzutc()
>>> type(utctz)
<class 'dateutil.tz.tz.tzutc'>
>>>
>>> startutc = start.astimezone(utctz)
>>> startutc
datetime.datetime(2023, 6, 26, 7, 0, tzinfo=tzutc())
>>> print(startutc)
2023-06-26 07:00:00+00:00
>>> type(startutc)
<class 'datetime.datetime'>
>>> type(startutc.utcoffset())
<class 'datetime.timedelta'>
>>>
>>> startlocal = start.astimezone(influxtz)
>>> startlocal
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('US/Pacific'))
>>> print(startlocal)
2023-06-26 00:00:00-07:00
>>> type(startlocal)
<class 'datetime.datetime'>
>>> type(startlocal.utcoffset())
<class 'datetime.timedelta'>

@lukemcguire
Copy link
Author

Huh... good catch @mcbirse. I didn't notice that the timezone call was basically giving +07 instead of -07. Something's clearly cracker jacks.

Weather History Tool

First, let's take a look at the weather-history tool.

$ python3 weather-history.py --start "2023-06-26 00:00:00" --end "2023-06-27 23:59:59"
---------------------------------------------------
Weather History Import Tool for Powerwall-Dashboard
---------------------------------------------------
Running for period: [2023-06-26 00:00:00+00:00] - [2023-06-27 23:59:59+00:00] (1 day, 23:59:59s)

It doesn't appear to be getting the timezone data correctly either so I'm not sure why that's working when the tesla-history tool isn't. I can see that the check_datetime() function appears to be the same in both tools.

Python Shell

Here's the output from the python shell

$ python3
Python 3.9.14 (main, Mar 24 2023, 10:04:27) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dateutil.parser import isoparse
>>> from dateutil import tz
>>> 
>>> start = isoparse("2023-06-26 00:00:00")
>>> start
datetime.datetime(2023, 6, 26, 0, 0)
>>> print(start)
2023-06-26 00:00:00
>>> type(start)
<class 'datetime.datetime'>
>>> type(start.utcoffset())
<class 'NoneType'>
>>> 
>>> influxtz = tz.gettz("America/Los_Angeles")
>>> influxtz
tzfile('/usr/share/zoneinfo/America/Los_Angeles')
>>> print(influxtz)
tzfile('/usr/share/zoneinfo/America/Los_Angeles')
>>> type(influxtz)
<class 'dateutil.tz.tz.tzfile'>
>>> 
>>> start = start.replace(tzinfo=influxtz)
>>> start
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/America/Los_Angeles'))
>>> print(start)
2023-06-26 00:00:00+00:00
>>> type(start)
<class 'datetime.datetime'>
>>> type(start.utcoffset())
<class 'datetime.timedelta'>
>>> 
>>> utctz = tz.tzutc()
>>> utctz
tzutc()
>>> print(utctz)
tzutc()
>>> type(utctz)
<class 'dateutil.tz.tz.tzutc'>
>>> 
>>> startutc = start.astimezone(utctz)
>>> startutc
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzutc())
>>> print(startutc)
2023-06-26 00:00:00+00:00
>>> type(startutc)
<class 'datetime.datetime'>
>>> type(startutc.utcoffset())
<class 'datetime.timedelta'>
>>> 
>>> startlocal = start.astimezone(influxtz)
>>> startlocal
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/America/Los_Angeles'))
>>> print(startlocal)
2023-06-26 00:00:00+00:00
>>> type(startlocal)
<class 'datetime.datetime'>
>>> type(startlocal.utcoffset())
<class 'datetime.timedelta'>

It's clearly doing something different from your installation and it looks like it's breaking around the influxtz command (different class type and what not).

Solution?

However, I was able to get some progress here thanks to the suggestion from @jasonacox .

@lukemcguire do you have another system that could be used to run the script? You would just edit the conf file to point to the IP address of your Dashboard host (for Influxdb).

Yes, I was able to run the script from a laptop and sure enough, it found the data gaps and was able to import the data.

$ python3 tesla-history.py --start "2023-06-26 00:00:00" --end "2023-06-27 23:59:59"
----------------------------------------
Tesla account: XXXX@XXXX.com
----------------------------------------
      Site ID: XXXX
    Site type: Powerwall x1
    Site name: My Home
     Timezone: America/Los_Angeles
    Installed: 2023-05-15 14:11:56-07:00
  System time: 2024-03-26 11:58:12-07:00
----------------------------------------
Running for period: [2023-06-26 00:00:00-07:00] - [2023-06-27 23:59:59-07:00] (1 day, 23:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2023-06-26 16:55:00-07:00] - [2023-06-27 00:00:00-07:00] (7:05:00s)
* Found data gap: [2023-06-27 16:55:00-07:00] - [2023-06-27 23:59:59-07:00] (7:04:59s)

Searching InfluxDB for data gaps (grid status)
* None found

Retrieving data for gap: [2023-06-26 16:56:00-07:00] - [2023-06-26 23:59:59-07:00] (7:03:59s)
* Loading daily history: [2023-06-26] (America/Los_Angeles)
Retrieving data for gap: [2023-06-27 16:56:00-07:00] - [2023-06-27 23:59:59-07:00] (7:03:59s)
* Loading daily history: [2023-06-27] (America/Los_Angeles)

Writing to InfluxDB
Updating InfluxDB
Done.

So, while I can get my data imported, the script still isn't working when run on the NAS. 🤷

@mcbirse
Copy link
Collaborator

mcbirse commented Apr 7, 2024

Thanks for testing and sorry for late reply!

So to me this does not appear to be an bug with the script, but more of an issue with either the host system (Synology NAS) and/or the python dateutil module.

What's the deal with the timezone files on Synology NAS?

python dateutil is interpreting the below as UTC, and not the timezone specified.

>>> start = start.replace(tzinfo=influxtz)
>>> start
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/America/Los_Angeles'))
>>> print(start)
2023-06-26 00:00:00+00:00

Above should show the offset as -07:00 if it was working correctly.

Can you run the below test on your Synology NAS system and see what is returned? This is using the built-in zoneinfo module (available since python 3.9), instead of dateutil.

from dateutil.parser import isoparse
from zoneinfo import ZoneInfo

start = isoparse("2023-06-26 00:00:00")
start
print(start)

influxtz = ZoneInfo("America/Los_Angeles")
influxtz
print(influxtz)
type(influxtz)

start = start.replace(tzinfo=influxtz)
start
print(start)

@mcbirse mcbirse mentioned this issue Apr 7, 2024
Open
@lukemcguire
Copy link
Author

It looks like this might resolve the issue.

$ python3
Python 3.9.14 (main, Mar 24 2023, 10:04:27) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dateutil.parser import isoparse
>>> from zoneinfo import ZoneInfo
>>> 
>>> start = isoparse("2023-06-26 00:00:00")
>>> start
datetime.datetime(2023, 6, 26, 0, 0)
>>> print(start)
2023-06-26 00:00:00
>>> 
>>> influxtz = ZoneInfo("America/Los_Angeles")
>>> influxtz
zoneinfo.ZoneInfo(key='America/Los_Angeles')
>>> print(influxtz)
America/Los_Angeles
>>> type(influxtz)
<class 'zoneinfo.ZoneInfo'>
>>> 
>>> start = start.replace(tzinfo=influxtz)
>>> start
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='America/Los_Angeles'))
>>> print(start)
2023-06-26 00:00:00-07:00

@dJOS1475
Copy link

@lukemcguire I gave up trying to get the history tools working on my DS1621+ and just fired up an Ubuntu Server VM instead. Aside from Ubuntu breaking some of the PIP install instructions, it's been smooth sailing to get everything running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jasonacox @lukemcguire @dJOS1475 @mcbirse