Meters #1853

adrian-lara · 2019-04-01T05:14:08Z

Implementation of Meter uploads using a time-series enhanced database (Postgres + TimescaleDB extension).

Any background context you want to provide?

Notable details/caveats:

Documentation for how Thermal Conversions are brought into app has been shared with NREL via 'NREL-Devetry' shared directory.
As a reminder, it's been decided that this Meters implementation needs to be supplemented with a follow up script and code update to allow PropertyState models to be associated to each other by Property models.
Successfully imported meter files, importfiles, show up in importrecord detail without associated cycles as their data is not bounded by cycles.
An issue exists when/if duplicate times are provided in GreenButton import across different batches. This comes from batch upserting meters in groups of 1000, where duplicate meter entries could exist in different batches. One of the duplicate readings will be chosen according to which batch finishes last (randomly). If duplicates exist with the same batch, the app correctly identifies this as an error and does not upload these entries since it isn't known which entry should take precedence. (This is an edge/corner case that I decided to punt on for now due to time constraints.)
GreenButton types are assumed to be either "Electricity" or "Natural Gas". Corresponding units are parsed to the best of our ability. Only units known in Portfolio Manager imports (and their metric conversions, m, d, k, M, G etc.) are accepted from GreenButton as these are the only ones for which we have thermal conversions.

Assumptions:

A GreenButton meter import contains data for one meter of a single property.
There cannot be 2 different meters of the same type for a single property.

What's this PR do?

Key features include:

the ability to import Portfolio Manager monthly meter data
the ability to import GreenButton (sub-monthly) meter data
the ability to view exact meter data (w/ start and end times) as well as monthly and annually aggregated meter data

How should this be manually tested?

Import PM monthly meters for multiple properties and GB meters for single properties. View the uploaded meters using the new "Energy" tab in the property detail pages.

What are the relevant tickets?

Screenshots (if appropriate)

…t Iteration

…ings saved as kBtu - 1st iteration

…erlying pm_property_ids

…adings

…eaders

…ders

… imported

* TravisCI - install timescaledb * flake8 linter

…progress bar

…ings

…oint

…tions

* Update TimescaleDB OSX install instructions

… on meter data upload - More ui grid tweaks

nllong

hi @adrian-lara can you verify the timestamp imports for GreenButton? It looks like the data are in epoch time, but when importing and converting, it looks like that we are at least 1 hour off.

Raw file (example-GreenButton-data.xml)

<IntervalReading>
          <timePeriod>
            <duration>900</duration>
            <start>1299387600</start>
          </timePeriod>
          <value>1790</value>
        </IntervalReading>
        <IntervalReading>
          <timePeriod>
            <duration>900</duration>
            <start>1299388500</start>
          </timePeriod>
          <value>1791</value>
        </IntervalReading>

Epoch Calc

Seed

nllong · 2019-05-02T19:36:56Z

Can you move the Energy tab to be just to the right of the "Property Detail"?

nllong · 2019-05-02T19:37:33Z

Fix the spacing between the last two items:

nllong · 2019-05-02T19:48:04Z

seed/api/v2/urls.py

-    url(r'meters/(?P<pk>\w+)/timeseries/$',
-        MeterViewSet.as_view({'post': 'add_timeseries'}),
-        name='meters-add-timeseries'),
+    # url(r'meters/(?P<pk>\w+)/timeseries/$',


should we just remove these? Seems like these are old.

nllong · 2019-05-02T19:59:36Z

seed/data_importer/tests/test_greenbutton_import.py

+
+        self.assertEqual(meter_reading_10.start_time, make_aware(datetime(2011, 3, 5, 21, 0, 0), timezone=self.tz_obj))
+        self.assertEqual(meter_reading_10.end_time, make_aware(datetime(2011, 3, 5, 21, 15, 0), timezone=self.tz_obj))
+        self.assertEqual(meter_reading_10.reading, 1790 * 3.412 / 1000 * ureg('kBtu'))


nllong · 2019-05-02T19:59:57Z

seed/data_importer/tests/test_greenbutton_import.py

+        # Sanity check to be sure, nothing was changed with existing meter reading
+        self.assertEqual(meter_reading_12, existing_meter_reading)
+
+    def test_existing_meter_reading_has_reading_source_unit_and_conversion_factor_updated_if_import_file_references_previous_entry(self):


this_is_the_longest_function_name_ive_ever_seen

nllong · 2019-05-02T20:03:12Z

seed/lib/superperms/orgs/models.py

@@ -119,12 +136,17 @@ class Meta:
    created = models.DateTimeField(auto_now_add=True, null=True)
    modified = models.DateTimeField(auto_now=True, null=True)

+    # Default preferred all meter units to kBtu
+    display_meter_units = JSONField(default=_default_display_meter_units)


nllong · 2019-05-02T20:04:26Z

seed/migrations/0102_auto_20190415_1115.py

+            name='meter',
+        ),
+        migrations.RemoveField(
+            model_name='meter',


thanks for cleaning these up.

nllong · 2019-05-02T20:06:07Z

This interface is a bit strange... but i think we leave it for now. It is quite cool that users can specify the way to want to view meter data by units.

nllong · 2019-05-02T20:08:04Z

Fix the spacing between the last two items:

Per @RDmitchell -- And how about putting all the ESPM choices together? Which would mean the ESPM meter option would go right below the Upload Portfolio Manager Data button

adrian-lara · 2019-05-02T23:03:55Z

hi @adrian-lara can you verify the timestamp imports for GreenButton? It looks like the data are in epoch time, but when importing and converting, it looks like that we are at least 1 hour off.

Raw file (example-GreenButton-data.xml)
<IntervalReading>
          <timePeriod>
            <duration>900</duration>
            <start>1299387600</start>
          </timePeriod>
          <value>1790</value>
        </IntervalReading>
        <IntervalReading>
          <timePeriod>
            <duration>900</duration>
            <start>1299388500</start>
          </timePeriod>
          <value>1791</value>
        </IntervalReading>
Epoch Calc

Seed

@nllong, I specified GB epoch times to be read and saved using the default timezone of the app America/Los_Angeles specified here:

seed/config/settings/common.py

Line 34 in 41a307c

TIME_ZONE = 'America/Los_Angeles'

I used the same epoch converter site but changed the timezone (https://www.epochconverter.com/timezones?q=1299387600&tz=America%2FLos_Angeles) and got the time shown in your SEED screenshot.

nllong · 2019-05-03T00:47:01Z

Oh. Good. I'll double check that makes sense!

…

On Thu, May 2, 2019 at 17:03 Adrian Lara ***@***.***> wrote: hi @adrian-lara <https://github.com/adrian-lara> can you verify the timestamp imports for GreenButton? It looks like the data are in epoch time, but when importing and converting, it looks like that we are at least 1 hour off. Raw file (example-GreenButton-data.xml) <IntervalReading> <timePeriod> <duration>900</duration> <start>1299387600</start> </timePeriod> <value>1790</value> </IntervalReading> <IntervalReading> <timePeriod> <duration>900</duration> <start>1299388500</start> </timePeriod> <value>1791</value> </IntervalReading> Epoch Calc [image: image] <https://user-images.githubusercontent.com/1907354/57101734-1063e600-6cdf-11e9-9602-ffb17f95834f.png> Seed [image: image] <https://user-images.githubusercontent.com/1907354/57101606-d4308580-6cde-11e9-9420-68d4eaa43dd8.png> @nllong <https://github.com/nllong>, I specified GB epoch times to be read and saved using the default timezone of the app America/Los_Angeles specified here: https://github.com/SEED-platform/seed/blob/41a307c78d31929d421e728fa814b4385f2e4d4a/config/settings/common.py#L34 I used the same epoch converter site but changed the timezone ( https://www.epochconverter.com/timezones?q=1299387600&tz=America%2FLos_Angeles) and got the time shown in your SEED screenshot. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1853 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAORVGQ35OADSGNOUW3IFMDPTNXNZANCNFSM4HCTPG6Q> .

RDmitchell · 2019-05-03T01:12:23Z

Do we have a good source for greenbutton data? When we were implementing this before with CMU I generated greenbutton files from my house, which I can do again if that would be useful.

adrian-lara · 2019-05-03T16:04:59Z

The samples I used came from here: https://www.energy.gov/downloads/green-button-sample-data-pge

Do we have a good source for greenbutton data? When we were implementing this before with CMU I generated greenbutton files from my house, which I can do again if that would be useful.

nllong · 2019-05-06T19:19:49Z

Fix the spacing between the last two items:

Looks good. Thanks for the fixes.

nllong · 2019-05-06T19:26:57Z

I think this is good to go. Any last comments before I merge?

nllong · 2019-05-06T19:27:16Z

Can you move the Energy tab to be just to the right of the "Property Detail"?

bingo.

Adrian Lara and others added 30 commits February 12, 2019 10:30

Comment out previously unused Meter, TimeSeries, and GreenButton

6130aea

Meter and MeterReading models created and can be imported via FE - 1s…

6909041

…t Iteration

Pint library used to replace the unit field for Meters. All MeterRead…

ff22ae7

…ings saved as kBtu - 1st iteration

Merge branch 'gis-201902-release' into timeseries

8d60015

Shift Meter/MeterReading migrations up

feff1c7

Monthly meters correctly associated multiple properties with same und…

75d5bbf

…erlying pm_property_ids

Meter/MeterReading clarifications and WIP comments added.

457b7c1

Fix meter typo and assignment convention

5159395

Meter is_virtual indicator added to model

0596ca3

Merge branch 'develop' into timeseries

9ef2359

Move Meter Parser from util to it's own class

6b54b66

source_unit and conversion_factors during import are saved to MeterRe…

cb09432

…adings

1st iteration BE written to return types and units from meters file h…

f73ed5c

…eaders

On meter upload, user confirms types and units parsed from column hea…

c78e6f1

…ders

Meter upload confirmation BE enhanced to have summary of meters to be…

5cdc4ef

… imported

User confirms counts of pending reading additions prior to meter upload

7f3c198

Enhanced expected and actual meter import details displayed for user

5b69ef0

During Meter import, PM IDs unassociated to properties are noted to user

d05e1ef

Install TimescaleDB in TravisCI

7c75871

* TravisCI - install timescaledb * flake8 linter

Meter uploads are split across celery tasks and updates viewable via …

553cf98

…progress bar

Meter import error handling for meter with overlapping (by time) read…

23c2d23

…ings

Inventory detail energy tab and page created

fb03b66

1st it. of meter readings returned from propertyview's property

28b4879

Enhancements for property's Energy Page showing meter readings

ee46a30

Energy Page shows (placeholder) note when property has no data

191b888

Remove extra comma in inventory detail energy controller

61351c3

Organizations have display_meter_units preference

9603a9b

Meter unit display preferences returned from get organization(s) endp…

8b70e70

…oint

Meter API endpoint exists to return valid meter type and unit combina…

e5afcc5

…tions

Org save settings endpoint can receive and save display_meter_units

ed1903c

axelstudios approved these changes Apr 29, 2019

View reviewed changes

adrian-lara and others added 2 commits April 29, 2019 16:57

Meters timescale osx install docs update (#1864)

3a237d3

* Update TimescaleDB OSX install instructions

- Fixed a bug where numbers stored as text in Excel could cause a 500…

2d004f8

… on meter data upload - More ui grid tweaks

nllong reviewed May 2, 2019

View reviewed changes

SEED-platform deleted a comment from RDmitchell May 2, 2019

add defaults to migration

41a307c

Meters-related clean up items

9ee9b1c

nllong added 3 commits May 5, 2019 08:27

Merge branch 'develop' into meters

e32df03

recreate migration after merging up develop

839c3a5

fix flake8

ea61ad1

nllong merged commit af71f64 into develop May 6, 2019

nllong deleted the meters branch May 6, 2019 19:51

nllong removed the DO NOT MERGE label Sep 19, 2019

nllong mentioned this pull request Sep 19, 2019

SEED.x.GreenButton #57

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meters #1853

Meters #1853

adrian-lara commented Apr 1, 2019 •

edited

nllong left a comment •

edited

nllong commented May 2, 2019

nllong commented May 2, 2019

nllong May 2, 2019

nllong May 2, 2019

nllong May 2, 2019

nllong May 2, 2019

nllong May 2, 2019

nllong commented May 2, 2019

nllong commented May 2, 2019

adrian-lara commented May 2, 2019

nllong commented May 3, 2019 via email

RDmitchell commented May 3, 2019

adrian-lara commented May 3, 2019

nllong commented May 6, 2019 •

edited

nllong commented May 6, 2019

nllong commented May 6, 2019

Meters #1853

Meters #1853

Conversation

adrian-lara commented Apr 1, 2019 • edited

Any background context you want to provide?

What's this PR do?

How should this be manually tested?

What are the relevant tickets?

Screenshots (if appropriate)

nllong left a comment • edited

Choose a reason for hiding this comment

nllong commented May 2, 2019

nllong commented May 2, 2019

nllong May 2, 2019

Choose a reason for hiding this comment

nllong May 2, 2019

Choose a reason for hiding this comment

nllong May 2, 2019

Choose a reason for hiding this comment

nllong May 2, 2019

Choose a reason for hiding this comment

nllong May 2, 2019

Choose a reason for hiding this comment

nllong commented May 2, 2019

nllong commented May 2, 2019

adrian-lara commented May 2, 2019

nllong commented May 3, 2019 via email

RDmitchell commented May 3, 2019

adrian-lara commented May 3, 2019

nllong commented May 6, 2019 • edited

nllong commented May 6, 2019

nllong commented May 6, 2019

adrian-lara commented Apr 1, 2019 •

edited

nllong left a comment •

edited

nllong commented May 6, 2019 •

edited