Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meters #1853

Merged
merged 105 commits into from
May 6, 2019
Merged

Meters #1853

Show file tree
Hide file tree
Changes from 98 commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
6130aea
Comment out previously unused Meter, TimeSeries, and GreenButton
Feb 12, 2019
6909041
Meter and MeterReading models created and can be imported via FE - 1s…
Feb 12, 2019
ff22ae7
Pint library used to replace the unit field for Meters. All MeterRead…
Feb 18, 2019
8d60015
Merge branch 'gis-201902-release' into timeseries
Feb 18, 2019
feff1c7
Shift Meter/MeterReading migrations up
Feb 18, 2019
75d5bbf
Monthly meters correctly associated multiple properties with same und…
Feb 19, 2019
457b7c1
Meter/MeterReading clarifications and WIP comments added.
Feb 19, 2019
5159395
Fix meter typo and assignment convention
Feb 20, 2019
0596ca3
Meter is_virtual indicator added to model
Feb 20, 2019
9ef2359
Merge branch 'develop' into timeseries
Feb 20, 2019
6b54b66
Move Meter Parser from util to it's own class
Feb 21, 2019
cb09432
source_unit and conversion_factors during import are saved to MeterRe…
Feb 21, 2019
f73ed5c
1st iteration BE written to return types and units from meters file h…
Feb 25, 2019
c78e6f1
On meter upload, user confirms types and units parsed from column hea…
Feb 25, 2019
5cdc4ef
Meter upload confirmation BE enhanced to have summary of meters to be…
Feb 25, 2019
7f3c198
User confirms counts of pending reading additions prior to meter upload
Feb 25, 2019
5b69ef0
Enhanced expected and actual meter import details displayed for user
Feb 26, 2019
d05e1ef
During Meter import, PM IDs unassociated to properties are noted to user
Feb 26, 2019
7c75871
Install TimescaleDB in TravisCI
adrian-lara Feb 27, 2019
553cf98
Meter uploads are split across celery tasks and updates viewable via …
Feb 28, 2019
23c2d23
Meter import error handling for meter with overlapping (by time) read…
Mar 1, 2019
fb03b66
Inventory detail energy tab and page created
Mar 3, 2019
28b4879
1st it. of meter readings returned from propertyview's property
Mar 4, 2019
ee46a30
Enhancements for property's Energy Page showing meter readings
Mar 5, 2019
191b888
Energy Page shows (placeholder) note when property has no data
Mar 5, 2019
61351c3
Remove extra comma in inventory detail energy controller
Mar 5, 2019
9603a9b
Organizations have display_meter_units preference
Mar 7, 2019
8b70e70
Meter unit display preferences returned from get organization(s) endp…
Mar 7, 2019
e5afcc5
Meter API endpoint exists to return valid meter type and unit combina…
Mar 7, 2019
ed1903c
Org save settings endpoint can receive and save display_meter_units
Mar 7, 2019
3652bca
Users can update display meter unit preferences in the org. settings …
Mar 7, 2019
30252bd
Meter Page presents meters in the units specified by org
Mar 7, 2019
25501c9
Meters org settings test add new endpoint (#1841)
adrian-lara Mar 7, 2019
44721bf
Create Exporter class to hold logic for the energy usage meter view
Mar 8, 2019
3044f23
Monthly meter usage output is possible and endpoint exists
Mar 11, 2019
50d14c4
Energy page interval dropdown allows for the rendering of Exact and M…
Mar 11, 2019
6bf47f9
Energy page can show annual usage
Mar 12, 2019
0929453
Filtering added on Energy page
Mar 12, 2019
c654653
Revamp algorithm to aggregate readings in interval for export
Mar 13, 2019
d8e9ac1
Rename PMMeterParser to be less specificly for Portfolio Manager
Mar 13, 2019
daa5a02
Refactor MetersParser sub-monthly data intake flexibility
Mar 14, 2019
edeb57a
MetersParser can handle raw meters with start_time and duration
Mar 15, 2019
f92d427
GreenButton upload modal created for Energy page that can upload files
Mar 18, 2019
8b935e7
GreenButton parsed_meters_confirmation returns prelim. info about imp…
Mar 18, 2019
eb57631
Meter Usage start_time and end_time for monthly_imports can overlap
Mar 18, 2019
bbdfb13
GreenButtonParser WIP with temp hardcoding type-units
Mar 18, 2019
030ed7f
User presented with incoming readings info for GreenButton upload
Mar 19, 2019
f4343a9
All meter proposed_imports use generic "source_id" in back-end
Mar 19, 2019
1dcc308
Working (unrefined) GreenButton import from energy detail page
Mar 19, 2019
982c134
Python convention - prepend underscore for unused return values
Mar 19, 2019
21b16d8
GreenButtton XML reader can parse units and types
Mar 26, 2019
f2797b3
Energy controller filler_cycle updated from last used to first created
Mar 26, 2019
140d1fa
GreenButton progress bar and import results shown
Mar 26, 2019
4218600
Show 2 decimal places for energy readings
Mar 26, 2019
8e9a2a9
Save more to GreenButton source_id and UI still shows UsagePoint
Mar 27, 2019
a81b586
Refactor thermal conversion caching for MetersParser
Mar 27, 2019
fe93d57
GreenButton duplicate imports caught when batched together
Mar 27, 2019
8cb1b53
Create button to refresh energy page on GB import complete
Mar 27, 2019
424826f
Comments for GreenButton import task creator and tasks methods
Mar 27, 2019
5066fb8
Remove convoluting reverse enum lookup for Meter sources
Mar 27, 2019
2b2183b
Clean up of ImportFile matching_results_data temporarily used during …
Mar 27, 2019
17c7216
Recursion removed from Weighted Job algorithm implementation
Mar 28, 2019
f200123
If meter imports are cancelled at confirmation step, delete imported …
Mar 28, 2019
4664a87
Disassociate cycle from meter importfiles
Mar 28, 2019
83c7dd5
Meter upload confirmation and result tables have dynamic heights
Mar 29, 2019
e47411a
Update translations relating to meters
Mar 31, 2019
8c4075e
Add note on translation.rst and add comments to related scripts
Mar 31, 2019
bf6b1a4
GB upload results bug fix when upload finishes in one check
Apr 1, 2019
53d7e5f
Merge branch 'develop' into meters
Apr 1, 2019
c0ef515
Shift Meters migrations
Apr 1, 2019
140427b
linter - remove django.db.models.Q import
Apr 1, 2019
b5fb5e7
Remove old GreenButton module, it's dependencies, and references to it
Apr 1, 2019
4a8d14f
Leave note about missed meter/energy translations
Apr 1, 2019
cc237a6
GreenButton meter import error cases handled
Apr 2, 2019
95b2859
Property energy page with "No data" minimal styling
Apr 2, 2019
78d7d74
Remove TODO comment re: purposefully invalidating meter unit/type com…
Apr 2, 2019
be26e2e
PM meter import error cases handled
Apr 2, 2019
67b9071
Organizations can specify US or CAN thermal conversions
Apr 3, 2019
6409851
CAN thermal conversion factors can be used for meter imports
Apr 3, 2019
973dd41
Thermal Conversion settings used to return energy page readings
Apr 4, 2019
fc0818b
Meter unit names with parenthesis are able to be parsed
Apr 4, 2019
949c041
Add warning about changing meter thermal conversion assumption
Apr 5, 2019
c7095ca
Energy Page columns allow periods
Apr 5, 2019
00d6518
Typo in get_python_translations
Apr 5, 2019
97714cc
Migration to cleanup unused source_type from columnmapping
Apr 8, 2019
67ad8ce
Update comments for PM and GB meter -related methods
Apr 8, 2019
d6db31e
Remove unsed importfiles captured for dataset of an importfile
Apr 10, 2019
2c8c996
Merge branch 'develop' into meters
nllong Apr 15, 2019
973c0d6
merge migrations
nllong Apr 15, 2019
723e879
merge migrations
nllong Apr 15, 2019
0b041c4
update versions
nllong Apr 15, 2019
aace0fc
changelog
nllong Apr 15, 2019
da2d0f0
Travis/Docker updates
axelstudios Apr 25, 2019
c57a376
Fixed Protractor webdriver error, removed cruft
axelstudios Apr 27, 2019
36bc3b1
Lint fixes
axelstudios Apr 29, 2019
58a1712
Style and grid improvements
axelstudios Apr 29, 2019
a584ef9
Merge branch 'meters-update' into meters
axelstudios Apr 29, 2019
a0671dc
Merge branch 'develop' into meters
axelstudios Apr 29, 2019
3a237d3
Meters timescale osx install docs update (#1864)
adrian-lara Apr 29, 2019
2d004f8
- Fixed a bug where numbers stored as text in Excel could cause a 500…
axelstudios Apr 30, 2019
41a307c
add defaults to migration
nllong May 2, 2019
9ee9b1c
Meters-related clean up items
May 3, 2019
e32df03
Merge branch 'develop' into meters
nllong May 5, 2019
839c3a5
recreate migration after merging up develop
nllong May 5, 2019
ea61ad1
fix flake8
nllong May 6, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"protractor": true,
"qq": true,
"saveAs": true,
"Spinner": true
"Spinner": true,
"Terraformer": true
},
"plugins": ["angular", "lodash", "protractor"],
"rules": {
Expand Down
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,9 @@ htmlcov/
src/
node_modules/

# JetBrainz tends to suggest a venv directory in the project root for virtual environments
# JetBrains tends to suggest a venv directory in the project root for virtual environments
venv/

package-lock.json
dump.rdb
log/*.log
*.log
Expand Down
26 changes: 17 additions & 9 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
sudo: required
dist: trusty
group: deprecated-2017Q4
dist: xenial
cache:
directories:
- node_modules
Expand All @@ -10,37 +9,46 @@ cache:
- ".tox"
language: python
python:
- 3.6
- "3.6"
addons:
postgresql: '9.6'
chrome: stable
postgresql: "11.2"
apt:
packages:
- enchant
- google-chrome-stable
- gdal-bin
- postgresql-9.6-postgis-2.3
services:
- docker
- redis-server
- postgresql
- xvfb
before_install:
- sudo apt-get update
- sudo apt-get remove -y postgresql\*
- sudo apt-get install -y postgresql-11 postgresql-client-11
- sudo apt-get install -y postgis postgresql-11-postgis-2.5 postgresql-11-postgis-2.5-scripts
- sudo cp /etc/postgresql/{9.6,11}/main/pg_hba.conf
- sudo service postgresql restart 11
- sudo rm /usr/local/bin/docker-compose
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
install:
- pip install --upgrade pip
- pip install tox==2.7.0
- psql --version
- node --version
- npm --version
- nvm install stable
before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start &
- sleep 3
- psql -c "DROP DATABASE IF EXISTS seeddb;" -U postgres
- psql -c "DROP DATABASE IF EXISTS test_seeddb;" -U postgres
- mv config/settings/test_local_untracked.py config/settings/local_untracked.py
- sudo add-apt-repository ppa:timescale/timescaledb-ppa -y
- sudo apt-get update -q
- sudo apt-get install -y timescaledb-postgresql-11 timescaledb-tools
- sudo timescaledb-tune -yes
- sudo service postgresql restart
- psql -c "CREATE DATABASE seeddb;" -U postgres
- psql -d seeddb -c "CREATE EXTENSION postgis;" -U postgres
env:
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# SEED Version 2.6.0 - Beta

2.6.0-beta includes support for meters and time series data storage. In order to use this release, you must first install [timescaledb](https://docs.timescale.com/v1.2/getting-started). Note that this is a Beta release and should not be used on production datasets until we have completed testing.


# SEED Version 2.5.1

Date Range: 03/28/19 - 04/15/19
Expand Down
2 changes: 1 addition & 1 deletion Gruntfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ module.exports = function (grunt) {
files: ['seed/static/seed/js/**/*.js', '!seed/static/seed/js/decorators/**/*.js', '!seed/static/seed/js/seed.js'],
options: {
lazy: true,
basePath: "protractorInstrumented"
basePath: 'protractorInstrumented'
}
},
copy: {
Expand Down
2 changes: 1 addition & 1 deletion bin/protractor_start_server.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# starts local server and create test user for protractor tests

echo "updating webdriver"
./node_modules/protractor/bin/webdriver-manager update
./node_modules/.bin/webdriver-manager update --gecko=false
echo "migrating"
./manage.py migrate &> tox_migrate.log
echo "creating default user"
Expand Down
4 changes: 2 additions & 2 deletions deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -106,15 +106,15 @@ else
docker swarm init
fi

echo "Building lasest version of SEED"
echo "Building latest version of SEED"
# explicitly pull images from docker-compose. Note that you will need to keep the
# versions consistent between the compose file and what is below.
docker-compose pull
docker-compose build --pull

echo "Tagging local containers"
docker tag seedplatform/seed:latest 127.0.0.1:5000/seed
docker tag seedplatform/postgres-seed:11.1 127.0.0.1:5000/postgres-seed
docker tag seedplatform/postgres-seed:11.2 127.0.0.1:5000/postgres-seed
docker tag redis:5.0.1 127.0.0.1:5000/redis
docker tag seedplatform/oep:1.2 127.0.0.1:5000/oep

Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
version: '3'
services:
db-postgres:
image: seedplatform/postgres-seed:11.1
image: seedplatform/postgres-seed:11.2
build:
context: ./docker/postgres
environment:
Expand Down
23 changes: 16 additions & 7 deletions docker/postgres/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,15 +1,24 @@
FROM postgres:11.1
FROM postgres:11.2

ENV POSTGIS_MAJOR 2.5
ENV POSTGIS_VERSION 2.5.2+dfsg-1~exp1.pgdg90+1

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION \
postgis=$POSTGIS_VERSION \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && \
apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION \
postgis=$POSTGIS_VERSION \
apt-transport-https ca-certificates wget && \
rm -rf /var/lib/apt/lists/*

RUN sh -c "echo 'deb https://packagecloud.io/timescale/timescaledb/debian/ `lsb_release -c -s` main' > /etc/apt/sources.list.d/timescaledb.list" && \
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add - && \
apt-get update && \
apt-get install -y timescaledb-postgresql-$PG_MAJOR && \
apt-get purge -y --auto-remove apt-transport-https ca-certificates wget && \
rm -rf /var/lib/apt/lists/*

RUN mkdir -p /docker-entrypoint-initdb.d
COPY ./initdb-postgis.sh /docker-entrypoint-initdb.d/postgis.sh
COPY ./initdb-timescaledb.sh /docker-entrypoint-initdb.d/timescaledb.sh
COPY ./update-postgis.sh /usr/local/bin
6 changes: 3 additions & 3 deletions docker/postgres/initdb-postgis.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ EOSQL

# Load PostGIS into both template_database and $POSTGRES_DB
for DB in template_postgis "$POSTGRES_DB"; do
echo "Loading PostGIS extensions into $DB"
"${psql[@]}" --dbname="$DB" <<-'EOSQL'
CREATE EXTENSION IF NOT EXISTS postgis;
echo "Loading PostGIS extensions into $DB"
"${psql[@]}" --dbname="$DB" <<-'EOSQL'
CREATE EXTENSION IF NOT EXISTS postgis;
EOSQL
done
6 changes: 6 additions & 0 deletions docker/postgres/initdb-timescaledb.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/sh

set -e

timescaledb-tune -conf-path /var/lib/postgresql/data/postgresql.conf -quiet -yes
service postgresql restart
5 changes: 4 additions & 1 deletion docs/source/linux.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,14 @@ Install the following base packages to run SEED:

.. code-block:: console

sudo add-apt-repository ppa:timescale/timescaledb-ppa
sudo apt update
sudo apt upgrade
sudo apt install libpq-dev python3-dev python3-pip libatlas-base-dev \
gfortran build-essential nodejs npm libxml2-dev libxslt1-dev git \
libssl-dev libffi-dev curl uwsgi-core uwsgi-plugin-python mercurial
sudo apt install redis-server
sudo apt install postgresql postgresql-contrib
sudo apt install timescaledb-postgresql-10 postgresql-contrib


.. note:: postgresql ``>=9.3`` is required to support `JSON Type`_
Expand All @@ -43,6 +44,8 @@ default use password `seedpass` when prompted

.. code-block:: console

$ sudo timescaledb-tune
$ sudo service postgresql restart
$ sudo su - postgres
$ createuser -P "seeduser"
$ createdb "seeddb" --owner="seeduser"
Expand Down
19 changes: 18 additions & 1 deletion docs/source/migrations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,23 @@ Migrations

Django handles the migration of the database very well; however, there are various changes to SEED that may require some custom (manual) migrations. The migration documenation includes the required changes based on deployment and development for each release.

Version 2.6.0-Beta
------------------

2.6.0-beta includes support for meters and time series data storage. In order to use this release
you must first install [timescaledb](https://docs.timescale.com/v1.2/getting-started).

**Ubuntu Instructions**

.. code-block:: console

sudo add-apt-repository ppa:timescale/timescaledb-ppa
sudo apt update
sudo apt install timescaledb-postgresql-10
sudo timescaledb-tune
sudo service postgresql restart


Version 2.5.1
-------------

Expand All @@ -15,7 +32,7 @@ Docker-based Deployment
^^^^^^^^^^^^^^^^^^^^^^^

- Add the MapQuest API key to your organization.
- On deployment, the error below is indicative that you need to install the extensions in the postgres database. Run `docker exec <posgres_container_id> update-postgis.sh`.
- On deployment, the error below is indicative that you need to install the extensions in the postgres database. Run `docker exec <postgres_container_id> update-postgis.sh`.

django.db.utils.OperationalError: could not open extension control file "/usr/share/postgresql/11/extension/postgis.control": No such file or directory

Expand Down
2 changes: 0 additions & 2 deletions docs/source/modules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ Modules
modules/seed.data
modules/seed.data_importer
modules/seed.features
modules/seed.green_button
modules/seed.landing
modules/seed.lib
modules/seed.mappings
Expand All @@ -22,4 +21,3 @@ Modules
modules/seed.urls
modules/seed.utils
modules/seed.views

29 changes: 0 additions & 29 deletions docs/source/modules/seed.green_button.rst

This file was deleted.

22 changes: 0 additions & 22 deletions docs/source/modules/seed.green_button.tests.rst

This file was deleted.

4 changes: 2 additions & 2 deletions docs/source/translation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ Translating SEED

2. Copy lokalise.cfg.example to lokalise.cfg. Update API token.

3. Run scripts
3. Run scripts if you have Lokalise CLI installed. If not, see scripts for manual steps.

.. code:: bash

scripts/get_python_translations
script/get_python_translations
script/get_angular_translations

4. Verify and commit changes
Expand Down
Binary file modified locale/en_US/LC_MESSAGES/django.mo
Binary file not shown.
Binary file modified locale/fr_CA/LC_MESSAGES/django.mo
Binary file not shown.
Loading