Read csv with naive data, column filters and datetime/timedelta units by Flix6x · Pull Request #521 · FlexMeasures/flexmeasures

Flix6x · 2022-10-31T16:15:22Z

Closes #507, and introduces additional CSV import features.

Signed-off-by: F.N. Claessen <felix@seita.nl>

…ut a timezone set explicitly Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>

…source type Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>

coveralls · 2022-10-31T19:16:58Z

Pull Request Test Coverage Report for Build 3439649207

9 of 29 (31.03%) changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.06%) to 64.967%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
flexmeasures/data/queries/data_sources.py	1	4	25.0%
flexmeasures/utils/unit_utils.py	3	9	33.33%
flexmeasures/cli/data_add.py	5	16	31.25%

Files with Coverage Reduction	New Missed Lines	%
flexmeasures/cli/data_add.py	1	31.39%

Totals
Change from base Build 3436864090:	-0.06%
Covered Lines:	6570
Relevant Lines:	9491

💛 - Coveralls

nhoening

Looks useful. As so often, I want some documentation improvements.

flexmeasures/cli/data_add.py

documentation/cli/change_log.rst

flexmeasures/utils/unit_utils.py

Signed-off-by: F.N. Claessen <felix@seita.nl>

…ata-and-filters # Conflicts: # documentation/changelog.rst # documentation/cli/change_log.rst

Signed-off-by: F.N. Claessen <felix@seita.nl>

…since UNIX epoch) Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>

… digit Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>

nhoening

I approve the recent clarifications, but I feel something has gone wrong when merging?

documentation/changelog.rst

flexmeasures/data/schemas/assets.py

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x · 2022-11-10T18:30:16Z

Definitely something went range with merging. I believe I now have corrected all mistakes introduced.

nhoening

Great, one possibly missing word and an honest question.

documentation/cli/change_log.rst

Signed-off-by: F.N. Claessen <felix@seita.nl>

…nger assume utc, but raise instead (see PR #521) Signed-off-by: F.N. Claessen <felix@seita.nl>

…nger assume utc, but raise instead (since PR #521). Fix tutorial timezone (#535) * fix charging sensor name in tutorial Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521) Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl>

…context of a building with PV (#534) Expand toy tutorial and chart functionality (incl. a more sensible use of the color encoding and data source attribute in charts), by: - Expanding the toy tutorial with creation of solar sensor and solar forecasts from CSV - Expanding the toy tutorial with a new schedule that takes into account solar - Allowing nesting of sensors in the `sensors_to_show` attribute, to denote that multiple sensors should be shown together, and dealing with all of its implications (axis labels, use of colors, etc.) - Visually distinguishing forecasts, schedules and measurements by source type rather than by belief horizon (incl. a simple db migration) - Adding a CLI command to create a new DataSource (so that forecasts are visually presented as originating from a forecaster) * fix charging sensor name in tutorial Signed-off-by: F.N. Claessen <felix@seita.nl> * update price sensor name in tutorial (prefer non-capitalized sensor names) Signed-off-by: F.N. Claessen <felix@seita.nl> * Print out account ID if toy account already exists Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521) Signed-off-by: F.N. Claessen <felix@seita.nl> * Add solar power sensor to toy account, and allow existing toy accounts to be updated by simply running `flexmeasures add toy-account` again Signed-off-by: F.N. Claessen <felix@seita.nl> * Support JSON and Callable arguments Signed-off-by: F.N. Claessen <felix@seita.nl> * get or create price sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor to keep original order of sensor ids Signed-off-by: F.N. Claessen <felix@seita.nl> * Add CLI option to pass inflexible device sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand tutorial with scheduling against solar, which limits the available headroom for the battery Signed-off-by: F.N. Claessen <felix@seita.nl> * Add note about resampling Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Allow nesting sensors_to_show to support layering multiple sensors in a single row within the multi-row asset chart Signed-off-by: F.N. Claessen <felix@seita.nl> * Textual changes Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove print statement Signed-off-by: F.N. Claessen <felix@seita.nl> * Add missing timezone Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix call to make_schedule Signed-off-by: F.N. Claessen <felix@seita.nl> * Ensure storage specs, also when make_schedule is called directly (i.e. no job) Signed-off-by: F.N. Claessen <felix@seita.nl> * Legend shows which asset a sensor belongs to, too, and sensors that share a name don't share a color (note that sensors belonging to the same asset aren't allowed to have the same name) Signed-off-by: F.N. Claessen <felix@seita.nl> * Add CLI command for adding a data source Signed-off-by: F.N. Claessen <felix@seita.nl> * DB migration for source types 'forecaster' and 'scheduler' Signed-off-by: F.N. Claessen <felix@seita.nl> * Attribute toy solar forecasts to a new DataSource of type 'forecaster' Signed-off-by: F.N. Claessen <felix@seita.nl> * Check common cases of shared sensor types Signed-off-by: F.N. Claessen <felix@seita.nl> * Add version identifier and missing parenthesis Signed-off-by: F.N. Claessen <felix@seita.nl> * black and flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test (no legend in PositionFieldDef) Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove redundant detail encodings Signed-off-by: F.N. Claessen <felix@seita.nl> * Upgrade timely-beliefs for required feature from SeitaBV/timely-beliefs#122 Signed-off-by: F.N. Claessen <felix@seita.nl> * Merge db revisions Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Speed up viz by avoiding redundant client-side transformation Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump vega-lite and vegaembed Signed-off-by: F.N. Claessen <felix@seita.nl> * Ensure full-width bar width Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Update CLI scheduling command by adding `for-storage` Signed-off-by: F.N. Claessen <felix@seita.nl> * Improve introduction to using `flexmeasures add source` Signed-off-by: F.N. Claessen <felix@seita.nl> * Introduce follow-up solar example Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: rename "scheduling script" to "scheduler" and "forecasting script" to "forecaster" Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify event start domain setting Signed-off-by: F.N. Claessen <felix@seita.nl> * Raise instead of warn in case toy account already exists Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify loop over kwargs Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Merge steps Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify when we choose not to show the sensor name Signed-off-by: F.N. Claessen <felix@seita.nl> * Add return type annotations Signed-off-by: F.N. Claessen <felix@seita.nl> * Add default sensors_to_show attribute to test asset Signed-off-by: F.N. Claessen <felix@seita.nl> * Add test cases for bad sensors_to_show attribute Signed-off-by: F.N. Claessen <felix@seita.nl> * Validate sensors_to_show attribute Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify code lines Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move util function Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Changelog entry plus upgrade instructions Signed-off-by: F.N. Claessen <felix@seita.nl> * Add documentation for nested sensors_to_show attribute Signed-off-by: F.N. Claessen <felix@seita.nl> * Resolve import error Signed-off-by: F.N. Claessen <felix@seita.nl> * redundant space Signed-off-by: F.N. Claessen <felix@seita.nl> * Resolve circular import Signed-off-by: F.N. Claessen <felix@seita.nl> * add a missing output style Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Fix GH Issue #604 Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct spacing of header markings Signed-off-by: F.N. Claessen <felix@seita.nl> * Add sensors_to_show attribute to tutorial printout Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct sensor name in tutorial Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct column name in tutorial Signed-off-by: F.N. Claessen <felix@seita.nl> * Shift sensor ids in tutorial Signed-off-by: F.N. Claessen <felix@seita.nl> * fix capitalization Signed-off-by: F.N. Claessen <felix@seita.nl> * Update uniplot output (looks like later versions have a more intuitive choice of y-axis ticks) Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct data source IDs Signed-off-by: F.N. Claessen <felix@seita.nl> * test Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "test" This reverts commit b393cad. * Resolve circular import coming to light through mypy Signed-off-by: F.N. Claessen <felix@seita.nl> * Generalize util function Signed-off-by: F.N. Claessen <felix@seita.nl> * Add documentation for new CLI command Signed-off-by: F.N. Claessen <felix@seita.nl> * Add documentation for new CLI option Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Nicolas Höning <nicolas@seita.nl> Co-authored-by: Nicolas Höning <nicolas@seita.nl>

Flix6x added 7 commits October 31, 2022 17:05

Support unit conversion for python datetime and timedelta objects

3f950d3

Signed-off-by: F.N. Claessen <felix@seita.nl>

Allow to filter by column when reading in beliefs from CSV

57a050a

Signed-off-by: F.N. Claessen <felix@seita.nl>

Allow to set a timezone for reading in timezone naive data

e0040e7

Signed-off-by: F.N. Claessen <felix@seita.nl>

Allow throwing out NaN values when reading in beliefs

d1e3e32

Signed-off-by: F.N. Claessen <felix@seita.nl>

Support datetime unit conversion for aware datetimes with mixed offset

2e2580a

Signed-off-by: F.N. Claessen <felix@seita.nl>

Raise instead of assume UTC when reading in timezone naive data witho…

d9ddd0a

…ut a timezone set explicitly Signed-off-by: F.N. Claessen <felix@seita.nl>

Bump timely-beliefs dependency for read_csv

7b7fdf8

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x added the enhancement New feature or request label Oct 31, 2022

Flix6x self-assigned this Oct 31, 2022

Flix6x added 5 commits October 31, 2022 17:30

Refactor: flake8

26b2131

Signed-off-by: F.N. Claessen <felix@seita.nl>

CLI changelog entry

6ccb2dc

Signed-off-by: F.N. Claessen <felix@seita.nl>

changelog entry

e2eeb08

Signed-off-by: F.N. Claessen <felix@seita.nl>

mypy

994af8a

Signed-off-by: F.N. Claessen <felix@seita.nl>

Use sensor id field validation

401bc1d

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x added the CLI label Oct 31, 2022

Flix6x marked this pull request as ready for review October 31, 2022 18:33

Flix6x added 3 commits October 31, 2022 19:44

Querying for a source with a given id no longer requires knowing the …

6e14acf

…source type Signed-off-by: F.N. Claessen <felix@seita.nl>

make freeze-deps

9b018dc

Signed-off-by: F.N. Claessen <felix@seita.nl>

add optional dependency: timely-beliefs[forecast]

b7ad2d6

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x requested a review from nhoening October 31, 2022 19:31

nhoening requested changes Nov 6, 2022

View reviewed changes

flexmeasures/cli/data_add.py Outdated Show resolved Hide resolved

documentation/cli/change_log.rst Outdated Show resolved Hide resolved

flexmeasures/utils/unit_utils.py Outdated Show resolved Hide resolved

Flix6x added 7 commits November 10, 2022 12:56

Clarify help

2df7658

Signed-off-by: F.N. Claessen <felix@seita.nl>

Merge remote-tracking branch 'origin/main' into read-csv-with-naive-d…

c65910f

…ata-and-filters # Conflicts: # documentation/changelog.rst # documentation/cli/change_log.rst

Mention data conversion from 'datetime' or 'timedelta' units

d1acfe5

Signed-off-by: F.N. Claessen <felix@seita.nl>

Allow converting 'datetime' values to a duration other than seconds (…

74751fd

…since UNIX epoch) Signed-off-by: F.N. Claessen <felix@seita.nl>

Refactor and make convert_time_units a private function

d1a8392

Signed-off-by: F.N. Claessen <felix@seita.nl>

Refactor and add inline comment explaining why we check to_unit for a…

e77803e

… digit Signed-off-by: F.N. Claessen <felix@seita.nl>

mypy: PEP 484 prohibits implicit Optional

b3f700a

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x requested a review from nhoening November 10, 2022 13:45

nhoening requested changes Nov 10, 2022

View reviewed changes

documentation/changelog.rst Show resolved Hide resolved

documentation/changelog.rst Show resolved Hide resolved

flexmeasures/data/schemas/assets.py Show resolved Hide resolved

Flix6x added 3 commits November 10, 2022 17:55

Attempt to revert bugs introduced in merge with main

6041f4f

Signed-off-by: F.N. Claessen <felix@seita.nl>

black and flake8

35618ee

Signed-off-by: F.N. Claessen <felix@seita.nl>

A few more reverts

6e1510a

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x requested a review from nhoening November 10, 2022 18:30

nhoening approved these changes Nov 10, 2022

View reviewed changes

documentation/cli/change_log.rst Outdated Show resolved Hide resolved

documentation/cli/change_log.rst Outdated Show resolved Hide resolved

Fix typos

100f05c

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x merged commit b1bd21d into main Nov 10, 2022

Flix6x deleted the read-csv-with-naive-data-and-filters branch November 10, 2022 19:12

Flix6x added this to the 0.12.0 milestone Nov 10, 2022

Flix6x added a commit that referenced this pull request Nov 15, 2022

Fix for changed default when reading in timezone naive data: we no lo…

fd19381

…nger assume utc, but raise instead (see PR #521) Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x added a commit that referenced this pull request Nov 15, 2022

Fix for changed default when reading in timezone naive data: we no lo…

318b555

…nger assume utc, but raise instead (see PR #521) Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x mentioned this pull request Nov 15, 2022

Fix tutorial timezone #535

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read csv with naive data, column filters and datetime/timedelta units#521

Read csv with naive data, column filters and datetime/timedelta units#521
Flix6x merged 26 commits intomainfrom
read-csv-with-naive-data-and-filters

Flix6x commented Oct 31, 2022 •

edited

Loading

Uh oh!

coveralls commented Oct 31, 2022 •

edited

Loading

Uh oh!

nhoening left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nhoening left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Flix6x commented Nov 10, 2022 •

edited

Loading

Uh oh!

nhoening left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

Flix6x commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 3439649207

💛 - Coveralls

Uh oh!

nhoening left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nhoening left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Flix6x commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nhoening left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Flix6x commented Oct 31, 2022 •

edited

Loading

coveralls commented Oct 31, 2022 •

edited

Loading

Flix6x commented Nov 10, 2022 •

edited

Loading