Skip to content

Read csv with naive data, column filters and datetime/timedelta units #521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 26 commits into from
Nov 10, 2022
Merged

Read csv with naive data, column filters and datetime/timedelta units #521

merged 26 commits into from
Nov 10, 2022

Conversation

Flix6x
Copy link
Contributor

@Flix6x Flix6x commented Oct 31, 2022 •
edited
Loading

Closes #507, and introduces additional CSV import features.

Flix6x added 7 commits October 31, 2022 17:05
@Flix6x
Support unit conversion for python datetime and timedelta objects
3f950d3 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Allow to filter by column when reading in beliefs from CSV
57a050a 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Allow to set a timezone for reading in timezone naive data
e0040e7 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Allow throwing out NaN values when reading in beliefs
d1e3e32 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Support datetime unit conversion for aware datetimes with mixed offset
2e2580a 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Raise instead of assume UTC when reading in timezone naive data witho…
d9ddd0a 
…ut a timezone set explicitly

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Bump timely-beliefs dependency for read_csv
7b7fdf8 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added the enhancement New feature or request label Oct 31, 2022
@Flix6x Flix6x self-assigned this Oct 31, 2022
Flix6x added 5 commits October 31, 2022 17:30
@Flix6x
Refactor: flake8
26b2131 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
CLI changelog entry
6ccb2dc 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
changelog entry
e2eeb08 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
mypy
994af8a 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Use sensor id field validation
401bc1d 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added the CLI label Oct 31, 2022
@Flix6x Flix6x marked this pull request as ready for review October 31, 2022 18:33
Flix6x added 3 commits October 31, 2022 19:44
@Flix6x
Querying for a source with a given id no longer requires knowing the …
6e14acf 
…source type

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
make freeze-deps
9b018dc 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
add optional dependency: timely-beliefs[forecast]
b7ad2d6 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@coveralls
Copy link
Collaborator

coveralls commented Oct 31, 2022 •
edited
Loading

Pull Request Test Coverage Report for Build 3439649207

  • 9 of 29 (31.03%) changed or added relevant lines in 3 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.06%) to 64.967%
Changes Missing Coverage Covered Lines Changed/Added Lines %
flexmeasures/data/queries/data_sources.py 1 4 25.0%
flexmeasures/utils/unit_utils.py 3 9 33.33%
flexmeasures/cli/data_add.py 5 16 31.25%
Files with Coverage Reduction New Missed Lines %
flexmeasures/cli/data_add.py 1 31.39%
Totals Coverage Status
Change from base Build 3436864090: -0.06%
Covered Lines: 6570
Relevant Lines: 9491
💛 - Coveralls

@Flix6x Flix6x requested a review from nhoening October 31, 2022 19:31
nhoening
nhoening requested changes Nov 6, 2022
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks useful. As so often, I want some documentation improvements.

@Flix6x
Clarify help
2df7658 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Merge remote-tracking branch 'origin/main' into read-csv-with-naive-d…
c65910f 
…ata-and-filters

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
@Flix6x
Mention data conversion from 'datetime' or 'timedelta' units
d1acfe5 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Allow converting 'datetime' values to a duration other than seconds (…
74751fd 
…since UNIX epoch)

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Refactor and make convert_time_units a private function
d1a8392 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Refactor and add inline comment explaining why we check to_unit for a…
e77803e 
… digit

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
mypy: PEP 484 prohibits implicit Optional
b3f700a 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x requested a review from nhoening November 10, 2022 13:45
nhoening
nhoening requested changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve the recent clarifications, but I feel something has gone wrong when merging?

@Flix6x
Attempt to revert bugs introduced in merge with main
6041f4f 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
black and flake8
35618ee 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
A few more reverts
6e1510a 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Copy link
Contributor Author

Flix6x commented Nov 10, 2022 •
edited
Loading

Definitely something went range with merging. I believe I now have corrected all mistakes introduced.

@Flix6x Flix6x requested a review from nhoening November 10, 2022 18:30
nhoening
nhoening approved these changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, one possibly missing word and an honest question.

@Flix6x
Fix typos
100f05c 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x merged commit b1bd21d into main Nov 10, 2022
@Flix6x Flix6x deleted the read-csv-with-naive-data-and-filters branch November 10, 2022 19:12
@Flix6x Flix6x added this to the 0.12.0 milestone Nov 10, 2022
Flix6x added a commit that referenced this pull request Nov 15, 2022
@Flix6x
Fix for changed default when reading in timezone naive data: we no lo…
fd19381 
…nger assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Nov 15, 2022
@Flix6x
Fix for changed default when reading in timezone naive data: we no lo…
318b555 
…nger assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x mentioned this pull request Nov 15, 2022
Merged
Flix6x added a commit that referenced this pull request Nov 15, 2022
@Flix6x
Fix for changed default when reading in timezone naive data: we no lo…
06568b8 
…nger assume utc, but raise instead (since PR #521).

Fix tutorial timezone (#535)

* fix charging sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Apr 12, 2023
@Flix6x @nhoening
481 extend documentation and toy tutorial with scheduling within the …
81ea6c3 
…context of a building with PV (#534)

Expand toy tutorial and chart functionality (incl. a more sensible use of the color encoding and data source attribute in charts), by:

- Expanding the toy tutorial with creation of solar sensor and solar forecasts from CSV
- Expanding the toy tutorial with a new schedule that takes into account solar
- Allowing nesting of sensors in the `sensors_to_show` attribute, to denote that multiple sensors should be shown together, and dealing with all of its implications (axis labels, use of colors, etc.)
- Visually distinguishing forecasts, schedules and measurements by source type rather than by belief horizon (incl. a simple db migration)
- Adding a CLI command to create a new DataSource (so that forecasts are visually presented as originating from a forecaster)



* fix charging sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* update price sensor name in tutorial (prefer non-capitalized sensor names)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Print out account ID if toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add solar power sensor to toy account, and allow existing toy accounts to be updated by simply running `flexmeasures add toy-account` again

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Support JSON and Callable arguments

Signed-off-by: F.N. Claessen <felix@seita.nl>

* get or create price sensor

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor to keep original order of sensor ids

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI option to pass inflexible device sensors

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand tutorial with scheduling against solar, which limits the available headroom for the battery

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add note about resampling

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Allow nesting sensors_to_show to support layering multiple sensors in a single row within the multi-row asset chart

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Textual changes

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove print statement

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add missing timezone

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix call to make_schedule

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure storage specs, also when make_schedule is called directly (i.e. no job)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Legend shows which asset a sensor belongs to, too, and sensors that share a name don't share a color (note that sensors belonging to the same asset aren't allowed to have the same name)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI command for adding a data source

Signed-off-by: F.N. Claessen <felix@seita.nl>

* DB migration for source types 'forecaster' and 'scheduler'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Attribute toy solar forecasts to a new DataSource of type 'forecaster'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Check common cases of shared sensor types

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add version identifier and missing parenthesis

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black and flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix test (no legend in PositionFieldDef)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove redundant detail encodings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Upgrade timely-beliefs for required feature from SeitaBV/timely-beliefs#122

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge db revisions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Speed up viz by avoiding redundant client-side transformation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump vega-lite and vegaembed

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure full-width bar width

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update CLI scheduling command by adding `for-storage`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Improve introduction to using `flexmeasures add source`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Introduce follow-up solar example

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: rename "scheduling script" to "scheduler" and "forecasting script" to "forecaster"

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify event start domain setting

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Raise instead of warn in case toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify loop over kwargs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge steps

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify when we choose not to show the sensor name

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add return type annotations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add default sensors_to_show attribute to test asset

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add test cases for bad sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Validate sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify code lines

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: move util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Changelog entry plus upgrade instructions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for nested sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve import error

Signed-off-by: F.N. Claessen <felix@seita.nl>

* redundant space

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve circular import

Signed-off-by: F.N. Claessen <felix@seita.nl>

* add a missing output style

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Fix GH Issue #604

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct spacing of header markings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add sensors_to_show attribute to tutorial printout

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct column name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Shift sensor ids in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix capitalization

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update uniplot output (looks like later versions have a more intuitive choice of y-axis ticks)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct data source IDs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* test

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "test"

This reverts commit b393cad.

* Resolve circular import coming to light through mypy

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Generalize util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI command

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI option

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nhoening nhoening nhoening approved these changes

Assignees

@Flix6x Flix6x

Labels
CLI enhancement New feature or request
Projects
None yet
Milestone
0.12.0
Development

Successfully merging this pull request may close these issues.

Import data with local datetimes
3 participants
@Flix6x @coveralls @nhoening