Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CSV time series source with window buffer #420

Open
ckittl opened this issue Sep 2, 2021 · 1 comment · May be fixed by #421
Open

CSV time series source with window buffer #420

ckittl opened this issue Sep 2, 2021 · 1 comment · May be fixed by #421
Assignees
@ckittl
Labels
enhancement New feature or request
Projects
TaskBoard

Comments

@ckittl
Copy link
Member

ckittl commented Sep 2, 2021

The current implementation of csv time series source loads the full time series at initialization. This can be very resource consuming, if the time series is long or you'd like to load a lot of time series at the same time.

The targeted implementation shall use a window buffer with configurable look ahead interval. This means, only chunks of the time series are read in and held in storage at the same time. However, this has the drawback, that this source only supports linear querying resp. no possibility to jump back in time.
@ckittl ckittl added the enhancement New feature or request label Sep 2, 2021
@ckittl ckittl self-assigned this Sep 2, 2021
@ckittl ckittl added this to To do in TaskBoard via automation Sep 2, 2021
@ckittl ckittl linked a pull request Sep 2, 2021 that will close this issue
Open
@johanneshiry
Copy link
Member

Regarding your PR I have a question: does this require linear time discrete equidistant time series? What happens with non-equidistant time series? Do we support them atm? Do we support them after your change?

Don't get me wrong I'm fine with both, but maybe it makes sense to have a time series requirement documentation in the docs that state the assumptions and requirements of time series processing.

In any way I think time series are the best argument for a database or similar that allows range queries ... 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ckittl ckittl

Labels
enhancement New feature or request
Projects
No open projects
TaskBoard
  
To do
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Buffered, on-demand reading of time series ie3-institute/PowerSystemDataModel
2 participants
@johanneshiry @ckittl