-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a resampling actor #32
Add a resampling actor #32
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the ring buffer implementation is way too complicated and way too specific. I was suggesting putting it in the core
package too so it is something that can be reused. We even have a trivial ad-hoc ring buffer implementation in the channels repo (receivers in Broadcast
use a ring buffer for example).
Also the Sample
doesn't need to be parametrized, we agreed on using a float
always to keep thing simple.
Nitpick: I wouldn't name core.common
common. Maybe core.timeseries
as all in there seems to be related to timeseries?
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
09ab0f1
to
5a9c3e1
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
src/frequenz/sdk/data_ingestion/resampling/individual_timeseries_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/individual_timeseries_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/individual_timeseries_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/individual_timeseries_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/individual_timeseries_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
This comment was marked as outdated.
This comment was marked as outdated.
a5add3f
to
95b0870
Compare
src/frequenz/sdk/data_ingestion/resampling/timeseries_group_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
81c5a71
to
c5febe7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, just one follow-up comment and we have the subscription stuff pending, which now that the data sourcing actor is about to be merged we can change with more confidence.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more thing that is still missing from previous reviews.
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
OK, so the data sourcing actor is almost ready, so we have much more clarity on how the resampler needs to subscribe to data. Key changes needed in this PR:
Please see the data sourcing actor PR for more details, and rebase on top of it if needed: Optional unrelated change:
|
295b121
to
f9501b4
Compare
src/frequenz/sdk/data_ingestion/resampling/component_metrics_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metrics_resampling_actor.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/timeseries_resampling_actor.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few final minor comments.
src/frequenz/sdk/data_ingestion/resampling/component_metric_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metric_group_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metric_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metric_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metrics_resampling_actor.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
f9501b4
to
0d5e1b0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just 2 small suggestions, but we are good to go! 🎉
src/frequenz/sdk/data_ingestion/resampling/component_metric_group_resampler.py
Outdated
Show resolved
Hide resolved
src/frequenz/sdk/data_ingestion/resampling/component_metrics_resampling_actor.py
Outdated
Show resolved
Hide resolved
This class enables creating/removing resamplers for individual metrics and delegates resampling to them. Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
0d5e1b0
to
adf3cd7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated the PR title so it gets a better commit message when merged.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@leandro-lucarella-frequenz I added some comments, maybe we can discuss tomorrow.
Summary of this PR (25.10.2022):
Last updates:
DataProvider
core
package entirelySample
class and use the one in data pipelinesubscriptions.py
entirelyTimeSeriesResamplingActor
toComponentMetricsResamplingActor
ComponentMetricsResamplingActor
to:ComponentMetricRequest
class as the subscription request messageUpdates in the N -1 round:
TimeSeriesResamplingActor
TimeSeriesResamplingActor
with a dummy data source (because the real one is still under construction)NaN
valueUpdates in the N - 2 round:
RingBuffer
and using a simpledeque
in theTimeSeriesResampler
TimeSeriesResampler
's bufferIndividualTimeSeriesResampler
->TimeSeriesResampler
andTimeSeriesResampler
toGroupTimeSeriesResampler