Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Cache metric ID and forward the metric info to series writer #1062

Merged
merged 1 commit into from Jan 31, 2022

Conversation

antekresic
Copy link
Contributor

Description

This is an optimization to remove the need to fetch metric ID and
table name when storing new series data since we already fetch
this info in a prior step of the process.

Merge requirements

Please take into account the following non-code changes that you may need to make with your PR:

  • CHANGELOG entry for user-facing changes
  • Updated the relevant documentation

@antekresic antekresic added the Performance Improvements that are specifically related to performance label Jan 17, 2022
@antekresic antekresic self-assigned this Jan 17, 2022
@antekresic antekresic removed the request for review from niksajakovljevic January 17, 2022 13:31
Copy link
Contributor

@JamesGuthrie JamesGuthrie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no CHANGELOG entry, this change might be worth mentioning if it has an impact.

pkg/migrations/sql/idempotent/base.sql Outdated Show resolved Hide resolved
pkg/pgmodel/ingestor/metric_batcher.go Outdated Show resolved Hide resolved
pkg/pgmodel/ingestor/metric_batcher.go Outdated Show resolved Hide resolved
@antekresic antekresic force-pushed the optimize-write-series branch 2 times, most recently from 39fca2c to 197d681 Compare January 20, 2022 13:37
@antekresic
Copy link
Contributor Author

@JamesGuthrie I've added an entry to CHANGELOG and update the benchmarks. I've left the old function in and added a new benchmark so we can compare the improvement.

This is what I get on my machine:

$ go test -bench=BenchmarkGetSeriesIDForKeyValueArrayNewSeriesNewLabelsBatch -run=XXX -log-level=error -benchtime=10000x
...
BenchmarkGetSeriesIDForKeyValueArrayNewSeriesNewLabelsBatch-12
Labels DB time 722.503306ms per item: 72.214µs N-labels 10005
DB time 4.892545455s per item: 489.254µs N 10000
   10000            563624 ns/op
...
BenchmarkGetSeriesIDForKeyValueArrayNewSeriesNewLabelsBatchOptimized-12
Labels DB time 729.144623ms per item: 72.878µs N-labels 10005
DB time 2.778612362s per item: 277.861µs N 10000
   10000            352828 ns/op
PASS

I can always remove the old if we want to do it, I'll let Mat take a look and advise there.

Copy link
Contributor

@cevian cevian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very nice improvement. I think it's good to run the benchmark before and after the change and record diff in PR (as you have above). But I don't think we should keep the old functions around just for that purpose. Also I think that in a bunch of places (maybe all places) where you now pass around the metricID and tableName we should pass around a model.MetricInfo instead. That will make the code more future proof.

pkg/migrations/sql/idempotent/base.sql Outdated Show resolved Hide resolved
pkg/migrations/sql/idempotent/base.sql Outdated Show resolved Hide resolved
pkg/pgmodel/ingestor/metric_batcher.go Outdated Show resolved Hide resolved
pkg/pgmodel/ingestor/copier.go Outdated Show resolved Hide resolved
@antekresic antekresic force-pushed the optimize-write-series branch 2 times, most recently from 6827b90 to 3c47756 Compare January 25, 2022 10:41
@antekresic antekresic force-pushed the optimize-write-series branch 6 times, most recently from a392766 to 7d57fef Compare January 27, 2022 10:19
This is an optimization to remove the need to fetch metric ID and
table name when storing new series data since we already fetch
this info in a prior step of the process.
@antekresic antekresic merged commit fd0bffb into timescale:master Jan 31, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Performance Improvements that are specifically related to performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants