Use go modules and implement additional remote write flags. #8

krasi-georgiev · 2019-07-04T11:19:17Z

it send the requests for each series batch in parallel

For each sample it makes totalSeries/seriesPerRequest and it doesn't continue to the next sample until the current series are sent.

Using --remote-send-batch one can configure how many requests are sent in parallel.

For example if you want to send 10 requests in parallel the whole example would would look like:

avalance --remote-url=$URL
--metric-count=100
--series-count=100
--label-count=10
--remote-samples-count=1000
--remote-send-batch=1000

Also added continuous or a single pprof

For reference here is a more complete readme with examples:
https://github.com/krasi-georgiev/benchTSDB
Maybe I can even add these examples from my repo here.

Signed-off-by: Krasi Georgiev 8903888+krasi-georgiev@users.noreply.github.com

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

Gourav2906 · 2019-07-11T10:12:32Z

@krasi-georgiev are we planning to merge this ?

krasi-georgiev · 2019-07-11T10:27:36Z

That is the idea, just waiting on some feedback from the maintainers.

Would that change be useful to you as well?

Gourav2906 · 2019-07-12T04:00:53Z

yes, I am really hoping to use -remote-url asap

Gourav2906 · 2019-07-12T04:04:51Z

@csmarchbanks

csmarchbanks · 2019-07-12T04:06:39Z

I was hoping to get to this already but this week has gotten very busy :( Hopefully I will get to it this weekend!

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev · 2019-07-12T08:45:21Z

metrics/serve.go

 		}
 	}()

 	go func() {
 		for tick := range seriesTick.C {
 			fmt.Printf("%v: refreshing series cycle\n", tick)
+			metricsMux.Lock()


moved the lock here because otherwise there is a race for seriesCycle++ causing cycleValues to misbehave

csmarchbanks

Did a first pass, I with some comments, I will plan to do another pass/some testing when I get a chance. A couple general comments:

It would be nice to break some of the pprof code into separate functions so that the flow of the main code reads better. Need to think a bit on the best way to do this.
I like that the requests are sent in parallel for each batch, thanks for making sure of that.
What do you think of keeping vendor/ or not?

cmd/avalanche.go

csmarchbanks · 2019-07-13T15:48:37Z

go.mod

+	github.com/prometheus/client_golang v1.0.0
+	github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90
+	github.com/prometheus/common v0.4.1
+	github.com/prometheus/prometheus v0.0.0-20190302143042-82f98c825a14


I think it would be better to pin to a Prometheus release.

wasted few hours trying to ping it to v2.11.1 but go mod tidy keeps reverting it to the untagged version although the commit id is the same as v2.11.1

Well, that's silly. Thanks for trying.

csmarchbanks · 2019-07-13T15:51:44Z

metrics/write.go

+		merr            = &errors.MultiError{}
+	)
+
+	fmt.Printf("Sending:  %v timeseries, %v samples, %v timeseries per request, %v delay between requests\n", len(tss), c.config.SamplesCount, c.config.BatchSize, c.config.RequestInterval)


I would vote to consistently use log rather than fmt.

csmarchbanks · 2019-07-13T15:57:37Z

metrics/write.go

+					end = len(tss)
+				}
+				req := &prompb.WriteRequest{
+					Timeseries: tss[i:end],


Are the timestamps of the samples being updated anywhere? If not they should be updated for each send otherwise systems like Cortex will reject the samples as duplicates.

the timestamp is set here
https://github.com/open-fresh/avalanche/pull/8/files#diff-a3417332ad87f5118a22dec21e5038b5L99
So this PR doesn't change this behavior.

this means that when sending the batches the timestamp would be the same for concurrent requests, but these are samples for different timeseries anyway.

In other words something like:

batch1 request1 series1 timestamp1 request2 series2 timestamp1 batch2 request1 series1 timestamp2 request2 series2 timestamp2

would that be a problem?

the only difference is that now requests are sent in parallel, where before they were sent in serial.

oooh , no actually I just noticed that it doesn't update the timestamps, will think how to fix it now.

updated, the workflow is now as per my first comment
let me know if this would work.

👍 I think this looks good now!

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev · 2019-07-15T10:43:09Z

It would be nice to break some of the pprof code into separate functions so that the flow of the main code reads better. Need to think a bit on the best way to do this.

Yes I think the same and I tried few things as well, but each one hit a blocker so decided to keep it as is and will think how to improve later when/if needed.

What do you think of keeping vendor/ or not?

I don't like that it extends the time required to build the docker image significantly, but I thought that vendor-ed dirs would eventually need to go, but I don't feel strongly about it.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev · 2019-07-15T17:18:38Z

I just had an idea that we can mount the host mod cach dir and this way building the docker image would not need to download anything

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev · 2019-07-19T10:00:04Z

I improved the Dockerfile and now it caches the dependencies in a separate step so that it re-downloads everything only when go.mod/do.sum changes.

Relevant discussions in microsoft/vscode-go#1945

csmarchbanks

I like the dockerfile update, that will be nice. Left a couple of last comments, then I think this will be good to go.

cmd/avalanche.go

csmarchbanks · 2019-07-20T15:47:53Z

metrics/write.go

+					end = len(tss)
+				}
+				req := &prompb.WriteRequest{
+					Timeseries: tss[i:end],


👍 I think this looks good now!

metrics/write.go

cmd/avalanche.go

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

csmarchbanks

Thanks!

krasi-georgiev · 2019-07-22T23:31:37Z

Is there a docker image for this and can you update it with this change?

csmarchbanks · 2019-07-22T23:36:16Z

quay.io/freshtracks.io/avalanche should have been updated when I merged this. Latest tag, or the tag with the appropriate commit sha suffix will work.

Let me know if it doesn’t and I will fix the CI!

bboreham · 2019-08-24T09:09:52Z

I think the readme should mention the profiling capability.

…us-community#8) * Use go modules and implement remote write samples count flag. Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * concurent send Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * added continious profiling. Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * fix dockerfile Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * fix remotePprof wait Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * fix the series update lock race Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * total samples count Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * nits Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * update ts for each request Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * improve the cache for docker build Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com> * nits Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev force-pushed the remote-write-samples-count branch from 9e12867 to 8ad614d Compare July 5, 2019 13:25

Use go modules and implement remote write samples count flag.

c4d8749

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev force-pushed the remote-write-samples-count branch from 8ad614d to c4d8749 Compare July 5, 2019 13:42

concurent send

d7bfc6d

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev force-pushed the remote-write-samples-count branch 4 times, most recently from 3743ce5 to 96849d6 Compare July 9, 2019 00:11

krasi-georgiev changed the title ~~Use go modules and implement remote write samples count flag.~~ Use go modules and implement additional remote write flags. Jul 9, 2019

added continious profiling.

a92ed6c

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev force-pushed the remote-write-samples-count branch from 96849d6 to a92ed6c Compare July 9, 2019 14:58

fix dockerfile

457fcf8

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev added 3 commits July 12, 2019 11:17

fix remotePprof wait

3f06ace

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

fix the series update lock race

d28a7ac

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

total samples count

f77a694

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev commented Jul 12, 2019

View reviewed changes

csmarchbanks reviewed Jul 13, 2019

View reviewed changes

nits

dd3713c

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

krasi-georgiev force-pushed the remote-write-samples-count branch from 8c4c604 to dd3713c Compare July 15, 2019 10:28

update ts for each request

314052a

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

improve the cache for docker build

f04afd8

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

csmarchbanks reviewed Jul 20, 2019

View reviewed changes

nits

3c05ce9

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

csmarchbanks approved these changes Jul 22, 2019

View reviewed changes

csmarchbanks merged commit b960b24 into prometheus-community:master Jul 22, 2019

csmarchbanks mentioned this pull request Jul 22, 2019

race condition in code causes creation of more timeseries than requested #6

Closed

krasi-georgiev deleted the remote-write-samples-count branch July 22, 2019 23:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use go modules and implement additional remote write flags. #8

Use go modules and implement additional remote write flags. #8

krasi-georgiev commented Jul 4, 2019 •

edited

Loading

Gourav2906 commented Jul 11, 2019

krasi-georgiev commented Jul 11, 2019 •

edited

Loading

Gourav2906 commented Jul 12, 2019

Gourav2906 commented Jul 12, 2019

csmarchbanks commented Jul 12, 2019

krasi-georgiev Jul 12, 2019

csmarchbanks left a comment

csmarchbanks Jul 13, 2019

krasi-georgiev Jul 15, 2019

csmarchbanks Jul 19, 2019

csmarchbanks Jul 13, 2019

krasi-georgiev Jul 15, 2019

csmarchbanks Jul 13, 2019

krasi-georgiev Jul 15, 2019

krasi-georgiev Jul 15, 2019

krasi-georgiev Jul 15, 2019

csmarchbanks Jul 20, 2019

krasi-georgiev commented Jul 15, 2019

krasi-georgiev commented Jul 15, 2019

krasi-georgiev commented Jul 19, 2019 •

edited

Loading

csmarchbanks left a comment

csmarchbanks Jul 20, 2019

csmarchbanks left a comment

krasi-georgiev commented Jul 22, 2019

csmarchbanks commented Jul 22, 2019

bboreham commented Aug 24, 2019

Use go modules and implement additional remote write flags. #8

Use go modules and implement additional remote write flags. #8

Conversation

krasi-georgiev commented Jul 4, 2019 • edited Loading

Gourav2906 commented Jul 11, 2019

krasi-georgiev commented Jul 11, 2019 • edited Loading

Gourav2906 commented Jul 12, 2019

Gourav2906 commented Jul 12, 2019

csmarchbanks commented Jul 12, 2019

Choose a reason for hiding this comment

csmarchbanks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krasi-georgiev commented Jul 15, 2019

krasi-georgiev commented Jul 15, 2019

krasi-georgiev commented Jul 19, 2019 • edited Loading

csmarchbanks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csmarchbanks left a comment

Choose a reason for hiding this comment

krasi-georgiev commented Jul 22, 2019

csmarchbanks commented Jul 22, 2019

bboreham commented Aug 24, 2019

krasi-georgiev commented Jul 4, 2019 •

edited

Loading

krasi-georgiev commented Jul 11, 2019 •

edited

Loading

krasi-georgiev commented Jul 19, 2019 •

edited

Loading