Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/tempo-vulture: share rand.Rand instance across multiple searches #1763

Merged
merged 3 commits into from Oct 5, 2022
Merged

cmd/tempo-vulture: share rand.Rand instance across multiple searches #1763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6310e59
cmd/tempo-vulture: share rand.Rand instance across multiple searches
rfratto Sep 27, 2022
03402bf
update CHANGELOG
rfratto Sep 27, 2022
f535a97
Merge branch 'main' into vulture-randomization-fix
rfratto Oct 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Expand Up @@ -7,7 +7,7 @@
* [CHANGE] Make DNS address fully qualified to reduce DNS lookups in Kubernetes [#1687](https://github.com/grafana/tempo/pull/1687) (@electron0zero)
* [CHANGE] Improve parquet compaction memory profile when dropping spans [#1692](https://github.com/grafana/tempo/pull/1692) (@joe-elliott)
* [CHANGE] Increase default values for `server.grpc_server_max_recv_msg_size` and `server.grpc_server_max_send_msg_size` from 4MB to 16MB [#1688](https://github.com/grafana/tempo/pull/1688) (@mapno)
* [CHANGE] Use Parquet for local block search, tag search and tag value search instead of flatbuffers. A configuration value
* [CHANGE] Use Parquet for local block search, tag search and tag value search instead of flatbuffers. A configuration value
(`ingester.use_flatbuffer_search`) is provided to continue using flatbuffers.
- **BREAKING CHANGE** Makes Parquet the default encoding.
* [CHANGE] **BREAKING CHANGE** Use storage.trace.block.row_group_size_bytes to cut rows during compaction instead of
Expand Down Expand Up @@ -35,6 +35,7 @@ query_frontend:
hedge_requests_up_to: 3
```
* [ENHACEMENT] Update OTel collector to v0.57.2 [#1757](https://github.com/grafana/tempo/pull/1757) (@mapno)
* [ENHANCEMENT] Vulture now has improved distribution of the random traces it searches. [#1763](https://github.com/grafana/tempo/pull/1763) (@rfratto)
* [BUGFIX] Honor caching and buffering settings when finding traces by id [#1697](https://github.com/grafana/tempo/pull/1697) (@joe-elliott)
* [BUGFIX] Correctly propagate errors from the iterator layer up through the queriers [#1723](https://github.com/grafana/tempo/pull/1723) (@joe-elliott)
* [ENHANCEMENT] Upgrade opentelemetry-proto submodule to v0.18.0 [#1754](https://github.com/grafana/tempo/pull/1754) (@mapno)
Expand Down
10 changes: 6 additions & 4 deletions cmd/tempo-vulture/main.go
View file
Open in desktop
Expand Up @@ -82,6 +82,8 @@ func main() {
startTime := actualStartTime
tickerWrite := time.NewTicker(tempoWriteBackoffDuration)

r := rand.New(rand.NewSource(actualStartTime.Unix()))

var tickerRead *time.Ticker
if tempoReadBackoffDuration > 0 {
tickerRead = time.NewTicker(tempoReadBackoffDuration)
Expand Down Expand Up @@ -140,7 +142,7 @@ func main() {
go func() {
for now := range tickerRead.C {
var seed time.Time
startTime, seed = selectPastTimestamp(startTime, now, interval, tempoRetentionDuration)
startTime, seed = selectPastTimestamp(startTime, now, interval, tempoRetentionDuration, r)

log := logger.With(
zap.String("org_id", tempoOrgID),
Expand Down Expand Up @@ -173,7 +175,7 @@ func main() {
if tickerSearch != nil {
go func() {
for now := range tickerSearch.C {
_, seed := selectPastTimestamp(startTime, now, interval, tempoRetentionDuration)
_, seed := selectPastTimestamp(startTime, now, interval, tempoRetentionDuration, r)
log := logger.With(
zap.String("org_id", tempoOrgID),
zap.Int64("seed", seed.Unix()),
Expand Down Expand Up @@ -251,7 +253,7 @@ func pushMetrics(metrics traceMetrics) {
metricTracesErrors.WithLabelValues("notfound_search_attribute").Add(float64(metrics.notFoundSearchAttribute))
}

func selectPastTimestamp(start, stop time.Time, interval time.Duration, retention time.Duration) (newStart, ts time.Time) {
func selectPastTimestamp(start, stop time.Time, interval, retention time.Duration, r *rand.Rand) (newStart, ts time.Time) {
oldest := stop.Add(-retention)

if oldest.After(start) {
Expand All @@ -260,7 +262,7 @@ func selectPastTimestamp(start, stop time.Time, interval time.Duration, retentio
newStart = start
}

ts = time.Unix(generateRandomInt(newStart.Unix(), stop.Unix(), rand.New(rand.NewSource(start.Unix()))), 0)
ts = time.Unix(generateRandomInt(newStart.Unix(), stop.Unix(), r), 0)

return newStart.Round(interval), ts.Round(interval)
}
Expand Down