Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rules: manager: avoid panic when cloning when TZ != UTC #2925

Merged
merged 2 commits into from
Jul 27, 2020

Conversation

GiedriusS
Copy link
Member

@GiedriusS GiedriusS commented Jul 22, 2020

The following panic can be observed when calling /api/v1/rules without
this change when the TZ is not UTC:

2020/07/22 21:44:22 http: panic serving 127.0.0.1:49368: merger not found for type:int
goroutine 529 [running]:
net/http.(*conn).serve.func1(0xc00029e640)
        /usr/lib/go-1.14/src/net/http/server.go:1772 +0x139
panic(0x1e0a400, 0xc000368c30)
        /usr/lib/go-1.14/src/runtime/panic.go:975 +0x3e3
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo(0xc0008b8780)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:662 +0xfc6
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8780, 0xc0007447a0, 0xc00057bb00)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:113 +0x4a2
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func27(0xc000091b70, 0x3991910)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:545 +0x381
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8700, 0xc000091b60, 0x3991900)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func30(0xc00061e470, 0xc00061e370)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:587 +0x5c
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8340, 0xc00061e460, 0xc00061e360)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func28(0xc00061e460, 0xc00061e360)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:555 +0x3e
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b83c0, 0xc00061e400, 0xc00061e300)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*InternalMessageInfo).Merge(0x398a440, 0x2676ea0, 0xc00061e400, 0x2676ea0, 0xc00061e300)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:50 +0x4c
github.com/thanos-io/thanos/pkg/rules/rulespb.(*RecordingRule).XXX_Merge(0xc00061e400, 0x2676ea0, 0xc00061e300)
        /home/gstatkevicius/dev/thanos/pkg/rules/rulespb/rpc.pb.go:527 +0x57

Let's use the workaround given here:
gogo/protobuf#519.

Signed-off-by: Giedrius Statkevičius giedriuswork@gmail.com

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Forcing the conversion of LastEvaluation into UTC to fix the observed panic

Verification

PROMETHEUS_EXECUTABLE=~/dev/prometheus/prometheus ./scripts/quickstart.sh and opened http://localhost:19999/api/v1/rules

The following panic can be observed when calling `api/v1/rules` without
this change:

```
2020/07/22 21:44:22 http: panic serving 127.0.0.1:49368: merger not found for type:int
goroutine 529 [running]:
net/http.(*conn).serve.func1(0xc00029e640)
        /usr/lib/go-1.14/src/net/http/server.go:1772 +0x139
panic(0x1e0a400, 0xc000368c30)
        /usr/lib/go-1.14/src/runtime/panic.go:975 +0x3e3
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo(0xc0008b8780)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:662 +0xfc6
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8780, 0xc0007447a0, 0xc00057bb00)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:113 +0x4a2
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func27(0xc000091b70, 0x3991910)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:545 +0x381
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8700, 0xc000091b60, 0x3991900)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func30(0xc00061e470, 0xc00061e370)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:587 +0x5c
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b8340, 0xc00061e460, 0xc00061e360)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func28(0xc00061e460, 0xc00061e360)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:555 +0x3e
github.com/gogo/protobuf/proto.(*mergeInfo).merge(0xc0008b83c0, 0xc00061e400, 0xc00061e300)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:139 +0x46a
github.com/gogo/protobuf/proto.(*InternalMessageInfo).Merge(0x398a440, 0x2676ea0, 0xc00061e400, 0x2676ea0, 0xc00061e300)
        /home/gstatkevicius/go/pkg/mod/github.com/gogo/protobuf@v1.3.1/proto/table_merge.go:50 +0x4c
github.com/thanos-io/thanos/pkg/rules/rulespb.(*RecordingRule).XXX_Merge(0xc00061e400, 0x2676ea0, 0xc00061e300)
        /home/gstatkevicius/dev/thanos/pkg/rules/rulespb/rpc.pb.go:527 +0x57
```

Let's use the workaround given here:
gogo/protobuf#519.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
@GiedriusS GiedriusS changed the title rules: manager: avoid panic when cloning [WIP] rules: manager: avoid panic when cloning Jul 22, 2020
@onprem onprem mentioned this pull request Jul 22, 2020
2 tasks
Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
@GiedriusS GiedriusS changed the title [WIP] rules: manager: avoid panic when cloning rules: manager: avoid panic when cloning when TZ != UTC Jul 24, 2020
Copy link
Member

@onprem onprem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was able to reproduce the panic. This patch seems to fix it. LGTM.

Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 🎖

@GiedriusS GiedriusS merged commit 5719a33 into thanos-io:master Jul 27, 2020
GiedriusS added a commit to GiedriusS/thanos that referenced this pull request Jul 29, 2020
Fixes thanos-io#2938 and is a follow up
to thanos-io#2925.

Changes:
* Copying all fields properly to the group-level as well
* Set TZ to UTC explicitly in other `time.Time` fields to avoid a panic
* Make sure that `RuleGroup` always has an empty array in the `rules`
field because that's customary and that is what the new React UI
expects.

Manual testing for now.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
bwplotka pushed a commit that referenced this pull request Aug 11, 2020
* rules: fix silly mistakes in the rules API

Fixes #2938 and is a follow up
to #2925.

Changes:
* Copying all fields properly to the group-level as well
* Set TZ to UTC explicitly in other `time.Time` fields to avoid a panic
* Make sure that `RuleGroup` always has an empty array in the `rules`
field because that's customary and that is what the new React UI
expects.

Manual testing for now.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* rules: remove DeprecatedPartialResponseStrategy

It has been marked deprecated for more than 2 months. Remove it finally.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* e2e: rules_api: add checks for non-zero values

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* test: e2e: fix rule test

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* test: rule: fix govet error

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* Update CHANGELOG.md

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

Co-authored-by: Lucas Servén Marín <lserven@gmail.com>

* Update CHANGELOG.md

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

Co-authored-by: Lucas Servén Marín <lserven@gmail.com>

Co-authored-by: Lucas Servén Marín <lserven@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants