You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During thanos-compact, the process reports the known issue prometheus-junkyard/tsdb#347 but exists just after that.
the exit error is 1, so doesn't look a normal exist ( and i would expect some ending output too)
The s3 data have several years, from much older thanos versions (as old as thanos 0.5 IIRC)
i bypassed thanos compact for several years, as data wasn't that much and had many initial problems with compact, only recently gave it another try
What you expected to happen:
fix the problem and continue to work
How to reproduce it (as minimally and precisely as possible):
can reproduce locally, and probably can download the s3 data and send if needed
Full logs to relevant components:
Logs
level=warn ts=2022-09-06T00:28:16.993082834Z caller=index.go:267 msg="out-of-order label set: known bug in Prometheus 2.8.0 and below" labelset="{__name__=\"zookeeper_znode_count\", address=\"172.26.30.116:9009\", alias=\"172.26.30.116\", dc=\"interxion-fra6\", environment=\"staging\", exported_dc=\"eu-central-1a\", host=\"zookeeper-staging-a02\", id=\"zookeeper-staging-a02\", instance=\"172.26.30.116:9009\", job=\"HostsMetrics\", node=\"zookeeper-staging-a02\", port=\"2181\", region=\"eu-central-1\", rkt=\"true\", server=\"zookeeper-staging-a03\", service=\"telegraf:metrics\", service_id=\"zookeeper-staging-a02:telegraf\", service_port=\"9009\", state=\"follower\", tags=\",http,telegraf,host,output,metrics,alias_zookeeper-staging-a02,eu-central-1a,eu-central-1,\", type=\"type_t2_small\", version=\"3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0\", zookeeper=\"true\", url=\"http://localhost:10020\", vhost=\"jm_staging\", version_malloc_library=\"system jemalloc\", version_ssl_library=\"OpenSSL 1.0.2l 25 May 2017\", wsrep_patch_version=\"wsrep_25.21\", type=\"type_i3_2xlarge\", vault=\"true\", vault=\"true\", type=\"type_c5_large\", vault=\"true\"}" series=68344
level=warn ts=2022-09-06T00:28:17.187858177Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347"
level=info ts=2022-09-06T00:28:17.187929581Z caller=http.go:84 service=http/server component=compact msg="internal server is shutting down" err="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347"
level=info ts=2022-09-06T00:28:17.189210412Z caller=http.go:103 service=http/server component=compact msg="internal server is shutdown gracefully" err="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347"
level=info ts=2022-09-06T00:28:17.189271633Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347"
level=error ts=2022-09-06T00:28:17.189477697Z caller=main.go:158 err="group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:424\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:478\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:477\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nerror executing compaction\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:505\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:477\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:158\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"
and them the process terminates with error 1
Anything else we need to know:
The text was updated successfully, but these errors were encountered:
Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.
Thanos, Prometheus and Golang version used:
quay.io/thanos/thanos:v0.28.0
running inside a k8s setup with this startup options:
Object Storage Provider:
S3
What happened:
During thanos-compact, the process reports the known issue prometheus-junkyard/tsdb#347 but exists just after that.
the exit error is 1, so doesn't look a normal exist ( and i would expect some ending output too)
The s3 data have several years, from much older thanos versions (as old as thanos 0.5 IIRC)
i bypassed thanos compact for several years, as data wasn't that much and had many initial problems with compact, only recently gave it another try
What you expected to happen:
fix the problem and continue to work
How to reproduce it (as minimally and precisely as possible):
can reproduce locally, and probably can download the s3 data and send if needed
Full logs to relevant components:
level=warn ts=2022-09-06T00:28:16.993082834Z caller=index.go:267 msg="out-of-order label set: known bug in Prometheus 2.8.0 and below" labelset="{__name__=\"zookeeper_znode_count\", address=\"172.26.30.116:9009\", alias=\"172.26.30.116\", dc=\"interxion-fra6\", environment=\"staging\", exported_dc=\"eu-central-1a\", host=\"zookeeper-staging-a02\", id=\"zookeeper-staging-a02\", instance=\"172.26.30.116:9009\", job=\"HostsMetrics\", node=\"zookeeper-staging-a02\", port=\"2181\", region=\"eu-central-1\", rkt=\"true\", server=\"zookeeper-staging-a03\", service=\"telegraf:metrics\", service_id=\"zookeeper-staging-a02:telegraf\", service_port=\"9009\", state=\"follower\", tags=\",http,telegraf,host,output,metrics,alias_zookeeper-staging-a02,eu-central-1a,eu-central-1,\", type=\"type_t2_small\", version=\"3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0\", zookeeper=\"true\", url=\"http://localhost:10020\", vhost=\"jm_staging\", version_malloc_library=\"system jemalloc\", version_ssl_library=\"OpenSSL 1.0.2l 25 May 2017\", wsrep_patch_version=\"wsrep_25.21\", type=\"type_i3_2xlarge\", vault=\"true\", vault=\"true\", type=\"type_c5_large\", vault=\"true\"}" series=68344 level=warn ts=2022-09-06T00:28:17.187858177Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347" level=info ts=2022-09-06T00:28:17.187929581Z caller=http.go:84 service=http/server component=compact msg="internal server is shutting down" err="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347" level=info ts=2022-09-06T00:28:17.189210412Z caller=http.go:103 service=http/server component=compact msg="internal server is shutdown gracefully" err="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347" level=info ts=2022-09-06T00:28:17.189271633Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="error executing compaction: compaction: group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347" level=error ts=2022-09-06T00:28:17.189477697Z caller=main.go:158 err="group 0@2976064661857305202: invalid, but reparable block /prometheus/tmp/compact/0@2976064661857305202/01CVKPRHWYEZBZMZ65JWWV1AF1: found 48 chunks outside the block time range introduced by https://github.com/prometheus-junkyard/tsdb/issues/347\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:424\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:478\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:477\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nerror executing compaction\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:505\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:477\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:158\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"
and them the process terminates with error 1
Anything else we need to know:
The text was updated successfully, but these errors were encountered: