Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e tests: Move from Cortex packages to efficientgo/e2e #4579

Closed
matej-g opened this issue Aug 18, 2021 · 8 comments
Closed

e2e tests: Move from Cortex packages to efficientgo/e2e #4579

matej-g opened this issue Aug 18, 2021 · 8 comments
Assignees
@matej-g
Labels
feature request/improvement tests

Comments

@matej-g
Copy link
Collaborator

matej-g commented Aug 18, 2021

This relates to issue #3832 (comment), separating it so this change can be followed more clearly.

There is a (nearly) drop-in replacement for Cortex integration test packages (github.com/cortexproject/cortex/integration/e2e) which lives at github.com/efficientgo/e2e. In order to achieve one of the steps in aiming to not be dependent on Cortex, the e2e tests should be moved to this new package.
@matej-g
Copy link
Collaborator Author

matej-g commented Aug 18, 2021

also feel free to assign ☝️ to me

@matej-g
Copy link
Collaborator Author

matej-g commented Aug 23, 2021

Full resolution is blocked until efficientgo/e2e#7 is fixed

@bill3tt
Copy link
Contributor

bill3tt commented Aug 26, 2021

Hey @matej-g - I think that the upstream issue is now fixed and we can proceed 🚀

@matej-g matej-g mentioned this issue Aug 27, 2021
Merged
2 tasks
@matej-g
Copy link
Collaborator Author

matej-g commented Aug 27, 2021 •
edited

So saying it is a drop-in replacement was an overstatement 😉 There was a couple of small issues I had to work around, but all in all it seem to be working fine locally.

However, I still have suspicion there might be some flaky tests hidden or even other issues lurking. On occasion, I had failures from compactor tests, where compactor was failing at start up with uncommon errors (sometimes I see error fatal error: concurrent map read and map write; on other occasions I've seen panic: reflect: call of reflect.Value.Type on zero Value). I will try to provide full output of these runs once I collect it.

Anyway, I will be off for the next week, but still wanted to put it out there for others to try out - feel free to spin up the tests and see what you get on your end - #4610

@matej-g
Copy link
Collaborator Author

matej-g commented Sep 12, 2021

So I'm starting to suspect there might be data race in the compactor, but I haven't been able to chase it down completely. It's not always reproducible, but on a couple of runs, I can make the compactor panic with:

fatal error: concurrent map iteration and map write (full output) 
fatal error: concurrent map iteration and map write
goroutine 295 [running]:
runtime.throw({0x1ad3fc9, 0x4627b9})
/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000ac51b0 sp=0xc000ac5180 pc=0x435651
runtime.mapiternext(0x18)
/usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc000ac5220 sp=0xc000ac51b0 pc=0x40f82b
reflect.mapiternext(0x195)
/usr/local/go/src/runtime/map.go:1346 +0x19 fp=0xc000ac5238 sp=0xc000ac5220 pc=0x4626f9
reflect.(*MapIter).Next(0xc0007a8960)
/usr/local/go/src/reflect/value.go:1618 +0x99 fp=0xc000ac5258 sp=0xc000ac5238 pc=0x4bcd59
encoding/json.mapEncoder.encode({0x0}, 0xc000128f80, {0x180df80, 0xc000705798, 0x1}, {0x9, 0x0})
/usr/local/go/src/encoding/json/encode.go:799 +0x305 fp=0xc000ac5418 sp=0xc000ac5258 pc=0x5e6885
encoding/json.mapEncoder.encode-fm(0x19b6860, {0x180df80, 0xc000705798, 0xc000ac54b0}, {0x17, 0x0})
/usr/local/go/src/encoding/json/encode.go:779 +0x45 fp=0xc000ac5458 sp=0xc000ac5418 pc=0x5f05a5
encoding/json.structEncoder.encode({{{0xc00031ad80, 0x240, 0x199}, 0xc000676c60}}, 0xc000128f80, {0x19b6860, 0xc000705790, 0x1979520}, {0x0, 0x1})
/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc000ac5508 sp=0xc000ac5458 pc=0x5e62d4
encoding/json.structEncoder.encode-fm(0x5e5327, {0x19b6860, 0xc000705790, 0xc0000bd680}, {0x70, 0x46})
/usr/local/go/src/encoding/json/encode.go:732 +0x69 fp=0xc000ac5560 sp=0xc000ac5508 pc=0x5f04e9
encoding/json.typeEncoder.func1(0x19109c0, {0x19b6860, 0xc000705790, 0x10100c000044180}, {0x68, 0xa0})
/usr/local/go/src/encoding/json/encode.go:397 +0x85 fp=0xc000ac55c8 sp=0xc000ac5560 pc=0x5e4225
encoding/json.structEncoder.encode({{{0xc00031a900, 0xc000ac56b0, 0x5f86ee}, 0xc000676ba0}}, 0xc000128f80, {0x19109c0, 0xc000705700, 0x17}, {0x0, 0x1})
/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc000ac5678 sp=0xc000ac55c8 pc=0x5e62d4
encoding/json.structEncoder.encode-fm(0x17603e0, {0x19109c0, 0xc000705700, 0x4037e9}, {0x0, 0x7c})
/usr/local/go/src/encoding/json/encode.go:732 +
0x69 fp=0xc000ac56d0 sp=0xc000ac5678 pc=0x5f04e9
encoding/json.arrayEncoder.encode({0x8}, 0xc000128f80, {0x17603e0, 0xc000326ab0, 0x4035b2}, {0x90, 0x57})
/usr/local/go/src/encoding/json/encode.go:916 +0xb5 fp=0xc000ac5740 sp=0xc000ac56d0 pc=0x5e7c15
encoding/json.arrayEncoder.encode-fm(0x203000, {0x17603e0, 0xc000326ab0, 0x40e4f0}, {0xa8, 0x57})
/usr/local/go/src/encoding/json/encode.go:909 +0x45 fp=0xc000ac5780 sp=0xc000ac5740 pc=0x5f0725
encoding/json.sliceEncoder.encode({0x0}, 0xc000128f80, {0x17603e0, 0xc000326ab0, 0x17603e0}, {0x30, 0x0})
/usr/local/go/src/encoding/json/encode.go:889 +0x2e6 fp=0xc000ac5888 sp=0xc000ac5780 pc=0x5e7746
encoding/json.sliceEncoder.encode-fm(0x17603e0, {0x17603e0, 0xc000326ab0, 0x30}, {0x38, 0x2b
})
/usr/local/go/src/encoding/json/encode.go:869 +0x45 fp=0xc000ac58c8 sp=0xc000ac5888 pc=0x5f0665
encoding/json.(*encodeState).reflectValue(0x0, {0x17603e0, 0xc000326ab0, 0x40d187}, {0x78,
0x0})
/usr/local/go/src/encoding/json/encode.go:360 +0x78 fp=0xc000ac5928 sp=0xc000ac58c8 pc=0x5e3e78
encoding/json.(*encodeState).marshal(0x18, {0x17603e0, 0xc000326ab0}, {0x0, 0x0})
/usr/local/go/src/encoding/json/encode.go:332 +0xfa fp=0xc000ac59a0 sp=0xc000ac5928 pc=0x5e3a9a
encoding/json.Marshal
({0x17603e0, 0xc000326ab0})
/usr/local/go/src/encoding/json/encode.go:161 +0x45 fp=0xc000ac5a00 sp=0xc000ac59a0 pc=0x5e3105
github.com/thanos-io/thanos/pkg/ui.(*Bucket).Set(0xc000128c00, {0xc000704000, 0xc0009afbd0, 0x18044a0}, {0x0, 0x0})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/ui/bucket.go:89 +0xbe fp=0xc000ac5a48 sp=0xc000ac5a00 pc=0x11f86fe
main.runCompact.func11({
0xc000704000, 0x1e, 0x1e}, {0x0, 0x0})
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:523 +0x51 fp=0xc000ac5a90 sp=0xc000ac5a48 pc=0x1638fd1
github.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch(0xc000200840, {0x1dd13f8, 0xc00040fec0})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:503 +0x24d fp=
0xc000ac5c90 sp=0xc000ac5a90 pc=0xc3732d
main.runCompact.func14.1.1()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:550 +0x6a fp=0xc000ac5ce0 sp=0xc000ac5c90 pc=0x1638e6a
github.com/thanos-io/thanos/pkg/runutil.RetryWithLog({0x1d9b360, 0xc000b20820}, 0xc000680000, 0xc000b6a900, 0xc000ac5e18)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:98 +0xa6 fp=0xc000ac5dc0 sp=0xc000ac5ce0 pc=0xbcc626
main.runCompact.func14.1()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:546 +0xd3 fp=0xc000ac5e50 sp=0xc000ac5dc0 pc=0x1638dd3
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc000200840, 0xc000b6a900, 0xc000ac5f40)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:75 +0x90 fp=0xc000ac5ef0 sp=0xc000ac5e50 pc=0xbcc3f0
main.runCompact.func14()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:545 +0x125 fp=0xc000ac5f88 sp=0xc000ac5ef0 pc=0x1638ce5
github.com/oklog/run.(*Group).Run.func1({0xc0000ad740, 0xc00056f9c0})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000ac5fc0 sp=0xc000ac5f88 pc=0x5fbcaf
github.com/oklog/run.(*Group).Run·dwrap·1()
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x31 fp=0xc000ac5fe0 sp=0xc000ac5fc0 pc=0x5fbc51
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1581
+0x1 fp=0xc000ac5fe8 sp=0xc000ac5fe0 pc=0x468981
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 1 [chan receive]:
github.com/oklog/run.(*Group).Run(0xc000b3f740)
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x7c
main.main()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:155 +0x15ba
goroutine 10 [chan receive]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
/home/mgera/go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.81/util/log/logger.go:362 +0x65
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
/home/mgera/go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.81/util/log/logger.go:359 +0x107
goroutine 37 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000220180)
/home/mgera/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xb9
created by go.opencensus.io/stats/view.init.0
/home/mgera/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x92
goroutine 132 [chan receive]:
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact.func2()
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:957 +0x105
created by github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:955 +0x93b
goroutine 291 [chan receive]:
main.main.func2()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:115 +0x33
github.com/oklog/run.(*Group).Run.func1({
0xc000b3f758, 0xc000c6cf60})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 292 [IO wait
]:
internal/poll.runtime_pollWait(0x7faec259b108, 0x72)
/usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc0000d6380, 0x2, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +
0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0000d6380)
/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc0000d6380)
/usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc0008fe018)
/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc0008fe018)
/usr/local/go/src/net/tcpsock.go:262 +0x3d
net/http.(*Server).Serve(0xc00031e2a0, {0x1dbaee0, 0xc0008fe018})
/usr/local/go/src/net/http/server.go:3001 +0x394
github.com/prometheus/exporter-toolkit/web.Serve({0x1dbaee0, 0xc0008fe018}, 0xc00031e2a0, {0x0, 0x10}, {0x7faec23ced18,
0xc000b21720})
/home/mgera/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.6.0/web/tls_config.go:192 +0x17f
github.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc00031e2a0, {0x0, 0x0}, {0x7faec23ced18, 0xc000b21720})
/home/mgera/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.6.0/web/tls_config.go
:184 +0xf9
github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe(0xc000c8ebd0)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/server/http/http.go:68 +0x1a5
main.runCompact.func1()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:190 +0x35
github.com/oklog/run.(*Group).Run.func1({0xc000b53ae0, 0xc000b53b00})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 293 [runnable]:
github.com/thanos-io/thanos/pkg/block.(*ReplicaLabelRemover).Modify(0xc0009cd830, {0x1dd1388, 0xc00021f200}, 0xc000728d80, 0xc0001506e0)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:717 +0x492
github.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch(0xc000285b60, {
0x1dd1388, 0xc00021f200}, 0xc00021ecc0, {0xc0000945a0, 0x5, 0x1d7d990}, {0xc000283b10, 0x1, 0x1})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:464 +0xa87
github.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch(0xc000285ec0, {0x1dd1388, 0xc00021f200})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:497 +0x76
github.com/thanos-io/thanos/pkg/compact.(*Syncer).SyncMetas(0xc000128a00, {0x1dd1388, 0xc00021f200})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:137 +0xd0
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact(0xc000128a80
, {0x1dd1388, 0xc00021f200})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:1000 +0x28a
main.runCompact.func7()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:415 +0xdd
main.runCompact.func8.1()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:470 +0x53
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc00021f200, 0xc000b6a900, 0xc000ac1ef0)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:75 +0x90
main.runCompact.func8()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:469 +0x1cd
github.com/oklog/run.(*Group).Run.func1({0xc000094f00, 0xc0004162e0})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 294 [semacquire]:
sync.runtime_SemacquireMutex(0x0, 0x0, 0x64)
/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc000128a40)
/usr/local/go/src/sync/mutex.go:138 +
0x165
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:81
github.com/thanos-io/thanos/pkg/compact.(*Syncer).SyncMetas(0xc000128a00, {0x1dd1388, 0xc00021f200})
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:134 +0x7f
main.runCompact.func6()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:
400 +0x145
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x0, 0xc000b6a900, 0xc000044960)
/home/mgera/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:75 +0x90
main.runCompact.func12
()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:533 +0x50
github.com/oklog/run.(*Group).Run.func1
({0xc0002eb8c0, 0xc00056f9b0})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 296 [select]:
main.interrupt({0x1d9b360, 0xc000b20820}, 0xc0005ee7e0)
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:166 +0x10e
main.main.func4()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:139 +0x25
github.com/oklog/run.(*Group).Run.func1({0xc000151d60, 0xc00056fae0})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 297 [select]:
main.reload({0x1d9b360, 0xc000b20820}, 0xc0005ee840, 0x0)
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:179 +0x105
main.main.func6()
/home/mgera/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:149 +0x29
github.com/oklog/run.(*Group).Run.func1({0xc0002eb9b0, 0xc00056fb00})
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
/home/mgera/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x22f
goroutine 299 [syscall]:
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x2c
goroutine 1244 [IO wait]:
internal/poll.runtime_pollWait(0x7faec2273920, 0x72)
/usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000944700, 0xc000f46000, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000944700, {0xc000f46000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000944700, {0xc000f46000, 0x18,
0x18})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00040c118, {0xc000f46000, 0xc0002bc000, 0xc0008fe3c0})
/usr/local/go/src/net/net.go:183 +0x45
net/http.(*persistConn).Read(0xc000249b00, {0xc000f46000, 0x40503d, 0x60})
/usr/local/go/src/net/http/transport.go:1926 +
0x4e
bufio.(*Reader).fill(0xc00071b4a0
)
/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).Peek(0xc00071b4a0, 0x1)
/usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop(0xc000249b00)
/usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05
goroutine 1245 [select]:
net/http.(*persistConn).writeLoop(0xc000249b00)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 1243 [select]:
net/http.(*persistConn).writeLoop(0xc0006ea900)
/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1748 +0x1e65
goroutine 1242 [IO wait]:
internal/poll.runtime_pollWait(0x7faec2273838, 0x72)
/usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc00075c380,
0xc000f44000, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00075c380, {0xc000f44000, 0x1000
, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00075c380, {0xc000f44000, 0x18, 0x18842e0})
/usr/local/go/src/net/fd_posix.go:56 +
0x29
net.(*conn).Read(0xc00040c108, {0xc000f44000, 0xc0006b0db0, 0x0})
/usr/local/go/src/net/net.go:183 +0x45
net/http.(*persistConn).Read(
0xc0006ea900, {0xc000f44000, 0x40503d, 0x60})
/usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc00071b440)
/usr/local/go/src/bufio/bufio.go
:101 +0x103
bufio.(*Reader).Peek(0xc00071b440, 0x1)
/usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop
(0xc0006ea900)
/usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1747 +0x1e05

Alternatively, I have seen the compactor panic with panic: reflect: call of reflect.Value.Type on zero Value instead (but in the same place in code).

It looks like this occurs during marshaling of blocks in bucket UI's Set, originating from here in cmd/compact.go.

Best way to attempt to reproduce is by checking out to #4610, doing make docker and subsequently running go test ./... -v -run=TestCompact from with the e2e directory (for best results, run with -count=N to ensure the tests are repeated and that you increase the chance of catching the above issue).

@bwplotka
Copy link
Member

Reason is that we modify block meta here:

thanos/pkg/block/fetcher.go

Line 718 in 1964c0c

delete(l, replicaLabel)

and we use same pointer here:

thanos/pkg/ui/bucket.go

Line 89 in 3d7c0dd

dataB, err := json.Marshal(blocks)

Checking what we can do

This was referenced Sep 16, 2021
Merged
Closed
@matej-g
Copy link
Collaborator Author

matej-g commented Sep 22, 2021

The blocking issue has been resolved, PR is open for business 👍

@matej-g
Copy link
Collaborator Author

matej-g commented Oct 6, 2021

This is done now, closing the issue.

@matej-g matej-g closed this as completed Oct 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matej-g matej-g

Labels
feature request/improvement tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@GiedriusS @bwplotka @bill3tt @matej-g