Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[global sort] add mv index using global sort can not finished with subtask hang #50307

Closed
seiya-annie opened this issue Jan 11, 2024 · 2 comments
Labels
affects-7.5 affects-7.6 component/ddl This issue is related to DDL of TiDB. severity/critical type/bug This issue is a bug.

Comments

@seiya-annie
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

MySQL [test]> alter table items add index domains1 ((cast(domains as char(253) array)));

2. What did you expect to see? (Required)

add index successfully

3. What did you see instead (Required)

hang for 4hours
| 144 | test | items | add index /* ingest cloud */ | write reorganization | 2 | 131 | 606105674 | 2024-01-11 10:36:31 | 2024-01-11 10:39:52 | NULL | running |

4. What is your TiDB version? (Required)

master

@seiya-annie seiya-annie added type/bug This issue is a bug. component/ddl This issue is related to DDL of TiDB. labels Jan 11, 2024
@Benjamin2037
Copy link
Collaborator

Do you keep the log?

@tangenta
Copy link
Contributor

It seems that there is something wrong with ks3 SDK. It never returns:

goroutine 19531629 [IO wait]:
internal/poll.runtime_pollWait(0x7fd4dac3aa78, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc036448000?, 0xc00a95f864?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc036448000, {0xc00a95f864, 0x3cf1, 0x3cf1})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc036448000, {0xc00a95f864?, 0x1c45e51?, 0x77e8736?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00fd72b70, {0xc00a95f864?, 0x77e86c7?, 0xc00a9472f0?})
        /usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc0129fb8c0, {0xc00a95f864?, 0x5555?, 0x5555?})
        /usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).Read(0xc0179965a0, {0xc00a95f864, 0x3cf1, 0x7fd4f63c76b8?})
        /usr/local/go/src/bufio/bufio.go:230 +0xe3
io.(*LimitedReader).Read(0xc008032e88, {0xc00a95f864?, 0x1fc4c9f?, 0x5a0f877?})
        /usr/local/go/src/io/io.go:480 +0x43
net/http.(*body).readLocked(0xc0114b6900, {0xc00a95f864?, 0xc00a9473f8?, 0x1c31351?})
        /usr/local/go/src/net/http/transfer.go:839 +0x3b
net/http.(*body).Read(0xc0038f3520?, {0xc00a95f864?, 0xc00a947490?, 0x1c4fe32?})
        /usr/local/go/src/net/http/transfer.go:831 +0x118
net/http.(*bodyEOFSignal).Read(0xc0114b6940, {0xc00a95f864, 0x3cf1, 0x3cf1})
        /usr/local/go/src/net/http/transport.go:2810 +0x13f
[github.com/pingcap/tidb/br/pkg/storage.(*ks3ObjectReader).Read](http://github.com/pingcap/tidb/br/pkg/storage.(*ks3ObjectReader).Read)(0xc006627500, {0xc00a95f864, 0xc00a947610?, 0x3cf1})
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/storage/ks3.go:528](http://github.com/pingcap/tidb/br/pkg/storage/ks3.go:528) +0x7a
io.ReadAtLeast({0x63e6a60, 0xc006627500}, {0xc00a95e000, 0x5555, 0x5555}, 0x5555)
        /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*byteReader).reload(0xc0168a60d0)](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*byteReader).reload(0xc0168a60d0))
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/byte_reader.go:300](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/byte_reader.go:300) +0x1b0
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*byteReader).readNBytes](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*byteReader).readNBytes)(0xc00a947870?, 0x57)
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/byte_reader.go:228](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/byte_reader.go:228) +0x271
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*kvReader).nextKV(0xc025215048)](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*kvReader).nextKV(0xc025215048))
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/kv_reader.go:64](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/kv_reader.go:64) +0x9b
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.kvReaderProxy.next](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.kvReaderProxy.next)({{0xc03c4ade00?, 0xc0161f6fc0?}, 0xc025215048?})
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:481](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:481) +0x25
[github.com/pingcap/tidb/br/pkg/lightning/backend/external](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external).(*mergeIter[...]).next(0x644c5e0)
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:300](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:300) +0x235
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*MergeKVIter).Next(...)](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*MergeKVIter).Next(...))
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:554](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:554)
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.mergeOverlappingFilesInternal](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.mergeOverlappingFilesInternal)({0x64151c0?, 0xc00f9aa280}, {0xc004c34b20, 0x3e, 0x34e}, {0x6434098, 0xc0058a4850}, 0xc00f73fd93?, 0x3?, {0xc007955600, ...}, ...)
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/merge.go:131](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/merge.go:131) +0x587
[github.com/pingcap/tidb/br/pkg/lightning/backend/external.MergeOverlappingFiles.func1()](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external.MergeOverlappingFiles.func1())
        /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/br/pkg/lightning/backend/external/merge.go:57](http://github.com/pingcap/tidb/br/pkg/lightning/backend/external/merge.go:57) +0x247
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
        /go/pkg/mod/[golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75](http://golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75) +0x56
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go) in goroutine 18539109
        /go/pkg/mod/[golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72](http://golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72) +0x96

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-7.5 affects-7.6 component/ddl This issue is related to DDL of TiDB. severity/critical type/bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

6 participants