Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic when querying the region by ID if the region doesn't have a leader #7630

Closed
rleungx opened this issue Dec 27, 2023 · 0 comments · Fixed by #7629
Closed

Panic when querying the region by ID if the region doesn't have a leader #7630

rleungx opened this issue Dec 27, 2023 · 0 comments · Fixed by #7629
Labels
affects-6.5 affects-7.1 affects-7.5 severity/major The issue's severity is major. type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports

Comments

@rleungx
Copy link
Member

rleungx commented Dec 27, 2023

Bug Report

When the region doesn't have a leader, if we find the region using pd-ctl, it will panic.

{
  "id": 81321620,
  "start_key": "78004E5700000000FB",
  "end_key": "78004E5800000000FB",
  "epoch": {
    "conf_ver": 365,
    "version": 16197
  },
  "peers": [
    {
      "role_name": "Voter",
      "id": 82078310,
      "store_id": 81024312
    },
    {
      "role_name": "Voter",
      "id": 82168241,
      "store_id": 81024294
    },
    {
      "role_name": "Voter",
      "id": 82168851,
      "store_id": 81024317
    }
  ],
  "leader": {
    "role_name": ""
  }
}

./bin/pd-ctl region 81321620               
Failed to get region: [500] PANIC: runtime error: invalid memory address or nil pointer dereference
goroutine 13729102 [running]:
[github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()](http://github.com/urfave/negroni.(*Recovery).ServeHTTP.func1())
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/recovery.go:159](http://github.com/urfave/negroni@v0.3.0/recovery.go:159) +0x9c
panic({0x124f840?, 0x2457b60?})
        /usr/local/go/src/runtime/panic.go:914 +0x218
encoding/json.(*encodeState).marshal.func1()
        /usr/local/go/src/encoding/json/encode.go:291 +0x80
panic({0x124f840?, 0x2457b60?})
        /usr/local/go/src/runtime/panic.go:914 +0x218
[github.com/tikv/pd/server/api.easyjson75d7afa0EncodeGithubComTikvPdServerApi1](http://github.com/tikv/pd/server/api.easyjson75d7afa0EncodeGithubComTikvPdServerApi1)(0x400ae01b30?, {0x0?, {0x0?, 0x0?}, 0x0?})
        /go/src/[github.com/tikv/pd/server/api/region_easyjson.go:491](http://github.com/tikv/pd/server/api/region_easyjson.go:491) +0x8c
[github.com/tikv/pd/server/api.easyjson75d7afa0EncodeGithubComTikvPdServerApi](http://github.com/tikv/pd/server/api.easyjson75d7afa0EncodeGithubComTikvPdServerApi)(_, {0x4d8de94, {0x4020bc9e00, 0x12}, {0x4020bc9e18, 0x12}, 0x40016704e0, {0x40239f30e0, 0x3, 0x3}, ...})
        /go/src/[github.com/tikv/pd/server/api/region_easyjson.go:226](http://github.com/tikv/pd/server/api/region_easyjson.go:226) +0x1e0
[github.com/tikv/pd/server/api.RegionInfo.MarshalJSON](http://github.com/tikv/pd/server/api.RegionInfo.MarshalJSON)({0x4d8de94, {0x4020bc9e00, 0x12}, {0x4020bc9e18, 0x12}, 0x40016704e0, {0x40239f30e0, 0x3, 0x3}, {0x0, ...}, ...})
        /go/src/[github.com/tikv/pd/server/api/region_easyjson.go:316](http://github.com/tikv/pd/server/api/region_easyjson.go:316) +0x68
encoding/json.marshalerEncoder(0x4011dfcbc0, {0x12fd880?, 0x401db46500?, 0x5e988?}, {0xb8?, 0x1e?})
        /usr/local/go/src/encoding/json/encode.go:442 +0xb8
encoding/json.(*encodeState).reflectValue(0x400ae01ef8?, {0x12fd880?, 0x401db46500?, 0x5e898?}, {0x1?, 0x0?})
        /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x0?, {0x12fd880?, 0x401db46500?}, {0x0?, 0x0?})
        /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x12fd880, 0x401db46500})
        /usr/local/go/src/encoding/json/encode.go:162 +0x94
encoding/json.MarshalIndent({0x12fd880?, 0x401db46500?}, {0x0, 0x0}, {0x14a5f1c, 0x2})
        /usr/local/go/src/encoding/json/encode.go:175 +0x34
[github.com/unrolled/render.JSON.Render](http://github.com/unrolled/render.JSON.Render)({{{0x4000d90660, 0x1f}, 0xc8}, 0x1, 0x0, {0x0, 0x0, 0x0}, 0x0}, {0xfffd0abfd550, ...}, ...)
        /go/pkg/mod/[github.com/unrolled/render@v1.0.1/engine.go:112](http://github.com/unrolled/render@v1.0.1/engine.go:112) +0x84
[github.com/unrolled/render.(*Render).Render](http://github.com/unrolled/render.(*Render).Render)(0x4000276420, {0xfffd0abfd550, 0x400abf2af8}, {0x17664e0?, 0x40127d1b80?}, {0x12fd880?, 0x401db46500?})
        /go/pkg/mod/[github.com/unrolled/render@v1.0.1/render.go:341](http://github.com/unrolled/render@v1.0.1/render.go:341) +0x48
[github.com/unrolled/render.(*Render).JSON](http://github.com/unrolled/render.(*Render).JSON)(0x4000276420, {0xfffd0abfd550, 0x400abf2af8}, 0xc8, {0x12fd880, 0x401db46500})
        /go/pkg/mod/[github.com/unrolled/render@v1.0.1/render.go:408](http://github.com/unrolled/render@v1.0.1/render.go:408) +0xe4
[github.com/tikv/pd/server/api.(*regionHandler).GetRegionByID](http://github.com/tikv/pd/server/api.(*regionHandler).GetRegionByID)(0x4000557a20, {0xfffd0abfd4a0?, 0x400abf2af8}, 0xe958?)
        /go/src/[github.com/tikv/pd/server/api/region.go:260](http://github.com/tikv/pd/server/api/region.go:260) +0x150
[github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1](http://github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1)({0xfffd0abfd4a0, 0x400abf2af8}, 0x400ae02301?, 0x4004d21ca0)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:56](http://github.com/urfave/negroni@v0.3.0/negroni.go:56) +0x34
[github.com/urfave/negroni.HandlerFunc.ServeHTTP](http://github.com/urfave/negroni.HandlerFunc.ServeHTTP)(0x155e958?, {0xfffd0abfd4a0?, 0x400abf2af8?}, 0x171ef8?, 0x400ae02588?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:29](http://github.com/urfave/negroni@v0.3.0/negroni.go:29) +0x3c
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x17647c0?, 0x4000557b10?}, 0x400064e288?}, {0xfffd0abfd4a0, 0x400abf2af8}, 0x17e79c?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP](http://github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP)(0xfffd0a6ebc18?, {0xfffd0abfd4a0?, 0x400abf2af8?}, 0x4000064400?, 0x4015b02f20?)
        /go/src/[github.com/tikv/pd/server/api/middleware.go:170](http://github.com/tikv/pd/server/api/middleware.go:170) +0x2a4
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x1764fa0?, 0x40005391b0?}, 0x400064e270?}, {0xfffd0abfd4a0, 0x400abf2af8}, 0x4015b02f20?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP](http://github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP)(0x40005391a8, {0xfffd0abfd4a0, 0x400abf2af8}, 0x401db46300, 0x4004d21c60)
        /go/src/[github.com/tikv/pd/server/api/middleware.go:150](http://github.com/tikv/pd/server/api/middleware.go:150) +0x478
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x1764fc0?, 0x40005391a8?}, 0x400064e258?}, {0xfffd0abfd4a0, 0x400abf2af8}, 0x108f290?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP](http://github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP)(0x8?, {0xfffd0abfd4a0?, 0x400abf2af8?}, 0x401db46200, 0x4004d21c40)
        /go/src/[github.com/tikv/pd/server/api/middleware.go:76](http://github.com/tikv/pd/server/api/middleware.go:76) +0x2ac
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x1764fe0?, 0x40005391a0?}, 0x400064e240?}, {0xfffd0abfd4a0, 0x400abf2af8}, 0x395b44?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/urfave/negroni.(*Negroni).ServeHTTP](http://github.com/urfave/negroni.(*Negroni).ServeHTTP)(0x40006191d0, {0xfffd0abfd4a0?, 0x400abf2af0}, 0x24b8a40?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:96](http://github.com/urfave/negroni@v0.3.0/negroni.go:96) +0x108
[github.com/tikv/pd/server/api.clusterMiddleware.Middleware-fm.clusterMiddleware.Middleware.func1](http://github.com/tikv/pd/server/api.clusterMiddleware.Middleware-fm.clusterMiddleware.Middleware.func1)({0xfffd0abfd4a0?, 0x400abf2af0}, 0x40005d57c0?)
        /go/src/[github.com/tikv/pd/server/api/middleware.go:99](http://github.com/tikv/pd/server/api/middleware.go:99) +0xac
net/http.HandlerFunc.ServeHTTP(0x401db46000?, {0xfffd0abfd4a0?, 0x400abf2af0?}, 0x27400?)
        /usr/local/go/src/net/http/server.go:2136 +0x38
[github.com/gorilla/mux.(*Router).ServeHTTP](http://github.com/gorilla/mux.(*Router).ServeHTTP)(0x4000534d80, {0xfffd0abfd4a0, 0x400abf2af0}, 0x40108f9f00)
        /go/pkg/mod/[github.com/gorilla/mux@v1.7.4/mux.go:210](http://github.com/gorilla/mux@v1.7.4/mux.go:210) +0x194
[github.com/tikv/pd/server/api.NewHandler.Wrap.func21](http://github.com/tikv/pd/server/api.NewHandler.Wrap.func21)({0xfffd0abfd4a0, 0x400abf2af0}, 0x4a3a1079d201?, 0x4004d21be0)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:46](http://github.com/urfave/negroni@v0.3.0/negroni.go:46) +0x54
[github.com/urfave/negroni.HandlerFunc.ServeHTTP](http://github.com/urfave/negroni.HandlerFunc.ServeHTTP)(0x1471600?, {0xfffd0abfd4a0?, 0x400abf2af0?}, 0x402da0bcf8?, 0x40040eefa8?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:29](http://github.com/urfave/negroni@v0.3.0/negroni.go:29) +0x3c
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x17647c0?, 0x400064f500?}, 0x400064f548?}, {0xfffd0abfd4a0, 0x400abf2af0}, 0x18?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP](http://github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP)(0x400018ed60, {0xfffd0abfd4a0?, 0x400abf2af0}, 0x40108f9f00, 0x4004d21ba0)
        /go/src/[github.com/tikv/pd/pkg/utils/apiutil/serverapi/middleware.go:181](http://github.com/tikv/pd/pkg/utils/apiutil/serverapi/middleware.go:181) +0xc0
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x17647a0?, 0x400018ed60?}, 0x400064f530?}, {0xfffd0abfd4a0, 0x400abf2af0}, 0x1f44c?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP](http://github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP)(0x8?, {0xfffd0abfd4a0, 0x400abf2af0}, 0x0?, 0x4004d21b80)
        /go/src/[github.com/tikv/pd/pkg/utils/apiutil/serverapi/middleware.go:45](http://github.com/tikv/pd/pkg/utils/apiutil/serverapi/middleware.go:45) +0xa0
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x1765140?, 0x4000794bc0?}, 0x400064f518?}, {0xfffd0abfd4a0, 0x400abf2af0}, 0x175fe08?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/urfave/negroni.(*Negroni).ServeHTTP](http://github.com/urfave/negroni.(*Negroni).ServeHTTP)(0x40003d49c0, {0xfffd0abfd4a0?, 0x400abf2ae8}, 0xffff92104a68?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:96](http://github.com/urfave/negroni@v0.3.0/negroni.go:96) +0x108
[github.com/gorilla/mux.(*Router).ServeHTTP](http://github.com/gorilla/mux.(*Router).ServeHTTP)(0x4000534780, {0xfffd0abfd4a0, 0x400abf2ae8}, 0x40108f9c00)
        /go/pkg/mod/[github.com/gorilla/mux@v1.7.4/mux.go:210](http://github.com/gorilla/mux@v1.7.4/mux.go:210) +0x194
[github.com/gorilla/mux.(*Router).ServeHTTP](http://github.com/gorilla/mux.(*Router).ServeHTTP)(0x4000534c00, {0xfffd0abfd4a0, 0x400abf2ae8}, 0x40108f9900)
        /go/pkg/mod/[github.com/gorilla/mux@v1.7.4/mux.go:210](http://github.com/gorilla/mux@v1.7.4/mux.go:210) +0x194
[github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1](http://github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1)({0xfffd0abfd4a0, 0x400abf2ae8}, 0x1764601?, 0x4004d21b60)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:46](http://github.com/urfave/negroni@v0.3.0/negroni.go:46) +0x54
[github.com/urfave/negroni.HandlerFunc.ServeHTTP](http://github.com/urfave/negroni.HandlerFunc.ServeHTTP)(0x3331382f64692f6e?, {0xfffd0abfd4a0?, 0x400abf2ae8?}, 0x14a7070?, 0x400e10e888?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:29](http://github.com/urfave/negroni@v0.3.0/negroni.go:29) +0x3c
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x17647c0?, 0x400064f5d8?}, 0x400064f608?}, {0xfffd0abfd4a0, 0x400abf2ae8}, 0xeb64?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/urfave/negroni.(*Recovery).ServeHTTP](http://github.com/urfave/negroni.(*Recovery).ServeHTTP)(0x8?, {0xfffd0abfd4a0?, 0x400abf2ae8?}, 0x0?, 0x400e10e988?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/recovery.go:193](http://github.com/urfave/negroni@v0.3.0/recovery.go:193) +0x78
[github.com/urfave/negroni.middleware.ServeHTTP](http://github.com/urfave/negroni.middleware.ServeHTTP)({{0x1764aa0?, 0x4000093ae0?}, 0x400064f5f0?}, {0xfffd0abfd4a0, 0x400abf2ae8}, 0xb77c00?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:38](http://github.com/urfave/negroni@v0.3.0/negroni.go:38) +0xa8
[github.com/urfave/negroni.(*Negroni).ServeHTTP](http://github.com/urfave/negroni.(*Negroni).ServeHTTP)(0x40003d4120, {0x1770de0?, 0x40036a7340}, 0x40180330e0?)
        /go/pkg/mod/[github.com/urfave/negroni@v0.3.0/negroni.go:96](http://github.com/urfave/negroni@v0.3.0/negroni.go:96) +0x108
net/http.(*ServeMux).ServeHTTP(0x1770de0?, {0x1770de0, 0x40036a7340}, 0x40108f9
@rleungx rleungx added the type/bug The issue is confirmed as a bug. label Dec 27, 2023
@github-actions github-actions bot added this to Need Triage in Questions and Bug Reports Dec 27, 2023
@rleungx rleungx mentioned this issue Dec 27, 2023
Merged
@rleungx rleungx added the severity/major The issue's severity is major. label Dec 28, 2023
@ti-chi-bot ti-chi-bot bot closed this as completed in #7629 Jan 2, 2024
Questions and Bug Reports automation moved this from Need Triage to Closed Jan 2, 2024
ti-chi-bot bot added a commit that referenced this issue Jan 2, 2024
@rleungx @ti-chi-bot
api: fix panic when region doesn't have a leader (#7629)
5bd84c2 
close #7630

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
This was referenced Jan 2, 2024
Merged
Merged
Merged
pingandb pushed a commit to pingandb/pd that referenced this issue Jan 18, 2024
@rleungx @ti-chi-bot @pingandb
api: fix panic when region doesn't have a leader (tikv#7629)
1e97ccf 
close tikv#7630

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: pingandb <songge102@pingan.com.cn>
ti-chi-bot bot pushed a commit that referenced this issue Jan 23, 2024
@ti-chi-bot @rleungx
api: fix panic when region doesn't have a leader (#7629) (#7651)
4506d63 
close #7630

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ryan Leung <rleungx@gmail.com>
@niubell niubell mentioned this issue Jan 30, 2024
Merged
17 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 9, 2024
@ti-chi-bot @rleungx
api: fix panic when region doesn't have a leader (#7629) (#7650)
ae9db49 
close #7630

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ryan Leung <rleungx@gmail.com>
ti-chi-bot bot added a commit that referenced this issue Feb 21, 2024
@ti-chi-bot @rleungx @ti-chi-bot
api: fix panic when region doesn't have a leader (#7629) (#7649)
c7b5055 
close #7630

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ryan Leung <rleungx@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 severity/major The issue's severity is major. type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
  
Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

api: fix panic when region doesn't have a leader rleungx/pd
2 participants
@rleungx @HuSharp