Sometimes got "panic: runtime error: invalid memory address or nil pointer dereference" #21065

369pie · 2021-03-25T07:17:58Z

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

execute "SHOW TAG VALUES CARDINALITY WITH KEY = "myTagKey""

Expected behavior:
show tag values CARDINALITY

Actual behavior:
influxdb process got crashed

Environment info:
uname -a
Linux ip-10-15-29-164.ap-southeast-1.compute.internal 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
influxdb version: 1.8.2

Logs:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x92e790]

goroutine 306157422 [running]:
github.com/influxdata/influxdb/query.Iterators.Close(0xc1065da000, 0x28, 0x161, 0xc0f8196aa0, 0x255ac00)
/go/src/github.com/influxdata/influxdb/query/iterator.go:48 +0x50
github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator(0xc3e9a9e040, 0x25490a0, 0xc22597a420, 0xc0f8196aa0, 0x253e320, 0xc1e0a19840, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/influxdata/influxdb/coordinator/shard_mapper.go:202 +0x33a
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildVarRefIterator.func1(0xc0fe6e4ab8, 0x25490a0, 0xc22597a420, 0xc0fe6e4940, 0xc1e0a19840, 0x10000c2ccd697f8, 0x0)
/go/src/github.com/influxdata/influxdb/query/select.go:176 +0x15c
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildVarRefIterator(0xc0fe6e4ab8, 0x25490a0, 0xc22597a420, 0xc1e0a19840, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/query/select.go:196 +0xd2
github.com/influxdata/influxdb/query.buildExprIterator(0x25490a0, 0xc22597a420, 0x253e320, 0xc1e0a19840, 0x7daa7d156688, 0xc3e9a9e040, 0xc04a754840, 0x1, 0x1, 0x253e320, ...) /go/src/github.com/influxdata/influxdb/query/select.go:154 +0x27dgithub.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator(0xc003447a80, 0x25490a0, 0xc22597a420, 0xc22597a540, 0x20303900000000, 0x7ef89656afff,0x7ef78000a790, 0x60010000000000) /go/src/github.com/influxdata/influxdb/query/select.go:222 +0x2c47
github.com/influxdata/influxdb/query.buildExprIterator(0x25490a0, 0xc22597a420, 0x253dfe0, 0xc22597a540, 0x7daa7d156688, 0xc3e9a9e040, 0xc04a754840, 0x1, 0x1, 0x253dfe0, ...) /go/src/github.com/influxdata/influxdb/query/select.go:156 +0x1f4github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator.func1(0xc22597a570, 0x25490a0, 0xc22597a420, 0xc0034498f0, 0xc0fe6e6dd0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/query/select.go:506 +0x5f4
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator(0xc0fe6e78f0, 0x25490a0, 0xc22597a420, 0xc22597a570, 0xc1050e28f0, 0x904368, 0xc0e7ac6a00, 0x0) /go/src/github.com/influxdata/influxdb/query/select.go:559 +0xd3egithub.com/influxdata/influxdb/query.buildExprIterator(0x25490a0, 0xc22597a420, 0x253dfe0, 0xc22597a570, 0x7daa7d156688, 0xc3e9a9e040, 0xc04a754840, 0x1, 0x1, 0x253dfe0, ...) /go/src/github.com/influxdata/influxdb/query/select.go:156 +0x1f4github.com/influxdata/influxdb/query.buildFieldIterator(0x25490a0, 0xc22597a420, 0x253dfe0, 0xc22597a570, 0x7daa7d156688, 0xc3e9a9e040, 0xc04a754840, 0x1, 0x1, 0x0,...) /go/src/github.com/influxdata/influxdb/query/select.go:870 +0x489
github.com/influxdata/influxdb/query.buildCursor.func1(0xc10980cf68, 0x0) /go/src/github.com/influxdata/influxdb/query/select.go:744 +0x127
github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc22597a5a0, 0xc18dec7d50)
/go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go
/go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

hongjijun233 · 2021-03-25T10:33:24Z

If there is an error in creating an Iterator, the input return by sg.CreateIterator(ctx, mm, opt) will be nil.

however, when close the Iterator, There is no operation to check the status, so there is a panic for close()

for more detail, please read this PR

：）

MilanJaro · 2021-03-26T07:48:00Z

Maybe related to this issue - adding the two recent panic samples from another environment:
ubuntu 20 LTS
uname -a
Linux cv38x114 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Influx version 1.8.4

ts=2021-03-18T08:16:31.809826Z lvl=error msg="SELECT value FROM dbinf202.autogen./^.*$/ WHERE (cluster_name =~ /^k8s-nprod-0324$/ AND \"group\" = 'STORAGE' AND value > 0) AND time >= now() - 1h GROUP BY \"group\", metric, quantile
[panic:runtime error: invalid memory address or nil pointer dereference] goroutine 1286130847 [running]:\n
runtime/debug.Stack(0xd4690e9390, 0x1, 0x1)\n\t
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d\n
github.com/influxdata/influxdb/query.(*Executor).recover(0xc0002fdc20, 0xd00b9d1f60, 0xd1c127c2a0)\n\t
    /go/src/github.com/influxdata/influxdb/query/executor.go:395 +0xbf\n
panic(0x144e700, 0x31ca820)\n\t
/usr/local/go/src/runtime/panic.go:679 +0x1b2\n
github.com/influxdata/influxdb/query.Iterators.Close(0xc4adb98000, 0x99b, 0x15c4, 0xc1015ac4b0, 0x259e6e0)\n\t
    /go/src/github.com/influxdata/influxdb/query/iterator.go:48 +0x50\n
github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator(0xd1198b5a80, 0x258cae0, 0xc333888e70, 0xc1015ac4b0, 0x0, 0x0, 0xce5a7b1440, 0x1, 0x1, 0x0, ...)\n\t
    /go/src/github.com/influxdata/influxdb/coordinator/shard_mapper.go:202 +0x33a\n
github.com/influxdata/influxdb/query.buildAuxIterator.func1(0xd4690e9e70, 0x1, 0x1, 0x7f74762a3680, 0xd1198b5a80, 0x258cae0, 0xc333888e70, 0xc10dad4bc8, 0xc10dad4b48, 0x7f7b736c6160, ...)\n\t
    /go/src/github.com/influxdata/influxdb/query/select.go:798 +0x15e\n
github.com/influxdata/influxdb/query.buildAuxIterator(0x258cae0, 0xc333888e70, 0x7f74762a3680, 0xd1198b5a80, 0xd4690e9e70, 0x1, 0x1, 0x0, 0x0, 0xce5a7b1440, ...)\n\t
    /go/src/github.com/influxdata/influxdb/query/select.go:818 +0x3f2\n
github.com/influxdata/influxdb/query.buildCursor(0x258cae0, 0xc333888e70, 0xc6059e6300, 0x7f74762a3680, 0xd1198b5a80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t
    /go/src/github.com/influxdata/influxdb/query/select.go:704 +0x14e2\n
github.com/influxdata/influxdb/query.(*preparedStatement).Select(0xe555a1f600, 0x258cca0, 0xc0f6b74300, 0x7f1d94438008, 0xd4e50eb8f0, 0x0, 0x0)\n\t
    /go/src/github.com/influxdata/influxdb/query/select.go:121 +0x180\n
github.com/influxdata/influxdb/query.Select(0x258cca0, 0xc0f6b74300, 0xc405b6da00, 0x25552c0, 0xc0002e9700, 0x7f1d94438008, 0xd4e50eb8f0, 0x0, 0x0, 0x0, ...)\n\t
    /go/src/github.com/influxdata/influxdb/query/select.go:98 +0x11f\n
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).createIterators(0xc00023f340, 0x258cca0, 0xc0f6b74300, 0xc405b6da00, 0xd037f9412e, 0x8, 0x0, 0x0, 0x7f1d94438008, 0xd4e50eb8f0, ...)\n\t
/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:638 +0x113\n
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement(0xc00023f340, 0xc405b6da00, 0xc0f6b74300, 0x0, 0x0)\n\t
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:542 +0xce\n
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc00023f340, 0x258dee0, 0xc405b6da00, 0xc0f6b74300, 0x1, 0x1)\n\t
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:64 +0x38d3\n
github.com/influxdata/influxdb/query.(*Executor).executeQuery(0xc0002fdc20, 0xd00b9d1f60, 0xd037f9412e, 0x8, 0x0, 0x0, 0x7f1d94438008, 0xd4e50eb8f0, 0x2710, 0x1, ...)\n\t
    /go/src/github.com/influxdata/influxdb/query/executor.go:335 +0x3ae\n
created by github.com/influxdata/influxdb/query.(*Executor).ExecuteQuery\n\t
   /go/src/github.com/influxdata/influxdb/query/executor.go:237 +0xc9\n"

ts=2021-03-26T07:06:52.987121Z lvl=info msg="Executing query" log_id=0T62RXJ0000 service=query query="SELECT last(value) FROM dbinf202.autogen.k8s_kb_jvm_memory_used_percent WHERE time > now() - 15m AND eamApplication =~ /^UnifiedTask
Queue$/ GROUP BY cluster_name, kubernetes_namespace, eamApplication"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x961750]
goroutine 534031480 [running]:
github.com/influxdata/influxdb/query.Iterators.Close(0xc003fa2000, 0x503, 0x15d3, 0xd68a4261e0, 0x259e6e0)
        /go/src/github.com/influxdata/influxdb/query/iterator.go:48 +0x50
github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator(0xd68a423ac0, 0x258cae0, 0xca607cc4b0, 0xd68a4261e0, 0x2581aa0, 0xca607cc5d0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/influxdata/influxdb/coordinator/shard_mapper.go:202 +0x33a
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).callIterator.func1(0xc0017798f0, 0x258cae0, 0xca607cc4b0, 0xc001777be0, 0xc001777b98, 0xca607cc5d0, 0xd5dce79ab8, 0x971ef2)
        /go/src/github.com/influxdata/influxdb/query/select.go:583 +0x15e
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).callIterator(0xd5dce7b8f0, 0x258cae0, 0xca607cc4b0, 0xca607cc5d0, 0x2581aa0, 0xca607cc5d0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/influxdata/influxdb/query/select.go:608 +0xdf
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator.func1(0xca607cc5d0, 0x258cae0, 0xca607cc4b0, 0xd5dce7b8f0, 0xd5dce7add0, 0x75757d796d7f5fff, 0xdffcadfdfda7f7d3, 0xbfbffe3efafeffef, 0xfefffffffbfbfbff)
        /go/src/github.com/influxdata/influxdb/query/select.go:515 +0xf5
github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator(0xc0017798f0, 0x258cae0, 0xca607cc4b0, 0xca607cc5d0, 0x0, 0x42e80c, 0x35f5020, 0x100000081)
        /go/src/github.com/influxdata/influxdb/query/select.go:559 +0xd3e
github.com/influxdata/influxdb/query.buildExprIterator(0x258cae0, 0xca607cc4b0, 0x2581aa0, 0xca607cc5d0, 0x7f6d949948d8, 0xd68a423ac0, 0xd68a420eb0, 0x1, 0x1, 0x2581aa0, ...)
        /go/src/github.com/influxdata/influxdb/query/select.go:156 +0x1f4
github.com/influxdata/influxdb/query.buildFieldIterator(0x258cae0, 0xca607cc4b0, 0x2581aa0, 0xca607cc5d0, 0x7f6d949948d8, 0xd68a423ac0, 0xd68a420eb0, 0x1, 0x1, 0x0, ...)
        /go/src/github.com/influxdata/influxdb/query/select.go:870 +0x489
github.com/influxdata/influxdb/query.buildCursor.func1(0xd70c45a768, 0x0)
        /go/src/github.com/influxdata/influxdb/query/select.go:744 +0x127
github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xca607cc600, 0xd2f248e000)
        /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go

369pie · 2021-03-31T04:09:41Z

If there is an error in creating an Iterator, the input return by sg.CreateIterator(ctx, mm, opt) will be nil.

however, when close the Iterator, There is no operation to check the status, so there is a panic for close()

for more detail, please read this PR

：）

i think this solution would be fine

timhallinflux · 2021-11-12T20:34:01Z

we are guiding community users to InfluxDB 2.1 -- this issue is resolved there. If you run into issues upgrading, please let us know.

hongjijun233 mentioned this issue Mar 25, 2021

Bugfix/21065 #21066

Closed

1 task

This was referenced Mar 25, 2021

bugfix: fix nil pointer dereference in shard_mapper #21067

Closed

fix(shard_mapper): fix nil pointer dereference in creating an Iterator #21068

Open

danxmoran added 1.x panic labels Apr 9, 2021

timhallinflux closed this as completed Nov 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sometimes got "panic: runtime error: invalid memory address or nil pointer dereference" #21065

Sometimes got "panic: runtime error: invalid memory address or nil pointer dereference" #21065

369pie commented Mar 25, 2021 •

edited

Loading

hongjijun233 commented Mar 25, 2021 •

edited

Loading

MilanJaro commented Mar 26, 2021 •

edited

Loading

369pie commented Mar 31, 2021

timhallinflux commented Nov 12, 2021

Sometimes got "panic: runtime error: invalid memory address or nil pointer dereference" #21065

Sometimes got "panic: runtime error: invalid memory address or nil pointer dereference" #21065

Comments

369pie commented Mar 25, 2021 • edited Loading

hongjijun233 commented Mar 25, 2021 • edited Loading

MilanJaro commented Mar 26, 2021 • edited Loading

369pie commented Mar 31, 2021

timhallinflux commented Nov 12, 2021

369pie commented Mar 25, 2021 •

edited

Loading

hongjijun233 commented Mar 25, 2021 •

edited

Loading

MilanJaro commented Mar 26, 2021 •

edited

Loading