Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: panic: runtime error: index out of range #30063

Closed
zhxqgithub opened this issue Feb 2, 2019 · 7 comments
Closed

runtime: panic: runtime error: index out of range #30063

zhxqgithub opened this issue Feb 2, 2019 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Go1.13

Comments

@zhxqgithub
Copy link

zhxqgithub commented Feb 2, 2019 •
edited by ALTree

What version of Go are you using (go version)?

$ go version
go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/zhxq/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/zhxq/go/"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build422435178=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I build kube-proxy on my mathine ,and deliver it to aws virtural mathine which is centos 7(3.18.6-2.el7.centos.x86_64 #1 SMP Mon Oct 24 13:01:33 CST 2016 x86_64 x86_64 x86_64 GNU/Linux),it occur error when run kube-proxy,the errors were below:

output log panic: runtime error: index out of range

goroutine 1 [running]:
reflect.typelinks(0x12b9208, 0xa, 0xb, 0xc420908b00, 0xb, 0x20)
/usr/local/go/src/runtime/runtime1.go:474 +0x3b9
reflect.typesByString(0xc420908b00, 0xb, 0x1, 0x12b9208, 0xa)
/usr/local/go/src/reflect/type.go:1751 +0x34
reflect.(*rtype).ptrTo(0x14eafa0, 0xc420908ba8)
/usr/local/go/src/reflect/type.go:1437 +0xe4
reflect.PtrTo(0x179c460, 0x14eafa0, 0xc42058a400, 0x21ffd30)
/usr/local/go/src/reflect/type.go:1422 +0x3c
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.newUnsafeType(0xc420099610, 0x179c460, 0x14eafa0, 0x15e57a0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_type.go:21 +0x50
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.newUnsafePtrType(0xc420099610, 0x179c460, 0x14eafa0, 0xc420908c88)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_ptr.go:14 +0x43
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).wrapType(0xc420099610, 0x179c460, 0x14eafa0, 0x0, 0x0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:190 +0x1fe
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).Type2(0xc420099610, 0x179c460, 0x14eafa0, 0x0, 0x0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:158 +0xa9
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).TypeOf(0xc420099610, 0x14eafa0, 0xc4206d22c0, 0x46f500, 0x14eafa0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:149 +0xb9
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.TypeOf(0x14eafa0, 0xc4206d22c0, 0x0, 0x0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:208 +0x41
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*Iterator).ReadVal(0xc420011700, 0x14eafa0, 0xc4206d22c0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:66 +0xe6
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc42004a500, 0xc42095c000, 0x182, 0x2f8, 0x14eafa0, 0xc4206d22c0, 0x0, 0x0)
/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/config.go:345 +0xb0

i suspect the len of 'modules' in /usr/local/go/src/runtime/runtime1.go:474 is zero,so i print it:
the result is
modules is [0/0]0x0

Starting program: /usr/bin/kube-proxy --logtostderr=true --v=6 --kubeconfig=/etc/kubernetes/proxy-kubeconfig
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
I0202 16:13:41.003867 10469 flags.go:27] FLAG: --alsologtostderr="false"
I0202 16:13:41.003897 10469 flags.go:27] FLAG: --bind-address="0.0.0.0"
I0202 16:13:41.003902 10469 flags.go:27] FLAG: --cleanup="false"
I0202 16:13:41.003906 10469 flags.go:27] FLAG: --cleanup-iptables="false"
I0202 16:13:41.003909 10469 flags.go:27] FLAG: --cleanup-ipvs="true"
I0202 16:13:41.003912 10469 flags.go:27] FLAG: --cluster-cidr=""
I0202 16:13:41.003916 10469 flags.go:27] FLAG: --config=""
I0202 16:13:41.003919 10469 flags.go:27] FLAG: --config-sync-period="15m0s"
I0202 16:13:41.003926 10469 flags.go:27] FLAG: --conntrack-max="0"
I0202 16:13:41.003930 10469 flags.go:27] FLAG: --conntrack-max-per-core="32768"
I0202 16:13:41.003933 10469 flags.go:27] FLAG: --conntrack-min="131072"
I0202 16:13:41.003936 10469 flags.go:27] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0202 16:13:41.003939 10469 flags.go:27] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0202 16:13:41.003943 10469 flags.go:27] FLAG: --feature-gates=""
I0202 16:13:41.003947 10469 flags.go:27] FLAG: --healthz-bind-address="0.0.0.0:10256"
I0202 16:13:41.003950 10469 flags.go:27] FLAG: --healthz-port="10256"
I0202 16:13:41.003953 10469 flags.go:27] FLAG: --help="false"
I0202 16:13:41.003956 10469 flags.go:27] FLAG: --hostname-override=""
I0202 16:13:41.003960 10469 flags.go:27] FLAG: --iptables-masquerade-bit="14"
I0202 16:13:41.003963 10469 flags.go:27] FLAG: --iptables-min-sync-period="0s"
I0202 16:13:41.003966 10469 flags.go:27] FLAG: --iptables-sync-period="30s"
I0202 16:13:41.003969 10469 flags.go:27] FLAG: --ipvs-min-sync-period="0s"
I0202 16:13:41.003972 10469 flags.go:27] FLAG: --ipvs-scheduler=""
I0202 16:13:41.003975 10469 flags.go:27] FLAG: --ipvs-sync-period="30s"
I0202 16:13:41.003978 10469 flags.go:27] FLAG: --kube-api-burst="10"
I0202 16:13:41.003981 10469 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0202 16:13:41.003984 10469 flags.go:27] FLAG: --kube-api-qps="5"
I0202 16:13:41.003989 10469 flags.go:27] FLAG: --kubeconfig="/etc/kubernetes/proxy-kubeconfig"
I0202 16:13:41.003992 10469 flags.go:27] FLAG: --log-backtrace-at=":0"
I0202 16:13:41.003997 10469 flags.go:27] FLAG: --log-dir=""
I0202 16:13:41.004000 10469 flags.go:27] FLAG: --log-flush-frequency="5s"
I0202 16:13:41.004003 10469 flags.go:27] FLAG: --logtostderr="true"
I0202 16:13:41.004006 10469 flags.go:27] FLAG: --masquerade-all="false"
I0202 16:13:41.004009 10469 flags.go:27] FLAG: --master=""
I0202 16:13:41.004012 10469 flags.go:27] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0202 16:13:41.004016 10469 flags.go:27] FLAG: --nodeport-addresses="[]"
I0202 16:13:41.004021 10469 flags.go:27] FLAG: --oom-score-adj="-999"
I0202 16:13:41.004024 10469 flags.go:27] FLAG: --profiling="false"
I0202 16:13:41.004027 10469 flags.go:27] FLAG: --proxy-mode=""
I0202 16:13:41.004031 10469 flags.go:27] FLAG: --proxy-port-range=""
I0202 16:13:41.004035 10469 flags.go:27] FLAG: --resource-container="/kube-proxy"
I0202 16:13:41.004038 10469 flags.go:27] FLAG: --stderrthreshold="2"
I0202 16:13:41.004041 10469 flags.go:27] FLAG: --udp-timeout="250ms"
I0202 16:13:41.004044 10469 flags.go:27] FLAG: --v="6"
I0202 16:13:41.004047 10469 flags.go:27] FLAG: --version="false"
I0202 16:13:41.004051 10469 flags.go:27] FLAG: --vmodule=""
I0202 16:13:41.004055 10469 flags.go:27] FLAG: --write-config-to=""
W0202 16:13:41.004060 10469 server.go:195] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0202 16:13:41.004088 10469 feature_gate.go:226] feature gates: &{map[]}
Detaching after fork from child process 10494.
Detaching after fork from child process 10495.
I0202 16:13:41.005956 10469 iptables.go:590] couldn't get iptables-restore version; assuming it doesn't support --wait
I0202 16:13:41.006009 10469 conn.go:149] in Dial tr is %s&{0xc4200201d8 false}
I0202 16:13:41.006028 10469 conn.go:150] in Dial err is ====%s
I0202 16:13:41.006034 10469 conn.go:118] in SystemBusPrivate %s&{0xc420941bc0 0xc420855aa0 false [] {{0 0} 0 0 0 0} {0 0} 1 map[0:true] map[] {{0 0} 0 0 0 0} map[] {{0 0} 0 0 0 0} 0xc420363800 false {{0 0} 0 0 0 0} [] {0 0} {0 0}}
I0202 16:13:41.006571 10469 iptables.go:199] Could not connect to D-Bus system bus: runtime error: index out of range
[New LWP 10492]
Detaching after fork from child process 10496.

modules is [0/0]0x0

panic: runtime error: index out of range

I did the same work on my another server which was a virtual mathine(3.18.6-2.el7.centos.x86_64 #1 SMP Mon Oct 24 13:01:33 CST 2016 x86_64 x86_64 x86_64 GNU/Linux) , however the kube-proxy can work well:
so,is there any difference between on aws machine and mine(has the same os)? or the goland need some moudle which must installed???

gdb kube-proxy
set args --logtostderr=true --v=6 --kubeconfig=/etc/kubernetes/proxy-kubeconfig
run

Starting program: /home/zhxq/./kube-proxy --logtostderr=true --v=6 --kubeconfig=/etc/kubernetes/proxy-kubeconfig
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
I0202 16:15:26.889983 11754 flags.go:27] FLAG: --alsologtostderr="false"
I0202 16:15:26.890082 11754 flags.go:27] FLAG: --bind-address="0.0.0.0"
I0202 16:15:26.890093 11754 flags.go:27] FLAG: --cleanup="false"
I0202 16:15:26.890102 11754 flags.go:27] FLAG: --cleanup-iptables="false"
I0202 16:15:26.890109 11754 flags.go:27] FLAG: --cleanup-ipvs="true"
I0202 16:15:26.890115 11754 flags.go:27] FLAG: --cluster-cidr=""
I0202 16:15:26.890123 11754 flags.go:27] FLAG: --config=""
I0202 16:15:26.890129 11754 flags.go:27] FLAG: --config-sync-period="15m0s"
I0202 16:15:26.890137 11754 flags.go:27] FLAG: --conntrack-max="0"
I0202 16:15:26.890145 11754 flags.go:27] FLAG: --conntrack-max-per-core="32768"
I0202 16:15:26.890155 11754 flags.go:27] FLAG: --conntrack-min="131072"
I0202 16:15:26.890162 11754 flags.go:27] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0202 16:15:26.890169 11754 flags.go:27] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0202 16:15:26.890176 11754 flags.go:27] FLAG: --feature-gates=""
I0202 16:15:26.890184 11754 flags.go:27] FLAG: --healthz-bind-address="0.0.0.0:10256"
I0202 16:15:26.890191 11754 flags.go:27] FLAG: --healthz-port="10256"
I0202 16:15:26.890198 11754 flags.go:27] FLAG: --help="false"
I0202 16:15:26.890204 11754 flags.go:27] FLAG: --hostname-override=""
I0202 16:15:26.890211 11754 flags.go:27] FLAG: --iptables-masquerade-bit="14"
I0202 16:15:26.890217 11754 flags.go:27] FLAG: --iptables-min-sync-period="0s"
I0202 16:15:26.890224 11754 flags.go:27] FLAG: --iptables-sync-period="30s"
I0202 16:15:26.890231 11754 flags.go:27] FLAG: --ipvs-min-sync-period="0s"
I0202 16:15:26.890237 11754 flags.go:27] FLAG: --ipvs-scheduler=""
I0202 16:15:26.890243 11754 flags.go:27] FLAG: --ipvs-sync-period="30s"
I0202 16:15:26.890250 11754 flags.go:27] FLAG: --kube-api-burst="10"
I0202 16:15:26.890256 11754 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0202 16:15:26.890263 11754 flags.go:27] FLAG: --kube-api-qps="5"
I0202 16:15:26.890272 11754 flags.go:27] FLAG: --kubeconfig="/etc/kubernetes/proxy-kubeconfig"
I0202 16:15:26.890279 11754 flags.go:27] FLAG: --log-backtrace-at=":0"
I0202 16:15:26.890288 11754 flags.go:27] FLAG: --log-dir=""
I0202 16:15:26.890295 11754 flags.go:27] FLAG: --log-flush-frequency="5s"
I0202 16:15:26.890302 11754 flags.go:27] FLAG: --logtostderr="true"
I0202 16:15:26.890309 11754 flags.go:27] FLAG: --masquerade-all="false"
I0202 16:15:26.890316 11754 flags.go:27] FLAG: --master=""
I0202 16:15:26.890322 11754 flags.go:27] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0202 16:15:26.890329 11754 flags.go:27] FLAG: --nodeport-addresses="[]"
I0202 16:15:26.890340 11754 flags.go:27] FLAG: --oom-score-adj="-999"
I0202 16:15:26.890346 11754 flags.go:27] FLAG: --profiling="false"
I0202 16:15:26.890359 11754 flags.go:27] FLAG: --proxy-mode=""
I0202 16:15:26.890369 11754 flags.go:27] FLAG: --proxy-port-range=""
I0202 16:15:26.890376 11754 flags.go:27] FLAG: --resource-container="/kube-proxy"
I0202 16:15:26.890383 11754 flags.go:27] FLAG: --stderrthreshold="2"
I0202 16:15:26.890390 11754 flags.go:27] FLAG: --udp-timeout="250ms"
I0202 16:15:26.890397 11754 flags.go:27] FLAG: --v="6"
I0202 16:15:26.890404 11754 flags.go:27] FLAG: --version="false"
I0202 16:15:26.890412 11754 flags.go:27] FLAG: --vmodule=""
I0202 16:15:26.890421 11754 flags.go:27] FLAG: --write-config-to=""
W0202 16:15:26.890435 11754 server.go:195] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0202 16:15:26.890541 11754 feature_gate.go:226] feature gates: &{map[]}
Detaching after fork from child process 11768.
Detaching after fork from child process 11769.
I0202 16:15:26.893229 11754 iptables.go:590] couldn't get iptables-restore version; assuming it doesn't support --wait
[New LWP 11763]
Detaching after fork from child process 11770.
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
modules is [1/1]0xc42000c000
I0202 16:15:26.900709 11754 loader.go:357] Config loaded from file /etc/kubernetes/proxy-kubeconfig
W0202 16:15:26.901570 11754 server_others.go:290] Can't use ipvs proxier, trying iptables proxier
Detaching after fork from child process 11771.
I0202 16:15:26.902698 11754 server_others.go:140] Using iptables Proxier.

What did you expect to see?

the kube-proxy can work well

What did you see instead?

panic: runtime error: index out of range
@agnivade agnivade changed the title runtime1.go:474 cause panic runtime error: index out of range runtime: panic: runtime error: index out of range Feb 2, 2019
@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 2, 2019
@agnivade agnivade added this to the Go1.13 milestone Feb 2, 2019
@agnivade
Copy link
Contributor

agnivade commented Feb 2, 2019

/cc @aclements @mknyszek

@randall77
Copy link
Contributor

This doesn't make much sense. There should always be at least one module.
Is the plugin package used by your program?

Can you add a few more prints:

  1. Print something at the start of modulesinit (it is in src/runtime/symtab.go).
  2. Print something each time through the first loop in modulesinit.
  3. Print something in the body of the if md.bad statement.

That will let us know if the data is computed incorrectly to start, or whether it gets lost subsequently.

The modules slice lives on the heap, so possibly the modules slice gets collected incorrectly by the GC and overwritten. Seems unlikely, but options to explain this are limited...

@zhxqgithub
Copy link
Author

@randall77 thanks for your reply for me.
I have add prints in runtime1.go and symtab.go

runtime1.go

func reflect_typelinks() ([]unsafe.Pointer, [][]int32) {
	modules := activeModules()
	println("in reflect_typelinks active modules is ", modules)
	sections := []unsafe.Pointer{unsafe.Pointer(modules[0].types)}
	ret := [][]int32{modules[0].typelinks}
	for _, md := range modules[1:] {
		sections = append(sections, unsafe.Pointer(md.types))
		ret = append(ret, md.typelinks)
	}
	return sections, ret
}

symtab.go

func activeModules() []*moduledata {
	p := (*[]*moduledata)(atomic.Loadp(unsafe.Pointer(&modulesSlice)))
	println("in activeModules *p is", *p)
	if p == nil {
		return nil
	}
	return *p
}

// modulesinit creates the active modules slice out of all loaded modules.
//
// When a module is first loaded by the dynamic linker, an .init_array
// function (written by cmd/link) is invoked to call addmoduledata,
// appending to the module to the linked list that starts with
// firstmoduledata.
//
// There are two times this can happen in the lifecycle of a Go
// program. First, if compiled with -linkshared, a number of modules
// built with -buildmode=shared can be loaded at program initialization.
// Second, a Go program can load a module while running that was built
// with -buildmode=plugin.
//
// After loading, this function is called which initializes the
// moduledata so it is usable by the GC and creates a new activeModules
// list.
//
// Only one goroutine may call modulesinit at a time.
func modulesinit() {
	modules := new([]*moduledata)
	for md := &firstmoduledata; md != nil; md = md.next {
		println("in modulesinit loop, the md is ", md)
		if md.bad {
			println("in modulesinit md.bad, the md is ", md)
			continue
		}
		*modules = append(*modules, md)
		if md.gcdatamask == (bitvector{}) {
			md.gcdatamask = progToPointerMask((*byte)(unsafe.Pointer(md.gcdata)), md.edata-md.data)
			md.gcbssmask = progToPointerMask((*byte)(unsafe.Pointer(md.gcbss)), md.ebss-md.bss)
		}
	}

Here was the print results:

kube-proxy --logtostderr=true --v=6 --kubeconfig=/etc/kubernetes/proxy-kubeconfig


in modulesinit loop, the md is  0x2088760
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in reflect_typelinks active modules is  [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
in activeModules *p is [1/1]0xc42000c000
I0212 10:07:14.211467   17594 flags.go:27] FLAG: --alsologtostderr="false"
I0212 10:07:14.211510   17594 flags.go:27] FLAG: --bind-address="0.0.0.0"
I0212 10:07:14.211516   17594 flags.go:27] FLAG: --cleanup="false"
I0212 10:07:14.211521   17594 flags.go:27] FLAG: --cleanup-iptables="false"
I0212 10:07:14.211524   17594 flags.go:27] FLAG: --cleanup-ipvs="true"
I0212 10:07:14.211527   17594 flags.go:27] FLAG: --cluster-cidr=""
I0212 10:07:14.211531   17594 flags.go:27] FLAG: --config=""
I0212 10:07:14.211534   17594 flags.go:27] FLAG: --config-sync-period="15m0s"
I0212 10:07:14.211538   17594 flags.go:27] FLAG: --conntrack-max="0"
I0212 10:07:14.211542   17594 flags.go:27] FLAG: --conntrack-max-per-core="32768"
I0212 10:07:14.211545   17594 flags.go:27] FLAG: --conntrack-min="131072"
I0212 10:07:14.211548   17594 flags.go:27] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0212 10:07:14.211552   17594 flags.go:27] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0212 10:07:14.211555   17594 flags.go:27] FLAG: --feature-gates=""
I0212 10:07:14.211560   17594 flags.go:27] FLAG: --healthz-bind-address="0.0.0.0:10256"
I0212 10:07:14.211563   17594 flags.go:27] FLAG: --healthz-port="10256"
I0212 10:07:14.211566   17594 flags.go:27] FLAG: --help="false"
I0212 10:07:14.211569   17594 flags.go:27] FLAG: --hostname-override=""
I0212 10:07:14.211572   17594 flags.go:27] FLAG: --iptables-masquerade-bit="14"
I0212 10:07:14.211575   17594 flags.go:27] FLAG: --iptables-min-sync-period="0s"
I0212 10:07:14.211578   17594 flags.go:27] FLAG: --iptables-sync-period="30s"
I0212 10:07:14.211582   17594 flags.go:27] FLAG: --ipvs-min-sync-period="0s"
I0212 10:07:14.211585   17594 flags.go:27] FLAG: --ipvs-scheduler=""
I0212 10:07:14.211588   17594 flags.go:27] FLAG: --ipvs-sync-period="30s"
I0212 10:07:14.211591   17594 flags.go:27] FLAG: --kube-api-burst="10"
I0212 10:07:14.211594   17594 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0212 10:07:14.211597   17594 flags.go:27] FLAG: --kube-api-qps="5"
I0212 10:07:14.211602   17594 flags.go:27] FLAG: --kubeconfig="/etc/kubernetes/proxy-kubeconfig"
I0212 10:07:14.211605   17594 flags.go:27] FLAG: --log-backtrace-at=":0"
I0212 10:07:14.211610   17594 flags.go:27] FLAG: --log-dir=""
I0212 10:07:14.211613   17594 flags.go:27] FLAG: --log-flush-frequency="5s"
I0212 10:07:14.211617   17594 flags.go:27] FLAG: --logtostderr="true"
I0212 10:07:14.211620   17594 flags.go:27] FLAG: --masquerade-all="false"
I0212 10:07:14.211623   17594 flags.go:27] FLAG: --master=""
I0212 10:07:14.211626   17594 flags.go:27] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0212 10:07:14.211629   17594 flags.go:27] FLAG: --nodeport-addresses="[]"
I0212 10:07:14.211634   17594 flags.go:27] FLAG: --oom-score-adj="-999"
I0212 10:07:14.211638   17594 flags.go:27] FLAG: --profiling="false"
I0212 10:07:14.211641   17594 flags.go:27] FLAG: --proxy-mode=""
I0212 10:07:14.211645   17594 flags.go:27] FLAG: --proxy-port-range=""
I0212 10:07:14.211648   17594 flags.go:27] FLAG: --resource-container="/kube-proxy"
I0212 10:07:14.211652   17594 flags.go:27] FLAG: --stderrthreshold="2"
I0212 10:07:14.211654   17594 flags.go:27] FLAG: --udp-timeout="250ms"
I0212 10:07:14.211658   17594 flags.go:27] FLAG: --v="6"
I0212 10:07:14.211661   17594 flags.go:27] FLAG: --version="false"
I0212 10:07:14.211665   17594 flags.go:27] FLAG: --vmodule=""
I0212 10:07:14.211669   17594 flags.go:27] FLAG: --write-config-to=""
W0212 10:07:14.211674   17594 server.go:195] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0212 10:07:14.211710   17594 feature_gate.go:226] feature gates: &{map[]}
I0212 10:07:14.213204   17594 iptables.go:589] couldn't get iptables-restore version; assuming it doesn't support --wait
in activeModules *p is [0/0]0x0
in reflect_typelinks active modules is  [0/0]0x0
I0212 10:07:14.213778   17594 iptables.go:198] Could not connect to D-Bus system bus: runtime error: index out of range
in activeModules *p is [0/0]0x0
in reflect_typelinks active modules is  [0/0]0x0
panic: runtime error: index out of range

goroutine 1 [running]:
reflect.typelinks(0x12b9208, 0xa, 0xb, 0xc4208b8b80, 0xb, 0x20)
	/usr/local/go/src/runtime/runtime1.go:475 +0x48f
reflect.typesByString(0xc4208b8b80, 0xb, 0x1, 0x12b9208, 0xa)
	/usr/local/go/src/reflect/type.go:1751 +0x34
reflect.(*rtype).ptrTo(0x14eafa0, 0xc4208b8c28)
	/usr/local/go/src/reflect/type.go:1437 +0xe4
reflect.PtrTo(0x179c0c0, 0x14eafa0, 0xc4208c5800, 0x21ffd30)
	/usr/local/go/src/reflect/type.go:1422 +0x3c
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.newUnsafeType(0xc420099600, 0x179c0c0, 0x14eafa0, 0x15e57a0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_type.go:21 +0x50
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.newUnsafePtrType(0xc420099600, 0x179c0c0, 0x14eafa0, 0xc4208b8d08)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_ptr.go:14 +0x43
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).wrapType(0xc420099600, 0x179c0c0, 0x14eafa0, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:190 +0x1fe
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).Type2(0xc420099600, 0x179c0c0, 0x14eafa0, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:158 +0xa9
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*frozenConfig).TypeOf(0xc420099600, 0x14eafa0, 0xc4208822c0, 0x46fc00, 0x14eafa0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:149 +0xb9
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.TypeOf(0x14eafa0, 0xc4208822c0, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/reflect2.go:208 +0x41
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*Iterator).ReadVal(0xc420554f00, 0x14eafa0, 0xc4208822c0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:66 +0xe6
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc42004a500, 0xc42090e000, 0x182, 0x2f8, 0x14eafa0, 0xc4208822c0, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/config.go:345 +0xb0
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc420416f40, 0xc4206fa000, 0x16a, 0x36a, 0xc4207c8f00, 0x176d0a0, 0xc4200a4900, 0x28, 0x30, 0xc4207c8f00, ...)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:203 +0x740
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Decode(0xc42041c580, 0xc4206fa000, 0x16a, 0x36a, 0xc4207c8f00, 0x176d0a0, 0xc4200a4900, 0x0, 0xc4208b9280, 0x5c5493, ...)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:98 +0xd6
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.Load(0xc4206fa000, 0x16a, 0x36a, 0x16a, 0x36a, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:394 +0x1c7
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.LoadFromFile(0x7fff2163e6fa, 0x20, 0x0, 0x0, 0x1)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:353 +0x85
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.(*ClientConfigLoadingRules).Load(0xc4200b4a00, 0x16bd100, 0xc4200b4ad8, 0xc420000180)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:195 +0x160
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.(*DeferredLoadingClientConfig).createClientConfig(0xc4200b4aa0, 0x0, 0x0, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/merged_client_builder.go:69 +0xf8
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.(*DeferredLoadingClientConfig).ClientConfig(0xc4200b4aa0, 0xc4200b4aa0, 0xc4208b9860, 0x126138a)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/merged_client_builder.go:99 +0x2f
k8s.io/kubernetes/cmd/kube-proxy/app.createClients(0x7fff2163e6fa, 0x20, 0x0, 0x0, 0x164745d, 0x23, 0xa40a00000, 0x0, 0x0, 0x0, ...)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:416 +0x171
k8s.io/kubernetes/cmd/kube-proxy/app.newProxyServer(0xc42048a600, 0x8030100, 0xc420417280, 0x0, 0x0, 0x222d290, 0x222d290, 0xc4208b9c28)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server_others.go:110 +0x49a
k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyServer(0xc420882000, 0x0, 0x0, 0x16bac80)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server_others.go:58 +0x5a
k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0xc420882000, 0x0, 0x0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:234 +0x38
k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0xc4206a0000, 0xc4207c8360, 0x0, 0x3)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:357 +0xc5
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc4206a0000, 0xc420038090, 0x3, 0x3, 0xc4206a0000, 0xc420038090)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:757 +0x2c1
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4206a0000, 0xc4207c82a0, 0x16bb470, 0xc4208b9ee8)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:843 +0x30a
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4206a0000, 0x16bc498, 0x220e9c0)
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:791 +0x2b
main.main()
	/home/zhxq/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:49 +0x196

@agnivade
Copy link
Contributor

agnivade commented Mar 8, 2019

ping @randall77

@randall77
Copy link
Contributor

The modules slice is getting corrupted somehow. I don't see how that's possible. The rest of this post is grasping at straws...

Bug in GC?
Maybe the write barrier in atomicstorep isn't right, it is not well-used code. Try adding at the end of modulesinit:

func modulesinit() {
   ...
   atomicstorep(unsafe.Pointer(&modulesSlice), unsafe.Pointer(modules))
   sink = modules
}
var sink *[]*moduledata

That will provide another GC root that might keep the modules slice alive.

You might also try 1.11 and 1.12, to see if that affects anything.

@zhxqgithub
Copy link
Author

@agnivade @randall77 thanks ,I will have a try.

@agnivade agnivade added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 19, 2019
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Apr 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Go1.13
Development

No branches or pull requests
4 participants
@agnivade @randall77 @zhxqgithub @gopherbot