runtime: "panic: runtime error: growslice: cap out of range" when creating slice from an unsafe.Pointer in another part of program

[Lomanic]

What version of Go are you using (go version)?

$ go version
go version go1.13 windows/amd64

Does this issue reproduce with the latest release?

Yes (1.13, also reproducible on 1.12.9)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\user\AppData\Local\go-build
set GOENV=C:\Users\user\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\user\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\user\AppData\Local\Temp\go-build617861358=/tmp/go-build -gno-record-gcc-switches

What did you do?

As we are calling specific win32 API not available on previous versions, Windows 8.1+ is needed to reproduce.

go get -u github.com/shirou/gopsutil/process
cd  "%HOME%\go\src\github.com\shirou\gopsutil\"
git remote add fork https://github.com/Lomanic/gopsutil.git
git checkout fork/issue250-bugreport
go run <path_to_main.go>

main.go

package main

import (
        "fmt"
        "os"
        "time"

        "github.com/shirou/gopsutil/process"
)

func main() {
        start := time.Now()
        procs, err := process.Processes()
        if err != nil {
                fmt.Println("err", err)
                os.Exit(1)
        }
        for _, proc := range procs {
                fmt.Println("PID", proc.Pid)
                fmt.Println(proc.Cmdline())
        }
        elapsed := time.Now().Sub(start)
        fmt.Println("Elapsed time", elapsed, "(", elapsed/(time.Duration(len(procs))), "per process).", len(procs), "processes")
}

What did you expect to see?

The commandline for all (non-elevated) processes on the host. Extract sample using WMI (current implementation in gopsutil master):

PID 4580
"C:\Users\user\Downloads\cmder\vendor\conemu-maximus5\ConEmu64.exe"  <nil>
PID 4616
 <nil>
PID 4652
"C:\Users\user\Downloads\cmder\vendor\conemu-maximus5\ConEmu\ConEmuC64.exe"  /CINMODE=600020 /AID=4636 /GID=4580 /GHWND=00030172 /BW=238 /BH=53 /BZ=1000 "/FN=Lucida Console" /FW=3 /FH=5 /TA=10100007 /HIDE /ROOT cmd /k ""%ConEmuDir%\..\init.bat" " <nil>
PID 4660
\??\C:\Windows\system32\conhost.exe 0x4 <nil>
PID 4756
C:\Users\user\AppData\Local\Temp\go-build614473794\b001\exe\proccmdline.exe <nil>
PID 4760
"C:\Windows\SYSTEM32\cmd.exe"  /k ""C:\Users\user\Downloads\cmder\vendor\conemu-maximus5\..\init.bat" " <nil>
PID 4852
"C:\Windows\SYSTEM32\cmd.exe"  /k ""C:\Users\user\Downloads\cmder\vendor\conemu-maximus5\..\init.bat" " <nil>
PID 4884
 <nil>
PID 4948
go  run proccmdline.go  <nil>
Elapsed time 3.5995457s ( 52.167328ms per process). 69 processes

What did you see instead?

When running as an elevated user, the main.go sample is able to retrieve every possible process commandline it can retrieve like the WMI implementation. The issue occurs with non-elevated user with a panic runtime as explained in shirou/gopsutil#250 (comment).

PID 0
 <nil>
PID 4
 <nil>
PID 68
 <nil>
PID 140
sihost.exe <nil>
PID 184
"C:\Windows\system32\cmd.exe"  <nil>
PID 304
 <nil>
PID 332
 <nil>
PID 392
 <nil>
PID 460
 <nil>
PID 468
 <nil>
PID 528
 <nil>
PID 552
 <nil>
PID 560
 <nil>
PID 672
 <nil>
PID 680
 <nil>
PID 688
 <nil>
PID 696
\??\C:\Windows\system32\conhost.exe 0x4 <nil>
PID 776
 <nil>
PID 784
"C:\Users\user\Downloads\cmder\vendor\conemu-maximus5\ConEmu64.exe"  <nil>
PID 856
panic: runtime error: growslice: cap out of range

goroutine 1 [running]:
fmt.(*buffer).writeString(...)
        c:/go/src/fmt/print.go:82
fmt.(*fmt).padString(0xc000044e10, 0x75006d0045006e, 0x65002e00340036)
        c:/go/src/fmt/format.go:110 +0xfa
fmt.(*fmt).fmtS(0xc000044e10, 0x75006d0045006e, 0x65002e00340036)
        c:/go/src/fmt/format.go:359 +0x68
fmt.(*pp).fmtString(0xc000044dd0, 0x75006d0045006e, 0x65002e00340036, 0xc000000076)
        c:/go/src/fmt/print.go:447 +0x138
fmt.(*pp).printArg(0xc000044dd0, 0x51bfc0, 0xc0000f8300, 0x76)
        c:/go/src/fmt/print.go:698 +0x87e
fmt.(*pp).doPrintln(0xc000044dd0, 0xc000075e80, 0x2, 0x2)
        c:/go/src/fmt/print.go:1173 +0xb9
fmt.Fprintln(0x578480, 0xc000006020, 0xc000075e80, 0x2, 0x2, 0x0, 0x0, 0x0)
        c:/go/src/fmt/print.go:264 +0x5f
fmt.Println(...)
        c:/go/src/fmt/print.go:274
main.main()
        C:/Users/user/go/src/github.com/Lomanic/gopsutiltest/proccmdline.go:25 +0x1b2
exit status 2

Commenting out this line doesn't trigger the panic and hints that the issue resides in the unicodeString.String() function (got it from an UNICODE_STRING implementation in hillu/go-ntdll). What's puzzling is that it works properly running as an elevated user and that the stacktrace looks completely unrelated (in fmt stdlib), the stacktrace appears even for a PID where buf.String() is not even called, hinting about some memory/stack corruption maybe?

[randall77]

Your unicodeString.String method looks unsafe.
The docs of reflect.SliceHeader say:

the Data field is not sufficient to guarantee the data it references will not be garbage collected, so programs must keep a separate, correctly typed pointer to the underlying data.

Are you sure you're doing that?

A safer bet is to do:

s := (*[1e9]uint16)(unsafe.Pointer(u.Buffer))[:u.Length/2]

Then there's no chance a missing write barrier could corrupt the world (like I think might be happening with your code).

[Lomanic]

Thanks a lot for your help.

Unfortunately changing var s []uint16 to s := (*[1e9]uint16)(unsafe.Pointer(u.Buffer))[:u.Length/2] on this line doesn't fix the issue (exact same crash at runtime). Also tried defer runtime.KeepAlive(hdr) (and for s) to no effect.

diff --git a/process/process_windows.go b/process/process_windows.go
index 1abf7ac..3582b72 100644
--- a/process/process_windows.go
+++ b/process/process_windows.go
@@ -7,6 +7,7 @@ import (
        "fmt"
        "os"
        "reflect"
+       "runtime"
        "strings"
        "syscall"
        "time"
@@ -304,8 +305,10 @@ type unicodeString struct {
 }

 func (u unicodeString) String() string {
-       var s []uint16
+       s := (*[1e9]uint16)(unsafe.Pointer(u.Buffer))[:u.Length/2]
+       defer runtime.KeepAlive(s)
        hdr := (*reflect.SliceHeader)(unsafe.Pointer(&s))
+       defer runtime.KeepAlive(hdr)
        hdr.Data = uintptr(unsafe.Pointer(u.Buffer))
        hdr.Len = int(u.Length / 2)
        hdr.Cap = int(u.MaximumLength / 2)

[randall77]

You want just:

func (u unicodeString) String() string {
	s := (*[1e9]uint16)(unsafe.Pointer(u.Buffer))[:u.Length/2]
	return string(utf16.Decode(s))
}

[Lomanic]

Same behavior when redefining the unicodeString.String() function like this unfortunately (and it even still works properly as an elevated user also).

[randall77]

I don't know what the problem might be then.
Probably needs a Windows person to take a look.
@alexbrainman

[alexbrainman]

Probably needs a Windows person to take a look.
@alexbrainman

I don't have time to debug this code.

Sorry.

Alex