x/sys/unix: invalid data returned by (syscall|unix).Sysctl() on (at least) FreeBSD 11 (amd64) and NetBSD 7 (i386)

[fuzzy]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

> go version
go version go1.7.4 freebsd/amd64

and

# go version
go version go1.7.4 netbsd/386

What operating system and processor architecture are you using (go env)?

> go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/usr/local/golang/gopath"
GORACE=""
GOROOT="/usr/local/golang/goroot"
GOTOOLDIR="/usr/local/golang/goroot/pkg/tool/freebsd_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build425403264=/tmp/go-build -gno-record-gcc-switches"
CXX="clang++"
CGO_ENABLED="1"

and

# go env
GOARCH="386"
GOBIN=""
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="netbsd"
GOOS="netbsd"
GOPATH="/usr/gopath"
GORACE=""
GOROOT="/usr/pkg/go"
GOTOOLDIR="/usr/pkg/go/pkg/tool/netbsd_386"
CC="gcc"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build860612224=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Using syscall.Sysctl(name string) (value string, err error) to get any string type value works fine, for example, kern.ostype will return "FreeBSD" or "NetBSD", however tyring to read a numerical value breaks, but does not error. such as hw.physmem which should return me (if things were converted properly) something like "2138501120" but instead gives me a 4 char string, with int values of 0 240 118 127, which is pretty clearly invalid. Perhaps syscall.Sysctl() should return an interface{} and let the user worry about type safety.

https://play.golang.org/p/wbc-OZGw-P for a quick freebsd test
https://play.golang.org/p/PcuDNWnvFz for a quick netbsd test

NOTE: I added "kern.ostype" to the netbsd test to show that the issues doesn't happen when reading a value containing characters, only when reading a numerical value.

What did you expect to see?

A (lets say for example) int64 that's been converted to a string, not unprintable characters.

What did you see instead?

Unprintable characters instead of a valid string. see below for the output of the NetBSD test:

$ cat test.go
package main

import (
	"fmt"
	"syscall"
)

func main() {
	var data []string
	data = []string{"hw.physmem", "hw.usermem", "kern.ostype"}
	for _, s := range data {
		if os, er := syscall.Sysctl(s); er != nil {
			panic(er)
		} else {
			fmt.Println("DEBUG:", os)
			for i := 0; i < len(os); i++ {
				fmt.Println("DEBUG: index:", i, "int:", os[i], "str:", string(os[i]))
			}
		}
	}
}


$ go build test.go
$ ./test
DEBUG: �v
DEBUG: index: 0 int: 0 str: 
DEBUG: index: 1 int: 240 str: ð
DEBUG: index: 2 int: 118 str: v
DEBUG: index: 3 int: 127 str: 
DEBUG: p4
DEBUG: index: 0 int: 0 str: 
DEBUG: index: 1 int: 112 str: p
DEBUG: index: 2 int: 52 str: 4
DEBUG: index: 3 int: 127 str: 
DEBUG: NetBSD
DEBUG: index: 0 int: 78 str: N
DEBUG: index: 1 int: 101 str: e
DEBUG: index: 2 int: 116 str: t
DEBUG: index: 3 int: 66 str: B
DEBUG: index: 4 int: 83 str: S
DEBUG: index: 5 int: 68 str: D
$

[bradfitz]

I only skimmed this bug report, but wanted to say that any fix probably belongs in https://godoc.org/golang.org/x/sys/unix instead of the syscall package, which is frozen.

[bradfitz]

/cc @mdempsky

[fuzzy]

fair enough, but for the record, it happens with golang.org/x/sys/unix as well. I'll file a second report if that's the best course of action, but you can see below it's the same behaviour:

$ cat test.go 
package main

import (
	"fmt"
	"golang.org/x/sys/unix"
)

func main() {
	var data []string
	data = []string{"hw.physmem", "hw.usermem", "kern.ostype"}
	for _, s := range data {
		if os, er := unix.Sysctl(s); er != nil {
			panic(er)
		} else {
			fmt.Println("DEBUG:", os)
			for i := 0; i < len(os); i++ {
				fmt.Println("DEBUG: index:", i, "int:", os[i], "str:", string(os[i]))
			}
		}
	}
}


$ go build test.go
$ ./test
DEBUG: �v
DEBUG: index: 0 int: 0 str: 
DEBUG: index: 1 int: 240 str: ð
DEBUG: index: 2 int: 118 str: v
DEBUG: index: 3 int: 127 str: 
DEBUG: p4
DEBUG: index: 0 int: 0 str: 
DEBUG: index: 1 int: 112 str: p
DEBUG: index: 2 int: 52 str: 4
DEBUG: index: 3 int: 127 str: 
DEBUG: NetBSD
DEBUG: index: 0 int: 78 str: N
DEBUG: index: 1 int: 101 str: e
DEBUG: index: 2 int: 116 str: t
DEBUG: index: 3 int: 66 str: B
DEBUG: index: 4 int: 83 str: S
DEBUG: index: 5 int: 68 str: D
$ 

[mdempsky]

such as hw.physmem which should return me (if things were converted properly) something like "2138501120" but instead gives me a 4 char string, with int values of 0 240 118 127, which is pretty clearly invalid.

syscall.Sysctl is just returning the raw byte data returned by the OS. []byte{0, 240, 118, 127} is the little-endian encoding of 2138501120.

[fuzzy]

I haven't been able to validate that, all attempts to unpack the value end with an 'unexpected EOF' can you show some example code that validates that? Also since the docs say (unix|syscall).Sysctl() returns a string, should it either a) do the binary unpacking and actually return a string like it says. b) say in a note, that it can return non string data and it's up to you to figure it out. or c) return an interface{} and allow the user to do type assertions?

[mdempsky]

can you show some example code that validates that?

https://play.golang.org/p/6ibnltrMd6

Also since the docs say (unix|syscall).Sysctl() returns a string, should it either

syscall.Sysctl is returning a string. Go strings aren't necessarily Unicode text.

I believe this is working as intended.