Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: get standard library building with -d=checkptr #34972

Open
mdempsky opened this issue Oct 17, 2019 · 46 comments

Comments

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Oct 17, 2019

As a prerequisite for #34964, the standard library's tests need to all pass when -d=checkptr is enabled:

$ go test -a -short -gcflags=all=-d=checkptr std cmd

Currently, that's not the case.

  1. There are also some panics because of (*[Big]T)(p)[:n] expressions, like in package reflect. This code pattern should be recognized by cmd/compile.

  2. There are a few tests that fail because CL 201781 leads them to heap allocate when before things would only stack allocate; e.g., context.TestAllocs or database/sql.TestRawBytesAllocs. Not sure how to handle this; maybe for now the escape analysis change should only be enabled for -d=checkptr=2, and -race/-msan only enable -d=checkptr=1.

  3. In sync/atomic.hammerStoreLoadPointer, there's (in effect):

     new := uintptr(LoadPointer(addr))
     new += offset
     StorePointer(addr, unsafe.Pointer(new))
    

    This violates package unsafe's pointer rules: pointers have to be converted to uintptr and back to unsafe.Pointer in a single expression without being stored in a uintptr-typed variable, as is being done with new. (This just needs someone to grok exactly what the test is doing, and fix it to follow pointer rules correctly.)

There seem to be other failures still that need to be diagnosed. If folks want to try running the command above, claiming a failure, and then investigating what's going on, that would be great.

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 17, 2019

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 17, 2019

This code in package reflect is currently failing in checkptrAlignment, and seems to violate the safety rules because of the conversion to *[16]byte (sometimes gcdata points to <16 bytes):

go/src/reflect/type.go

Lines 2184 to 2192 in 8c6876e

kmask := (*[16]byte)(unsafe.Pointer(ktyp.gcdata))
for i := uintptr(0); i < ktyp.ptrdata/ptrSize; i++ {
if (kmask[i/8]>>(i%8))&1 != 0 {
for j := uintptr(0); j < bucketSize; j++ {
word := base + j*ktyp.size/ptrSize + i
mask[word/8] |= 1 << (word % 8)
}
}
}

(There's similar code for etype just below.)

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Oct 17, 2019

I'll cover the sync/atomic.hammer* tests fixup :)

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

This failure looks genuinely suspicious to me:

$ go test -a -short -gcflags=all=-d=checkptr testing/quick
runtime: pointer 0xc00022a3e0 to unallocated span span.base()=0xc000222000 span.limit=0xc00023a000 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 18, 2019

Change https://golang.org/cl/201841 mentions this issue: all: disable tests that fail under -d=checkptr

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 18, 2019

Change https://golang.org/cl/201839 mentions this issue: cmd/compile: recognize (*[Big]T)(ptr)[:n:m] pattern for -d=checkptr

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

Uploaded a couple CLs to improve -d=checkptr, and address some issues.

CL 201841 enumerates all of the remaining std cmd test failures that I'm aware of, but haven't investigated yet.

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Oct 18, 2019

Hey @mdempsky in regards to the sync/atomic.HammerLoad* tests that are flagged, what they do is that for 8 goroutines, run pairwise increments of the high and low parts of an int* or *uint and panic if high != low. To save the value at the end of the function, we invoke

StorePointer(addr, newValue)

where newValue is a raw uint64 value such as 0xc0000000c0 and that value is being written to addr.

checkptr tries to find the object associated with newValue but unfortunately that's a raw value that's not attached to any object and perhaps beyond Go's address space. Writing such code is a valid usecase and here is a minimal repro to test that edge case

package main

import (
	"fmt"
	"sync/atomic"
	"unsafe"
)

func main() {
	val := uint64(820338753727)
	addr := (*unsafe.Pointer)(unsafe.Pointer(&val))
	v := uintptr(atomic.LoadPointer(addr))
	atomic.StorePointer(addr, unsafe.Pointer(v+1+1<<32))

	xs := uintptr(atomic.LoadPointer(addr))
	fmt.Printf("xs: %#x\n", xs)
}
$ go run -gcflags=all=-d=checkptr main.go 
panic: (runtime.ptrArith) (0x10c09c0,0xc00000c080)

goroutine 1 [running]:
main.main()
	/Users/emmanuelodeke/Desktop/openSrc/bugs/golang/34972/main.go:13 +0x91
exit status 2

Trying to recognize cases that do plain arithmetic is going to be a tricky one :)

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 18, 2019

Change https://golang.org/cl/201877 mentions this issue: syscall: fix wrong unsafe.Pointer align in anyToSockaddr

@cuonglm

This comment has been minimized.

Copy link
Contributor

@cuonglm cuonglm commented Oct 18, 2019

Another instance in reflect:

return (*[1 << 20]*rtype)(add(unsafe.Pointer(t), uadd, "t.inCount > 0"))[:t.inCount]

and some lines below:

return (*[1 << 20]*rtype)(add(unsafe.Pointer(t), uadd, "outCount > 0"))[t.inCount : t.inCount+outCount]

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

@odeke-em Thanks for looking into it.

Writing such code is a valid usecase

Regardless of the merit of the use case, if the example violates package unsafe's safety rules, then it's not valid Go.

Your repro case can be simplified to just:

package main

import (
	"fmt"
	"unsafe"
)

func main() {
	fmt.Println(unsafe.Pointer(uintptr(0xc000000000)))
}

This is failing because 0xc000000000 is an offset into the Go heap (at least where Go tries to allocate it by default on 64-bit OSes), and package unsafe's safety rules don't allow you to fabricate a pointer into the heap out of nothing.

To fix the test, it should either:

  1. Exclusively use pointers outside of the Go heap. (This is challenging, because you don't know where the Go heap is, and allocating non-Go-heap memory in a portable way is tricky.)

  2. Alternatively, allocate a large chunk of memory, find an adequately aligned pointer within that chunk of memory, and then restrict to modifying the lower bits of the address. For example:

     buf := make([]byte, 1 << 21) // allocate 2MB
     p := unsafe.Pointer(&buf[0])
     p = unsafe.Pointer((uintptr(p) + 1<<20 - 1) &^ (1<<20 - 1))  // align p to 1MB boundary
    

    p is now 20-bit aligned, and you can change any of the lower 20 bits and be guaranteed it still points into buf.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 18, 2019

There are also some panics because of (*[Big]T)(p)[:n] expressions, like in package reflect. This code pattern should be recognized by cmd/compile.

I saw that one coming in #13656 (comment)!

One could easily envision a compiler that, say, sanity-checks that all pointers to arrays point to addresses that are mapped in the program's address space.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 18, 2019

There are a few tests that fail because CL 201781 leads them to heap allocate when before things would only stack allocate

If so, those tests are overspecified: the language spec does not make any guarantees about allocations or escapes. They should be updated to skip (or use different thresholds) when they are not using a specific compiler in a specific configuration.

gopherbot pushed a commit that referenced this issue Oct 18, 2019
Caught with:

	go test -a -short -gcflags=all=-d=checkptr log/syslog

and:

	grep -rE '\*\[([^2]|.{2,})\].*\)\(unsafe.Pointer' syscall

Updates #34972

Change-Id: Iafd199b3a34beb7cc3e88484bf2fbae45183f951
Reviewed-on: https://go-review.googlesource.com/c/go/+/201877
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 18, 2019

They should be updated to skip (or use different thresholds) when they are not using a specific compiler in a specific configuration.

Agreed, though I don't think currently there's a way for tests to check what compiler flags were used. (Keep in mind too that different compiler flags can be used for different packages.)

@cuonglm

This comment has been minimized.

Copy link
Contributor

@cuonglm cuonglm commented Oct 19, 2019

@mdempsky on OSX, I got a lot of:

fatal error: stack growth after fork

runtime stack:
runtime.throw(0x12e77be, 0x17)
	/Users/cuonglm/sources/go/src/runtime/panic.go:774 +0x72
runtime.newstack()
	/Users/cuonglm/sources/go/src/runtime/stack.go:928 +0xd04
runtime.morestack()
	/Users/cuonglm/sources/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 19 [running]:
runtime.checkptrAlignment(0xc00008a9e8, 0x126e840)
	/Users/cuonglm/sources/go/src/runtime/checkptr.go:14 +0x137 fp=0xc00008a8e8 sp=0xc00008a8e0 pc=0x1006177
syscall.funcPC(...)
	/Users/cuonglm/sources/go/src/syscall/syscall_darwin.go:469
syscall.forkAndExecInChild(0xc00008e120, 0xc000082ba0, 0x5, 0x5, 0xc00006a900, 0x2e, 0x2e, 0x0, 0x0, 0xc00008ac00, ...)
	/Users/cuonglm/sources/go/src/syscall/exec_darwin.go:65 +0xe6 fp=0xc00008aa60 sp=0xc00008a8e8 pc=0x10703d6
syscall.forkExec(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ac00, 0x1140b6b, 0x0, 0x0)
	/Users/cuonglm/sources/go/src/syscall/exec_unix.go:201 +0x35b fp=0xc00008ab70 sp=0xc00008aa60 pc=0x10718eb
syscall.StartProcess(...)
	/Users/cuonglm/sources/go/src/syscall/exec_unix.go:248
os.startProcess(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ad98, 0x0, 0x0, 0x0)
	/Users/cuonglm/sources/go/src/os/exec_posix.go:52 +0x2c0 fp=0xc00008ac58 sp=0xc00008ab70 pc=0x1093300
os.StartProcess(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ad98, 0x2d, 0x0, 0x0)
	/Users/cuonglm/sources/go/src/os/exec.go:102 +0x7c fp=0xc00008acb0 sp=0xc00008ac58 pc=0x1092d2c
os/exec.(*Cmd).Start(0xc0000bc2c0, 0xc00008af01, 0xc000082b70)
	/Users/cuonglm/sources/go/src/os/exec/exec.go:416 +0x50c fp=0xc00008adf0 sp=0xc00008acb0 pc=0x116cbac
os/exec.(*Cmd).Run(0xc0000bc2c0, 0xc000082b70, 0xc0000bc2c0)
	/Users/cuonglm/sources/go/src/os/exec/exec.go:338 +0x2b fp=0xc00008ae18 sp=0xc00008adf0 pc=0x116c63b
os/exec.(*Cmd).CombinedOutput(0xc0000bc2c0, 0x20, 0xc00008af30, 0x3, 0x3, 0xc0000bc2c0)
	/Users/cuonglm/sources/go/src/os/exec/exec.go:561 +0x91 fp=0xc00008ae48 sp=0xc00008ae18 pc=0x116d7a1
go/importer.TestForCompiler(0xc0000f0300)
	/Users/cuonglm/sources/go/src/go/importer/importer_test.go:23 +0x11d fp=0xc00008af70 sp=0xc00008ae48 pc=0x124dfed
testing.tRunner(0xc0000f0300, 0x12f26a0)
	/Users/cuonglm/sources/go/src/testing/testing.go:909 +0xc9 fp=0xc00008afd0 sp=0xc00008af70 pc=0x10f8249
runtime.goexit()
	/Users/cuonglm/sources/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc00008afd8 sp=0xc00008afd0 pc=0x105cd11
created by testing.(*T).Run
	/Users/cuonglm/sources/go/src/testing/testing.go:960 +0x351

when testing std, even after 46aa835

Note

This error does not happen on Linux.

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 19, 2019

@cuonglm Thanks for the report.

It looks like maybe checkptrAlignment and checkptrArithmetic need to be labeled //go:nosplit.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 19, 2019

Change https://golang.org/cl/202157 mentions this issue: runtime: fix unsafe.Pointer alignment on Linux

@cuonglm

This comment has been minimized.

Copy link
Contributor

@cuonglm cuonglm commented Oct 19, 2019

@cuonglm Thanks for the report.

It looks like maybe checkptrAlignment and checkptrArithmetic need to be labeled //go:nosplit.

I tried, but still fails with go:nosplit for checkptrAlignment and checkptrArithmetic. But it does pass if findObject is marked go:nosplit, too.

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 19, 2019

Ugh, right.

Okay, in that case, we should probably just make //go:nosplit imply //go:nocheckptr, like how //go:cgo_unsafe_args does. (See CL 201823.)

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 19, 2019

Change https://golang.org/cl/202158 mentions this issue: cmd/compile: disable checkptr for //go:nosplit functions

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 19, 2019

Change https://golang.org/cl/202177 mentions this issue: windows, unix: fix wrong unsafe.Pointer alignment in syscall

gopherbot pushed a commit to golang/sys that referenced this issue Oct 20, 2019
Same as CL 201877 did for package syscall.

Updates golang/go#34972

Change-Id: I3929841ab32378516edafb1f02a84b1bdcc77bbd
Reviewed-on: https://go-review.googlesource.com/c/sys/+/202177
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 21, 2019

Change https://golang.org/cl/202580 mentions this issue: reflect, internal/reflectlite: set capacity when slicing unsafe pointers

gopherbot pushed a commit that referenced this issue Oct 21, 2019
A common idiom for turning an unsafe.Pointer into a slice is to write:

    s := (*[Big]T)(ptr)[:n:m]

This technically violates Go's unsafe pointer rules (rule #1 says T2
can't be bigger than T1), but it's fairly common and not too difficult
to recognize, so might as well allow it for now so we can make
progress on #34972.

This should be revisited if #19367 is accepted.

Updates #22218.
Updates #34972.

Change-Id: Id824e2461904e770910b6e728b4234041d2cc8bc
Reviewed-on: https://go-review.googlesource.com/c/go/+/201839
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue Oct 21, 2019
Follow the idiom for allowing -d=checkptr to recognize and verify
correctness.

Updates #22218.
Updates #34972.

Change-Id: Ib6001c6f0e6dc535a36bcfaa1ae48e29e0c737f8
Reviewed-on: https://go-review.googlesource.com/c/go/+/202580
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Oct 22, 2019
This test could be updated to use unsafe.Pointer arithmetic properly
(e.g., see discussion at #34972), but it doesn't seem worthwhile. The
test is just checking that LoadPointer and StorePointer are atomic.

Updates #34972.

Change-Id: I85a8d610c1766cd63136cae686aa8a240a362a18
Reviewed-on: https://go-review.googlesource.com/c/go/+/202597
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Oct 22, 2019
The code for generating gcdata was (technically) unsafe. It was also
rather repetitive. This CL refactors it a bit and abstracts use of
gcdata into a helper gcSlice method.

Updates #34972.

Change-Id: Ie86d7822eafe263f1d3d150eedf0ec66be1ec85d
Reviewed-on: https://go-review.googlesource.com/c/go/+/202582
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 22, 2019

Change https://golang.org/cl/202640 mentions this issue: net: fix checkptr failure on Windows

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 22, 2019

I think !windows platforms are in good shape now, and -d=checkptr is now enabled automatically when -race or -msan are used on !windows platforms.

This issue could still use a Windows developer to look at the cases I identified above, and also to try running go test -a -short -gcflags=all=-d=checkptr std cmd to check for any other failures.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Oct 22, 2019

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 28, 2019

Change https://golang.org/cl/203837 mentions this issue: sha3: align (*state).storage

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 28, 2019

Change https://golang.org/cl/203442 mentions this issue: internal/syscall/windows/registry: make '-gcflags=all=-d=checkptr' flag work

gopherbot pushed a commit that referenced this issue Oct 29, 2019
…ag work

Mostly replaced [:x] slice operation with [x]. 

According to @mdempsky, compiler specially recognizes when you combine
a pointer conversion with a full slice operation in a single expression
and makes an exception.

Updates #34972

Change-Id: I07d9de3b31da254d55f50d14c18155f8fc8f3ece
Reviewed-on: https://go-review.googlesource.com/c/go/+/203442
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 29, 2019

Change https://golang.org/cl/204017 mentions this issue: windows/registry: make '-gcflags=all=-d=checkptr' flag work

gopherbot pushed a commit to golang/sys that referenced this issue Oct 29, 2019
Mostly replaced [:x] slice operation with [x]. 

This change is copy of CL 203442.

Updates golang/go#34972

Change-Id: I9b725a39aea469bd5669cc1deb3d02a88912843e
Reviewed-on: https://go-review.googlesource.com/c/sys/+/204017
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Oct 30, 2019

crypto/x509/root_windows.go: simpleChains := ([1 << 20]syscall.CertSimpleChain)(unsafe.Pointer(simpleChain))[:]
crypto/x509/root_windows.go: elements := (
[1 << 20]syscall.CertChainElement)(unsafe.Pointer(lastChain.Elements))[:]
crypto/x509/root_windows.go: encodedCert := (
[1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]
crypto/x509/root_windows.go: buf := (
[1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]

@mdempsky checker does not complain about crypto/x509 code

c:\>go test -a -short -gcflags=all=-d=checkptr crypto/x509
ok      crypto/x509     0.393s

Why? All 4 lines deals with external to Go memory returned by Windows API. Is that why?

Thank you.

Alex

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 30, 2019

Change https://golang.org/cl/204218 mentions this issue: syscall: revert security_windows.go change of CL 201877

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Oct 30, 2019

Why? All 4 lines deals with external to Go memory returned by Windows API. Is that why?

That sounds plausible.

The way the check currently works is when you convert p into *T, we try to check that p and p + sizeof(T) - 1 both point into the same Go variable.

For pointers into the Go heap, we have very precise information about Go variable boundaries (because we need that for GC), so that's generally a very robust check.

For pointers into Goroutine stacks or into global Go variables, we have course grained information: basically just that it points into a particular stack, or into some particular EXE or DLL's global Go variables.

For C allocated memory, we don't have any visibility at the moment. Basically anything that wasn't allocated by Go is treated uniformly as non-Go memory.

However, the reason you still need to be careful even with pointers to C memory is that while p might point into C memory, it's possible that p + sizeof([1 << 20]SomeType) - 1 might overflow into the Go heap depending on where p was allocated in memory.

@changkun changkun referenced this issue Nov 1, 2019
0 of 16 tasks complete
@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Nov 3, 2019

The way the check currently works is

Thank you for taking time to explain.

I think we do need to change crypto/x509 package.

Alex

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 3, 2019

Change https://golang.org/cl/204621 mentions this issue: crypto/x509: make '-gcflags=all=-d=checkptr' flag work

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Nov 3, 2019

@mdempsky I need more help.

I am trying to convert this line

net/interface_windows.go: Name: syscall.UTF16ToString((([10000]uint16)(unsafe.Pointer(aa.FriendlyName)))[:]),

Name: syscall.UTF16ToString((*(*[10000]uint16)(unsafe.Pointer(aa.FriendlyName)))[:]),

But FriendlyName is 0 terminated uint16 string (see https://docs.microsoft.com/en-us/windows/win32/api/iptypes/ns-iptypes-ip_adapter_addresses_lh for the reference). There is no length of FriendlyName provided anywhere. What do we do here?

The test is also crashes

:\>u:\test -test.short
panic: runtime error: unsafe pointer conversion

goroutine 1 [running]:
net.interfaceTable(0x0, 0xa80, 0xa80, 0x66e940, 0x1, 0xc000087ae0)
        /home/a/go/src/net/interface_windows.go:61 +0xec
net.Interfaces(0x20, 0x28, 0xc0000667e0, 0x47fbb1, 0x890854)
        /home/a/go/src/net/interface.go:100 +0x35
net.loopbackInterface(0x65d0c0)
        /home/a/go/src/net/interface_test.go:20 +0x3b
net.setupTestData()
        /home/a/go/src/net/main_test.go:102 +0x5c
net.TestMain(0xc0000ac000)
        /home/a/go/src/net/main_test.go:49 +0x2d
main.main()
        _testmain.go:474 +0x13c

C:\>

Thank you.

Alex

gopherbot pushed a commit that referenced this issue Nov 3, 2019
Replace

buf := [HUGE_CONST]*T)(unsafe.Pointer(p))[:]

with

buf := [HUGE_CONST]*T)(unsafe.Pointer(p))[:n:n]

Pointer p points to n of T elements. New unsafe pointer conversion
logic verifies that both first and last elements point into the
same Go variable. And this change adjusts all code to comply with
this rule.

Verified by running

go test -a -short -gcflags=all=-d=checkptr crypto/x509

The test does not fail even with original version of this code. I
suspect it is because all variables I changed live outside of Go
memory. But I am just guessing, I don't really know how pointer
checker works.

Updates #34972

Change-Id: Ibc33fdc9e2023d9b14905c9badf2f0b683999ab8
Reviewed-on: https://go-review.googlesource.com/c/go/+/204621
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/crypto that referenced this issue Nov 5, 2019
Even on platforms that allow unaligned reads, the Go runtime assumes
that a pointer to a given type has the alignment required by that
type.

Fixes golang/go#35173
Updates golang/go#34972
Updates golang/go#34964

Change-Id: I90361e096e59162e42ebde2914985af92f777ece
Reviewed-on: https://go-review.googlesource.com/c/crypto/+/203837
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 7, 2019

Change https://golang.org/cl/204117 mentions this issue: syscall: revert wrong sid string length windows

gopherbot pushed a commit that referenced this issue Nov 7, 2019
This CL was verified by running:

	go test -gcflags=all=-d=checkptr=2 internal/syscall/windows

internal/syscall/windows.TestRunAtLowIntegrity uses code in question.

Updates #34972

Change-Id: I434530058e2d41f132e9bf154e8c64c03894e9c4
Reviewed-on: https://go-review.googlesource.com/c/go/+/204117
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Nov 8, 2019

@mdempsky I need more help.

Sorry, I missed this. (Well, I saw the GitHub notification emails, but I thought it was just gopherbot comments.)

There is no length of FriendlyName provided anywhere. What do we do here?

We'll need to compute the length then. E.g., a function like:

func utf16PtrToString(p *uint16) string {
    // Find NUL terminator.
    end := unsafe.Pointer(p)
    for *(*uint16)(end) != 0 {
        end = unsafe.Pointer(uintptr(end) + unsafe.Sizeof(*p))
    }

    n := (uintptr(end) - uintptr(unsafe.Pointer(p))) / unsafe.Sizeof(*p)
    s := (*[1<<30]uint16)(unsafe.Pointer(p))[:n:n]
    return string(utf16.Decode(s))
}
@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Nov 8, 2019

func utf16PtrToString(p *uint16) string {

I think this function will be useful to make other code '-gcflags=all=-d=checkptr' safe.

I could add this function to internal/syscall/windows package. But maybe we should add it to syscall package instead. We could recommend the function to be used in external code instead of syscall.UTF16ToString.

What do you think? Is it too earlier to make this decision?

Thank you.

Alex

@mdempsky

This comment has been minimized.

Copy link
Member Author

@mdempsky mdempsky commented Nov 8, 2019

I could add this function to internal/syscall/windows package.

That sounds reasonable to me.

But maybe we should add it to syscall package instead.

My understanding is package syscall is frozen, and no new functions are to be added.

This seems like maybe it would be a reasonable exception? But that's not my call, so I'd probably start by just adding it to x/sys/windows instead.

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Nov 10, 2019

My understanding is package syscall is frozen, and no new functions are to be added.

This seems like maybe it would be a reasonable exception?

I had to use new function in syscall package. So it cannot be in internal/syscall/windows, because internal/syscall/windows already imports syscall. I will put in syscall for now, and Ian could decide.

I went up to

os/os_windows_test.go: copy((*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:], target.pathBuf)

now. go test crashes with

panic: runtime error: unsafe pointer conversion [recovered]
        panic: runtime error: unsafe pointer conversion

goroutine 138 [running]:
testing.tRunner.func1(0xc00028e120)
        /home/a/go/src/testing/testing.go:916 +0x60b
panic(0x5afc80, 0xc000328680)
        /home/a/go/src/runtime/panic.go:961 +0x16b
os_test.createMountPoint(0xc0003394f0, 0x44, 0xc0002a5c10, 0xc00074007e, 0x3a)
        /home/a/go/src/os/os_windows_test.go:266 +0x119
os_test.TestDirectoryJunction.func1(0xc0003394f0, 0x44, 0xc00029a180, 0x3a, 0x44, 0xc0000861d0)
        /home/a/go/src/os/os_windows_test.go:285 +0xf5
os_test.testDirLinks(0xc00028e120, 0xc0004d4400, 0x3, 0x4)
        /home/a/go/src/os/os_windows_test.go:123 +0x671
os_test.TestDirectoryJunction(0xc00028e120)
        /home/a/go/src/os/os_windows_test.go:317 +0x28c
testing.tRunner(0xc00028e120, 0x5d9eb0)
        /home/a/go/src/testing/testing.go:954 +0xe3
created by testing.(*T).Run
        /home/a/go/src/testing/testing.go:1005 +0x35e

so it is definitely a problem. What is happening here, we have

package windows // import "internal/syscall/windows"

type MountPointReparseBuffer struct {
        // The integer that contains the offset, in bytes,
        // of the substitute name string in the PathBuffer array,
        // computed as an offset from byte 0 of PathBuffer. Note that
        // this offset must be divided by 2 to get the array index.
        SubstituteNameOffset uint16
        // The integer that contains the length, in bytes, of the
        // substitute name string. If this string is null-terminated,
        // SubstituteNameLength does not include the Unicode null character.
        SubstituteNameLength uint16
        // PrintNameOffset is similar to SubstituteNameOffset.
        PrintNameOffset uint16
        // PrintNameLength is similar to SubstituteNameLength.
        PrintNameLength uint16
        PathBuffer      [1]uint16
}

but PathBuffer is longer than one uint16 long - it is PrintNameLength elements long.

See MountPointReparseBuffer union part of REPARSE_DATA_BUFFER

https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_reparse_data_buffer

Windows reads or writes uint16 string starting from PathBuffer[1]. How do I write uint16 string there?

That is what we do at this moment.

copy((*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:], target.pathBuf)

Thank you.

Alex

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Nov 10, 2019

@alexbrainman, changing

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:]

to

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:buflen:buflen]

might do the trick. (See CL 201839.)

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Nov 11, 2019

changing

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:]

to

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:buflen:buflen]

might do the trick

Thank you, Bryan. That does the trick.

I knew the trick, but I forgotten about it. Moving on for now.

@mdempsky I also had to change

--- a/src/internal/syscall/windows/reparse_windows.go
+++ b/src/internal/syscall/windows/reparse_windows.go
@@ -60,8 +60,9 @@ type SymbolicLinkReparseBuffer struct {

 // Path returns path stored in rb.
 func (rb *SymbolicLinkReparseBuffer) Path() string {
-       p := (*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))
-       return syscall.UTF16ToString(p[rb.SubstituteNameOffset/2 : (rb.SubstituteNameOffset+rb.SubstituteNameLength)/2])
+       n1 := rb.SubstituteNameOffset / 2
+       n2 := (rb.SubstituteNameOffset + rb.SubstituteNameLength) / 2
+       return syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))[n1:n2:n2])
 }

 type MountPointReparseBuffer struct {
@@ -83,6 +84,7 @@ type MountPointReparseBuffer struct {

 // Path returns path stored in rb.
 func (rb *MountPointReparseBuffer) Path() string {
-       p := (*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))
-       return syscall.UTF16ToString(p[rb.SubstituteNameOffset/2 : (rb.SubstituteNameOffset+rb.SubstituteNameLength)/2])
+       n1 := rb.SubstituteNameOffset / 2
+       n2 := (rb.SubstituteNameOffset + rb.SubstituteNameLength) / 2
+       return syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))[n1:n2:n2])
 }

because go test crashed. But these weren't listed at #34972 (comment) I am worried we might miss more like it. Should I try different grep '\*\[.*unsafe.*:' expression?

Thank you.

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.