Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support go:linkname directives to method funcs #656

Closed
smjure opened this issue Jan 26, 2023 · 8 comments · Fixed by #665
Closed

support go:linkname directives to method funcs #656

smjure opened this issue Jan 26, 2023 · 8 comments · Fixed by #665

Comments

@smjure
Copy link

smjure commented Jan 26, 2023

What version of Garble and Go are you using?

$ garble version
mvdan.cc/garble v0.9.0

Build settings:
       -compiler gc
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v1

$ go version
go version go1.19.5 linux/amd64

What environment are you running Garble on?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oem/.cache/go-build"
GOENV="/home/oem/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/oem/golib/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/oem/golib"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.5"
GCCGO="/usr/bin/gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/oem/...project....../go.mod"
GOWORK="/home/oem/...project....../go.work"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2932583849=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Gargle runs on small project w/o problems, but now ran into a bit bigger project and I get the following error after running simply garble build project.go.

# command-line-arguments
/home/oem/.cache/garble/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-3318866579/go.o: in function `RWFKI7lUxVT.(*HAzA2B8).kVzGumOQDTz':
go.go:(.text+0x2c832f): undefined reference to `reflect.(*rtype).ptrTo'
/usr/bin/ld: go.go:(.text+0x2c83b1): undefined reference to `reflect.(*rtype).ptrTo'
/usr/bin/ld: /tmp/go-link-3318866579/go.o: in function `RWFKI7lUxVT.(*HAzA2B8).ggejxFgkq5yE':
go.go:(.text+0x2c8b97): undefined reference to `reflect.(*rtype).ptrTo'
/usr/bin/ld: /tmp/go-link-3318866579/go.o: in function `RWFKI7lUxVT.(*HAzA2B8).dqSpzzLFM':
go.go:(.text+0x2c8e08): undefined reference to `reflect.(*rtype).ptrTo'
/usr/bin/ld: /tmp/go-link-3318866579/go.o: in function `RWFKI7lUxVT.(*HAzA2B8).nMWdil':
go.go:(.text+0x2c9551): undefined reference to `reflect.(*rtype).ptrTo'
/usr/bin/ld: /tmp/go-link-3318866579/go.o:go.go:(.text+0x2c9624): more undefined references to `reflect.(*rtype).ptrTo' follow
collect2: error: ld returned 1 exit status

exit status 2

If I try with different compiler, i.e. garble build -compiler gccgo project.go, the following error occur:
go list error: exit status 2: # golang.org/x/exp/constraints

/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:13:9: error: invalid character 0x7e in input file
   13 |         ~int | ~int8 | ~int16 | ~int32 | ~int64
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:20:9: error: invalid character 0x7e in input file
   20 |         ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:27:9: error: expected signature or type name
   27 |         Signed | Unsigned
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:34:9: error: invalid character 0x7e in input file
   34 |         ~float32 | ~float64
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:41:9: error: invalid character 0x7e in input file
   41 |         ~complex64 | ~complex128
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:49:9: error: expected signature or type name
   49 |         Integer | Float | ~string
      |         ^
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/constraints/constraints.go:49:27: error: invalid character 0x7e in input file
   49 |         Integer | Float | ~string
      |                           ^
# golang.org/x/exp/slog/internal/buffer
/home/oem/golib/pkg/mod/golang.org/x/exp@v0.0.0-20221217163422-3c43f8badb15/slog/internal/buffer/buffer.go:17:21: error: use of undefined type ‘any’
   17 |         New: func() any {
      |                     ^
# go.uber.org/atomic
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:29:16: error: expected ‘]’
   29 | type Pointer[T any] struct {
      |                ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:31:9: error: expected declaration
   31 |         p atomic.Pointer[T]
      |         ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:32:1: error: expected declaration
   32 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:35:16: error: expected ‘(’
   35 | func NewPointer[T any](v *T) *Pointer[T] {
      |                ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:35:19: error: expected ‘]’
   35 | func NewPointer[T any](v *T) *Pointer[T] {
      |                   ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:35:19: error: expected ‘;’ or newline after top level declaration
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:37:9: error: expected declaration
   37 |         if v != nil {
      |         ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:39:9: error: expected declaration
   39 |         }
      |         ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:40:9: error: expected declaration
   40 |         return &p
      |         ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:41:1: error: expected declaration
   41 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:44:17: error: expected ‘)’
   44 | func (p *Pointer[T]) Load() *T {
      |                 ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:44:17: error: expected function name
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:44:17: error: expected ‘;’ or newline after top level declaration
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:46:1: error: expected declaration
   46 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:49:17: error: expected ‘)’
   49 | func (p *Pointer[T]) Store(val *T) {
      |                 ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:49:17: error: expected function name
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:49:17: error: expected ‘;’ or newline after top level declaration
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:51:1: error: expected declaration
   51 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:54:17: error: expected ‘)’
   54 | func (p *Pointer[T]) Swap(val *T) (old *T) {
      |                 ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:54:17: error: expected function name
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:54:17: error: expected ‘;’ or newline after top level declaration
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:56:1: error: expected declaration
   56 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:59:17: error: expected ‘)’
   59 | func (p *Pointer[T]) CompareAndSwap(old, new *T) (swapped bool) {
      |                 ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:59:17: error: expected function name
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:59:17: error: expected ‘;’ or newline after top level declaration
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/pointer_go119.go:61:1: error: expected declaration
   61 | }
      | ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/error.go:55:19: error: reference to undefined field or method ‘CompareAndSwap’
   55 |         return x.v.CompareAndSwap(packError(old), packError(new))
      |                   ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/error.go:61:31: error: reference to undefined field or method ‘Swap’
   61 |         return unpackError(x.v.Swap(packError(val)))
      |                               ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/string.go:58:19: error: reference to undefined field or method ‘CompareAndSwap’
   58 |         return x.v.CompareAndSwap(old, new)
      |                   ^
/home/oem/golib/pkg/mod/go.uber.org/atomic@v1.10.0/string.go:64:19: error: reference to undefined field or method ‘Swap’
   64 |         return x.v.Swap(val).(string)
      |                   ^
# golang.org/x/sys/unix
/home/oem/golib/pkg/mod/golang.org/x/sys@v0.1.0/unix/syscall.go:83:30: error: reference to undefined identifier ‘unsafe.Slice’
   83 |         return string(unsafe.Slice(p, n))
      |                              ^
/home/oem/golib/pkg/mod/golang.org/x/sys@v0.1.0/unix/syscall_linux.go:2255:23: error: reference to undefined identifier ‘unsafe.Slice’
 2255 |         return unsafe.Slice((*byte)(unsafe.Pointer(uintptr(unsafe.Pointer(&fh.fileHandle.Type))+4)), n)
      |                       ^
/home/oem/golib/pkg/mod/golang.org/x/sys@v0.1.0/unix/syscall_unix.go:118:21: error: reference to undefined identifier ‘unsafe.Slice’
  118 |         b := unsafe.Slice((*byte)(unsafe.Pointer(addr)), length)
      |                     ^
/home/oem/golib/pkg/mod/golang.org/x/sys@v0.1.0/unix/sysvshm_unix.go:33:21: error: reference to undefined identifier ‘unsafe.Slice’
   33 |         b := unsafe.Slice((*byte)(unsafe.Pointer(addr)), int(info.Segsz))
      |                     ^

What did you expect to see?

To build it like go build project.go, which works fine.

What did you see instead?

The above mentioned errors.

@mvdan
Copy link
Member

mvdan commented Jan 26, 2023

Can you provide a way for us to reproduce the failure? ptrTo is a regular Go method, which is being obfuscated by garble, so I'm not sure why you still have references to the unobfuscated name.

You could build with garble -debug build and look through the log to see which package is RWFKI7lUxVT, and which are the functions like kVzGumOQDTz or dqSpzzLFM. That should help narrow down where the failure is coming from.

We don't support gccgo at this point, so that part is not really relevant.

@smjure
Copy link
Author

smjure commented Jan 26, 2023

Thank you for the reply 🙏🏻 The project is pretty comprehensive, but will try to reproduce it via the suggested -debug flag and minimal example, it may take some time.. 🙏🏻

@mvdan
Copy link
Member

mvdan commented Jan 26, 2023

Also see #595, if you'd like my help there.

@smjure
Copy link
Author

smjure commented Jan 26, 2023

I think I found the cause, it's the goccy package. The following code also reports on ptrTo method. How could this be resolved? I'll also try with some other json package.

package main

import (
	"fmt"

	json "github.com/goccy/go-json"
)

type T struct {
	X int
	U *U
}

type U struct {
	T *T
}

func main() {
	b, err := json.Marshal(&T{
		X: 1,
		U: &U{
			T: &T{
				X: 2,
			},
		},
	})
	if err != nil {
		fmt.Printf("Err Marshal:%s\n", err)
	}
	fmt.Println(string(b)) // {"X":1,"U":{"T":{"X":2,"U":null}}}
}

@smjure
Copy link
Author

smjure commented Jan 26, 2023

Ok, I changed the goccy with jsoniter package. It compiles w/o problems, but when I run the binary, the following panic is thrown, which is not thrown, if I build it normally via go build project.go.

panic: runtime error: index out of range [-1]

goroutine 133 [running]:
main.d67ZD2y(0xc00c38a080, 0xc000405000, {0xc00037c09a?, 0xa?})
	FzWpLnHnUzjF.go:2 +0x150e
main.oOrcNgqAfp.func1({0xc00010f77c, 0xa})
	vmvf56K8Lt.go:2 +0x1d4
created by main.oOrcNgqAfp
	hH_psxyg.go:3 +0x2ea

How could I debug this? Namely, when I ran it with -debug flag, i could not find any of these FzWpLnHnUzjF, vmvf56K8Lt, hH_psxyg,...
I have not tried with the standard Go encode/json, it's just too slow when dealing with large files.

@mvdan
Copy link
Member

mvdan commented Jan 30, 2023

That run-time panic looks like a different bug to me, can you please open a separate issue with the bug template?

@smjure
Copy link
Author

smjure commented Jan 30, 2023

Sorry am not dev, still learning, will do so... Can this one be closed?

@mvdan
Copy link
Member

mvdan commented Jan 30, 2023

No need to close it - this bug is also valid :)

@mvdan mvdan changed the title running gcc failed: exit status 1 support linknames to method funcs Jan 31, 2023
@mvdan mvdan changed the title support linknames to method funcs support go:linkname directives to method funcs Jan 31, 2023
mvdan added a commit to mvdan/garble-fork that referenced this issue Feb 2, 2023
This is not common, but it is done by a few projects.
Namely, github.com/goccy/go-json reached into reflect's guts,
which included a number of methods:

	internal/runtime/rtype.go
	11://go:linkname rtype_Align reflect.(*rtype).Align
	19://go:linkname rtype_FieldAlign reflect.(*rtype).FieldAlign
	27://go:linkname rtype_Method reflect.(*rtype).Method
	35://go:linkname rtype_MethodByName reflect.(*rtype).MethodByName
	[...]

Add tests for such go:linkname directives pointing at methods.
Note that there are two possible symbol string variants;
"pkg/path.(*Receiver).method" for methods with pointer receivers,
and "pkg/path.Receiver.method" for the rest.

We can't assume that the presence of two dots means a method either.
For example, a package path may be "pkg/path.with.dots",
and so "pkg/path.with.dots.SomeFunc" is the function "SomeFunc"
rather than the method "SomeFunc" on a type "dots".
To account for this ambiguity, rather than splitting on the last dot
like we used to, try to find a package path prefix by splitting on an
increasing number of first dots.

This can in theory still be ambiguous. For example,
we could have the package "pkg/path" expose the method "foo.bar",
and the package "pkg/path.foo" expose the func "bar".
Then, the symbol string "pkg/path.foo.bar" could mean either of them.
However, this seems extremely unlikely to happen in practice,
and I'm not sure that Go's toolchain would support it either.

I also noticed that goccy/go-json still failed to build after the fix.
The reason was that the type reflect.rtype wasn't being obfuscated.
We could, and likely should, teach our assembly and linkname
transformers about which names we chose not to obfuscate due to the use
of reflection. However, in this particular case, reflect's own types
can be obfuscated safely, so just do that.

Fixes burrowers#656.
@mvdan mvdan closed this as completed in #665 Feb 2, 2023
mvdan added a commit that referenced this issue Feb 2, 2023
This is not common, but it is done by a few projects.
Namely, github.com/goccy/go-json reached into reflect's guts,
which included a number of methods:

	internal/runtime/rtype.go
	11://go:linkname rtype_Align reflect.(*rtype).Align
	19://go:linkname rtype_FieldAlign reflect.(*rtype).FieldAlign
	27://go:linkname rtype_Method reflect.(*rtype).Method
	35://go:linkname rtype_MethodByName reflect.(*rtype).MethodByName
	[...]

Add tests for such go:linkname directives pointing at methods.
Note that there are two possible symbol string variants;
"pkg/path.(*Receiver).method" for methods with pointer receivers,
and "pkg/path.Receiver.method" for the rest.

We can't assume that the presence of two dots means a method either.
For example, a package path may be "pkg/path.with.dots",
and so "pkg/path.with.dots.SomeFunc" is the function "SomeFunc"
rather than the method "SomeFunc" on a type "dots".
To account for this ambiguity, rather than splitting on the last dot
like we used to, try to find a package path prefix by splitting on an
increasing number of first dots.

This can in theory still be ambiguous. For example,
we could have the package "pkg/path" expose the method "foo.bar",
and the package "pkg/path.foo" expose the func "bar".
Then, the symbol string "pkg/path.foo.bar" could mean either of them.
However, this seems extremely unlikely to happen in practice,
and I'm not sure that Go's toolchain would support it either.

I also noticed that goccy/go-json still failed to build after the fix.
The reason was that the type reflect.rtype wasn't being obfuscated.
We could, and likely should, teach our assembly and linkname
transformers about which names we chose not to obfuscate due to the use
of reflection. However, in this particular case, reflect's own types
can be obfuscated safely, so just do that.

Fixes #656.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

2 participants