Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/types: fails to fully collect methods from embedded interfaces #22701

Closed
chewxy opened this issue Nov 13, 2017 · 5 comments
Closed

go/types: fails to fully collect methods from embedded interfaces #22701

chewxy opened this issue Nov 13, 2017 · 5 comments

Comments

@chewxy
Copy link

@chewxy chewxy commented Nov 13, 2017

What version of Go are you using (go version)?

Master branch (14th Nov 2017)

Does this issue reproduce with the latest release?

No. It works fine in Go 1.9

What operating system and processor architecture are you using (go env)?

go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/chewxy/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/chewxy/workspace/goddamnit"
GORACE=""
GOROOT="/home/chewxy/.gvm/gos/master"
GOTMPDIR=""
GOTOOLDIR="/home/chewxy/.gvm/gos/master/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-L/usr/lib -lopenblas"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build933801049=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Tried to run tests for gorgonia/tensor: https://github.com/gorgonia/tensor (go get gorgonia.org/tensor)

MINIMAL REPRODUCTION HERE

After some hours of debugging it seems order of declaration matters:

What did you expect to see?

The tests run.

What did you see instead?

vet: typecheck failures

Here's a sample line:

# gorgonia.org/tensor
./defaultengine_mapreduce.go:29:13: invalid operation: v (variable of type View) has no field or method Clone
./tensor.go:139:19: cannot use t.Slice(slices...) (value of type View) as Tensor value in assignment: missing method Clone

But here's the definitions:

View:

// View is any Tensor that can provide a view on memory
type View interface {
	Tensor
	IsView() bool
	IsMaterializable() bool
	Materialize() Tensor
}

Tensor:

// Tensor represents a variety of n-dimensional arrays. The most commonly used tensor is the Dense tensor.
// It can be used to represent a vector, matrix, 3D matrix and n-dimensional tensors.
type Tensor interface {
	// elided methods that are methods, instead here's a list of compositional interfaces that the Tensor type is built out of
	// data related interface
	Zeroer
	MemSetter
	Dataer
	Eq
	Cloner

	// formatters
	fmt.Formatter
	fmt.Stringer

	gob.GobEncoder
	gob.GobDecoder

	headerer
	arrayer
}

Cloner:

// Cloner is any type that can clone itself
type Cloner interface {
	Clone() interface{}
}

Guru does not see the Cloner interface:

$GOPATH/srcgorgonia.org/tensor/interfaces.go:62.6-62.9: definition of type View (size 16, align 8)
$GOPATH/srcgorgonia.org/tensor/interfaces.go:62.6-62.9: Methods:
$GOPATH/srcgorgonia.org/tensor/tensor.go:42.2-42.6: 	method (View) Apply(fn interface{}, opts ...FuncOpt) (Tensor, error)
$GOPATH/srcgorgonia.org/tensor/tensor.go:36.2-36.3: 	method (View) At(...int) (interface{}, error)
$GOPATH/srcgorgonia.org/tensor/tensor.go:28.2-28.9: 	method (View) DataSize() int
$GOPATH/srcgorgonia.org/tensor/tensor.go:26.2-26.5: 	method (View) Dims() int
$GOPATH/srcgorgonia.org/tensor/tensor.go:25.2-25.6: 	method (View) Dtype() Dtype
$GOPATH/srcgorgonia.org/tensor/tensor.go:58.2-58.7: 	method (View) Engine() Engine
$GOPATH/srcgorgonia.org/tensor/tensor.go:63.2-63.18: 	method (View) IsManuallyManaged() bool
$GOPATH/srcgorgonia.org/tensor/interfaces.go:65.2-65.17: 	method (View) IsMaterializable() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:62.2-62.21: 	method (View) IsNativelyAccessible() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:52.2-52.9: 	method (View) IsScalar() bool
$GOPATH/srcgorgonia.org/tensor/interfaces.go:64.2-64.7: 	method (View) IsView() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:32.2-32.9: 	method (View) Iterator() Iterator
$GOPATH/srcgorgonia.org/tensor/interfaces.go:66.2-66.12: 	method (View) Materialize() Tensor
$GOPATH/srcgorgonia.org/tensor/tensor.go:59.2-59.8: 	method (View) MemSize() uintptr
$GOPATH/srcgorgonia.org/tensor/tensor.go:61.2-61.8: 	method (View) Pointer() unsafe.Pointer
$GOPATH/srcgorgonia.org/tensor/tensor.go:71.2-71.8: 	method (View) ReadNpy(io.Reader) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:31.2-31.17: 	method (View) RequiresIterator() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:38.2-38.8: 	method (View) Reshape(...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:53.2-53.12: 	method (View) ScalarValue() interface{}
$GOPATH/srcgorgonia.org/tensor/tensor.go:37.2-37.6: 	method (View) SetAt(v interface{}, coord ...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:23.2-23.6: 	method (View) Shape() Shape
$GOPATH/srcgorgonia.org/tensor/tensor.go:27.2-27.5: 	method (View) Size() int
$GOPATH/srcgorgonia.org/tensor/tensor.go:24.2-24.8: 	method (View) Strides() []int
$GOPATH/srcgorgonia.org/tensor/tensor.go:39.2-39.2: 	method (View) T(axes ...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:41.2-41.10: 	method (View) Transpose() error
$GOPATH/srcgorgonia.org/tensor/tensor.go:40.2-40.3: 	method (View) UT()
$GOPATH/srcgorgonia.org/tensor/tensor.go:60.2-60.8: 	method (View) Uintptr() uintptr
$GOPATH/srcgorgonia.org/tensor/tensor.go:70.2-70.9: 	method (View) WriteNpy(io.Writer) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:75.2-75.15: 	method (View) standardEngine() standardEngine

But Guru sees the Cloner interface in Tensor:

$GOPATH/srcgorgonia.org/tensor/tensor.go:21.6-21.11: definition of type Tensor (size 16, align 8)
$GOPATH/srcgorgonia.org/tensor/tensor.go:21.6-21.11: Methods:
$GOPATH/srcgorgonia.org/tensor/tensor.go:42.2-42.6: 	method (Tensor) Apply(fn interface{}, opts ...FuncOpt) (Tensor, error)
$GOPATH/srcgorgonia.org/tensor/tensor.go:36.2-36.3: 	method (Tensor) At(...int) (interface{}, error)
$GOPATH/srcgorgonia.org/tensor/interfaces.go:21.2-21.6: 	method (Tensor) Clone() interface{}
$GOPATH/srcgorgonia.org/tensor/interfaces.go:26.2-26.5: 	method (Tensor) Data() interface{}
$GOPATH/srcgorgonia.org/tensor/tensor.go:28.2-28.9: 	method (Tensor) DataSize() int
$GOPATH/srcgorgonia.org/tensor/tensor.go:26.2-26.5: 	method (Tensor) Dims() int
$GOPATH/srcgorgonia.org/tensor/tensor.go:25.2-25.6: 	method (Tensor) Dtype() Dtype
$GOPATH/srcgorgonia.org/tensor/tensor.go:58.2-58.7: 	method (Tensor) Engine() Engine
$GOPATH/srcgorgonia.org/tensor/interfaces.go:16.2-16.3: 	method (Tensor) Eq(interface{}) bool
/home/chewxy/.gvm/gos/master/src/fmt/print.go:54.2-54.7: 	method (gorgonia.org/tensor.Tensor) Format(f State, c rune)
/home/chewxy/.gvm/gos/master/src/encoding/gob/type.go:797.2-797.10: 	method (gorgonia.org/tensor.Tensor) GobDecode([]byte) error
/home/chewxy/.gvm/gos/master/src/encoding/gob/type.go:788.2-788.10: 	method (gorgonia.org/tensor.Tensor) GobEncode() ([]byte, error)
$GOPATH/srcgorgonia.org/tensor/tensor.go:63.2-63.18: 	method (Tensor) IsManuallyManaged() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:62.2-62.21: 	method (Tensor) IsNativelyAccessible() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:52.2-52.9: 	method (Tensor) IsScalar() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:32.2-32.9: 	method (Tensor) Iterator() Iterator
$GOPATH/srcgorgonia.org/tensor/tensor.go:59.2-59.8: 	method (Tensor) MemSize() uintptr
$GOPATH/srcgorgonia.org/tensor/interfaces.go:47.2-47.7: 	method (Tensor) Memset(interface{}) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:61.2-61.8: 	method (Tensor) Pointer() unsafe.Pointer
$GOPATH/srcgorgonia.org/tensor/tensor.go:71.2-71.8: 	method (Tensor) ReadNpy(io.Reader) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:31.2-31.17: 	method (Tensor) RequiresIterator() bool
$GOPATH/srcgorgonia.org/tensor/tensor.go:38.2-38.8: 	method (Tensor) Reshape(...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:53.2-53.12: 	method (Tensor) ScalarValue() interface{}
$GOPATH/srcgorgonia.org/tensor/tensor.go:37.2-37.6: 	method (Tensor) SetAt(v interface{}, coord ...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:23.2-23.6: 	method (Tensor) Shape() Shape
$GOPATH/srcgorgonia.org/tensor/tensor.go:27.2-27.5: 	method (Tensor) Size() int
$GOPATH/srcgorgonia.org/tensor/interfaces.go:71.2-71.6: 	method (Tensor) Slice(...Slice) (View, error)
$GOPATH/srcgorgonia.org/tensor/tensor.go:24.2-24.8: 	method (Tensor) Strides() []int
/home/chewxy/.gvm/gos/master/src/fmt/print.go:63.2-63.7: 	method (gorgonia.org/tensor.Tensor) String() string
$GOPATH/srcgorgonia.org/tensor/tensor.go:39.2-39.2: 	method (Tensor) T(axes ...int) error
$GOPATH/srcgorgonia.org/tensor/tensor.go:41.2-41.10: 	method (Tensor) Transpose() error
$GOPATH/srcgorgonia.org/tensor/tensor.go:40.2-40.3: 	method (Tensor) UT()
$GOPATH/srcgorgonia.org/tensor/tensor.go:60.2-60.8: 	method (Tensor) Uintptr() uintptr
$GOPATH/srcgorgonia.org/tensor/tensor.go:70.2-70.9: 	method (Tensor) WriteNpy(io.Writer) error
$GOPATH/srcgorgonia.org/tensor/interfaces.go:37.2-37.5: 	method (Tensor) Zero()
$GOPATH/srcgorgonia.org/tensor/interfaces.go:137.2-137.4: 	method (Tensor) arr() array
$GOPATH/srcgorgonia.org/tensor/interfaces.go:138.2-138.7: 	method (Tensor) arrPtr() *array
$GOPATH/srcgorgonia.org/tensor/interfaces.go:133.2-133.4: 	method (Tensor) hdr() *gorgonia.org/tensor/internal/storage.Header
$GOPATH/srcgorgonia.org/tensor/tensor.go:75.2-75.15: 	method (Tensor) standardEngine() standardEngine

What I've Done:

I'm currently trying to replicate this, but I can't seem to find a replication in a separate file. Any ideas?

@bradfitz bradfitz changed the title Large Compositional Interface Methods Table is Borked (tip) go/types: Large Compositional Interface Methods Table is Borked (tip) Nov 13, 2017
@bradfitz bradfitz added this to the Go1.10 milestone Nov 13, 2017
@chewxy
Copy link
Author

@chewxy chewxy commented Nov 14, 2017

0d18875 is the specific cause of the issue of me not being able to run tests (previously go test doesn't call go vet). It would appear that go vet is the one that is reporting all these errors.

There appears to be something unholy happening when there are internal packages involved, or somehow there is an issue with the init order in go/types, or that there are way too many declarations after walking the nodes. (1 Cloner was seen, 7 View was seen, and 589 Tensor was seen)

I wrote a small program to print out what it sees:

var archSizes = types.SizesFor("gc", build.Default.GOARCH)

func check(pkgName string, fileset *token.FileSet, files []*ast.File) {
	var errs []error
	config := types.Config{
		Importer: importer.Default(),
		Error: func(err error) {
			errs = append(errs, err)
		},
		Sizes: archSizes,
	}

	defs := make(map[*ast.Ident]types.Object)
	uses := make(map[*ast.Ident]types.Object)
	selectors := make(map[*ast.SelectorExpr]*types.Selection)
	typs := make(map[ast.Expr]types.TypeAndValue)
	implicits := make(map[ast.Node]types.Object)
	info := &types.Info{
		Selections: selectors,
		Types:      typs,
		Defs:       defs,
		Uses:       uses,
		Implicits:  implicits,
	}
	_, err := config.Check(pkgName, fileset, files, info)
	log.Printf("ERROR %v", err)

	// inspection
	var buf bytes.Buffer
	fmt.Fprintf(&buf, "Types:\n")
	for n, o := range info.Types {
		if id, ok := n.(*ast.Ident); ok && o.IsType() {
			// if id.Name == "View"  {
			// if id.Name == "Cloner" {
			if id.Name == "Tensor" {
				fmt.Fprintf(&buf, "%v (%v, %v)\n", id.Name, id.NamePos, id.End())
				fmt.Fprintf(&buf, "UNDERLYING %v\n", o.Type.Underlying())
			}
		}
	}
	log.Println(buf.String())
}

I get the correct results (formatted nicely for github):

Tensor (80220, 80226)
UNDERLYING interface{
	Apply(fn interface{}, opts ...tensor.FuncOpt) (tensor.Tensor, error); 
	At(...int) (interface{}, error); 
	DataSize() int; 
	Dims() int; 
	Dtype() tensor.Dtype; 
	Engine() tensor.Engine; 
	IsManuallyManaged() bool; 
	IsNativelyAccessible() bool; 
	IsScalar() bool;
	Iterator() tensor.Iterator; 
	MemSize() uintptr; 
	Pointer() unsafe.Pointer; 
	ReadNpy(io.Reader) error; 
	RequiresIterator() bool; 
	Reshape(...int) error; 
	ScalarValue() interface{}; 
	SetAt(v interface{}, coord ...int) error; 
	Shape() tensor.Shape; 
	Size() int; 
	Strides() []int; 
	T(axes ...int) error; 
	Transpose() error; 
	UT(); 
	Uintptr() uintptr; 
	WriteNpy(io.Writer) error; 

	standardEngine() tensor.standardEngine; 
	tensor.Cloner; 
	tensor.Dataer; 
	tensor.Eq; 
	fmt.Formatter; 
	encoding/gob.GobDecoder;
	encoding/gob.GobEncoder; 
	tensor.MemSetter;
	tensor.Slicer; 
	fmt.Stringer; 
	tensor.Zeroer; 
	tensor.arrayer;
	 tensor.headerer
}
View (34380, 34384)
UNDERLYING interface{
	IsMaterializable() bool; 
	IsView() bool; 
	Materialize() tensor.Tensor; 
	tensor.Tensor
}
Cloner (462220, 462226)
UNDERLYING interface{
	Clone() interface{}
}

Perhaps a clue is that the Tensor type gets seen 589 times?

LMK if I can furnish with more information

Loading

@ianlancetaylor ianlancetaylor changed the title go/types: Large Compositional Interface Methods Table is Borked (tip) cmd/vet: fails to vet code that compiles successfully Nov 14, 2017
@chewxy
Copy link
Author

@chewxy chewxy commented Nov 14, 2017

OK, so I've figured out as much: Order of declaration matters (which in all practical sense in a multi-file program, is something you can't control). Here's the reproduction

Both program compiles AND runs as expected.

The difference is the location of

type Aaaer interface {
    Aaa(a, b Foo) Foo
}

When it's declared after everything has been declared, and when go/type checks, this is the sequence it will check it:

  1. typeExpr(Cloner)
  2. typeExpr(Barer) -> typeExpr(Bar) -> typeExpr(Foo) -> typeExpr(Cloner) (elided: typeExpr(Bar))
  3. typeExpr(Aaaer) -> typeExpr(Aaa) -> typeExpr(Foo) . Here, Foo had already been typed.

Therefore it works. go/types doesn't report any incomplete interfaces.

But in the case where it doesn't work, here's the sequence of typechecking:

  1. typeExpr(Aaaer) -> typeExpr(Foor) -> typeExpr(Barer) -> typeExpr(Bar)->typeExpr(Barer)...
  2. typeExpr(Bar) -- previously checked and incomplete
  3. typeExpr(foo) -- previously checked and complete
  4. typeExpr(Foo) -- previously checked and incomplete

this causes an error

Loading

@ianlancetaylor ianlancetaylor changed the title cmd/vet: fails to vet code that compiles successfully go/types: fails to fully collect methods from embedded interfaces Nov 14, 2017
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 14, 2017

Thanks. Here is another reproduction.

package p

func F(x I4) interface{} {
	return x.Method()
}

type Unused interface {
	RefersToI1(a I1)
}

type I1 interface {
	I2
	I3
}

type I2 interface {
	RefersToI4() I4
}

type I3 interface {
	Method() interface{}
}

type I4 interface {
	I1
}

Loading

@griesemer
Copy link
Contributor

@griesemer griesemer commented Nov 14, 2017

Most likely a duplicate of #18395.

Loading

@griesemer
Copy link
Contributor

@griesemer griesemer commented Nov 21, 2017

Confirmed duplicate of #18395 by running with https://go-review.googlesource.com/c/go/+/78955 in debug mode.

Loading

@griesemer griesemer closed this Nov 21, 2017
@golang golang locked and limited conversation to collaborators Nov 21, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants