Using go-yaml/yaml fails with "(reflect.Type).Elem() called on map type"

[mengqiy]

It seems tinygo's reflect package is not fully compatible with golang's native reflect package.

I have a golang program. It can be compiled to binary with go build. I tried to use tinygo to compile it to WASM, I run into the following issue:

$ tinygo build -o set-ns.wasm -target wasi ./main.go
# sigs.k8s.io/kustomize/kyaml/internal/forked/github.com/go-yaml/yaml
/usr/local/lib/tinygo/src/reflect/type.go:385:15: (reflect.Type).Elem() called on map type
  %0 = call i32 @"(reflect.rawType).elem"(i32 %t, i8* undef), !dbg !3004

traceback:
/usr/local/lib/tinygo/src/reflect/type.go:385:15:
  %0 = call i32 @"(reflect.rawType).elem"(i32 %t, i8* undef), !dbg !3004
/usr/local/lib/tinygo/src/reflect/type.go:384:18:
  %ret = call %runtime._interface @"(reflect.rawType).Elem"(i32 %unpack.int, i8* %1), !dbg !3000
/usr/local/google/home/mengqiy/golang/src/sigs.k8s.io/kustomize/kyaml/internal/forked/github.com/go-yaml/yaml:
  %15 = call %runtime._interface @"interface:{Align:func:{}{basic:int},AssignableTo:func:{named:reflect.Type}{basic:bool},Bits:func:{}{basic:int},ChanDir:func:{}{named:reflect.ChanDir},Comparable:func:{}{basic:bool},ConvertibleTo:func:{named:reflect.Type}{basic:bool},Elem:func:{}{named:reflect.Type},Field:func:{basic:int}{named:reflect.StructField},FieldAlign:func:{}{basic:int},FieldByName:func:{basic:string}{named:reflect.StructField,basic:bool},Implements:func:{named:reflect.Type}{basic:bool},In:func:{basic:int}{named:reflect.Type},IsVariadic:func:{}{basic:bool},Key:func:{}{named:reflect.Type},Kind:func:{}{named:reflect.Kind},Len:func:{}{basic:int},MethodByName:func:{basic:string}{named:reflect.Method,basic:bool},Name:func:{}{basic:string},NumField:func:{}{basic:int},NumIn:func:{}{basic:int},NumMethod:func:{}{basic:int},NumOut:func:{}{basic:int},Out:func:{basic:int}{named:reflect.Type},PkgPath:func:{}{basic:string},Size:func:{}{basic:uintptr},String:func:{}{basic:string}}.Elem$invoke"(i8* %14, i32 %invoke.func.typecode, i8* undef), !dbg !3001

Stacktrace shows there is something wrong here.

The error message comes from here.

[martinlindhe]

I just came here to report the same.

It is triggered by gopkg.in/yaml.v2 aka https://github.com/go-yaml/yaml

Here is output on my machine, tinygo 0.24:

ld.lld: warning: duplicate /export option: hypot
ld.lld: warning: duplicate /export option: nextafter
# gopkg.in/yaml.v2
C:\Users\marti\scoop\apps\tinygo\current\src\reflect\type.go:385:15: (reflect.Type).Elem() called on map type
  <badref> = callException 0xc0000005 0x0 0x288637fbe10 0x7ff73b66c637
PC=0x7ff73b66c637
signal arrived during external code execution

runtime.cgocall(0x7ff73ac9d7d0, 0xc009f1f4e0)
        C:/hostedtoolcache/windows/go/1.18.1/x64/src/runtime/cgocall.go:157 +0x4a fp=0xc009f1f4b8 sp=0xc009f1f480 pc=0x7ff73a9246ea
tinygo.org/x/go-llvm._Cfunc_LLVMDumpValue(0x28865592eb0)
        _cgo_gotypes.go:5189 +0x52 fp=0xc009f1f4e0 sp=0xc009f1f4b8 pc=0x7ff73ab52cf2
tinygo.org/x/go-llvm.Value.Dump.func1({0x1?})
        C:/Users/runneradmin/go/pkg/mod/tinygo.org/x/go-llvm@v0.0.0-20220626113704-45f1e2dbf887/ir.go:712 +0x3f fp=0xc009f1f518 sp=0xc009f1f4e0 pc=0x7ff73ab65c5f
tinygo.org/x/go-llvm.Value.Dump({0xc011a00020?})
        C:/Users/runneradmin/go/pkg/mod/tinygo.org/x/go-llvm@v0.0.0-20220626113704-45f1e2dbf887/ir.go:712 +0x19 fp=0xc009f1f530 sp=0xc009f1f518 pc=0x7ff73ab65bf9
main.printCompilerError(0x7ff73f06d910, {0x7ff73f0da920?, 0xc010762e00?})
        D:/a/tinygo/tinygo/main.go:1170 +0x4d8 fp=0xc009f1f7c8 sp=0xc009f1f530 pc=0x7ff73ac8e4f8
main.handleCompilerError({0x7ff73f0da920?, 0xc010762e00?})
        D:/a/tinygo/tinygo/main.go:1203 +0x37 fp=0xc009f1f7f0 sp=0xc009f1f7c8 pc=0x7ff73ac8ee17
main.main()
        D:/a/tinygo/tinygo/main.go:1511 +0x20ea fp=0xc009f1ff80 sp=0xc009f1f7f0 pc=0x7ff73ac916aa
runtime.main()
        C:/hostedtoolcache/windows/go/1.18.1/x64/src/runtime/proc.go:250 +0x1fe fp=0xc009f1ffe0 sp=0xc009f1ff80 pc=0x7ff73a959d9e
runtime.goexit()
        C:/hostedtoolcache/windows/go/1.18.1/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc009f1ffe8 sp=0xc009f1ffe0 pc=0x7ff73a9854e1
rax     0x53
rbx     0x288637fbe00
rcx     0x920efff1f8
rdi     0x920efff1e0
rsi     0x0
rbp     0xc009f1f4a8
rsp     0x920efff1a0
r8      0x60
r9      0x920effec60
r10     0x0
r11     0x246
r12     0x28865592eb0
r13     0x920efff4e0
r14     0x920efff200
r15     0x920efff1f8
rip     0x7ff73b66c637
rflags  0x10246
cs      0x33
fs      0x53
gs      0x2b

tinygo 0.23:

ld.lld: warning: duplicate /export option: hypot
ld.lld: warning: duplicate /export option: nextafter
# gopkg.in/yaml.v2
C:\Users\marti\scoop\apps\tinygo\current\src\reflect\type.go:385:15: (reflect.Type).Elem() called on map type
  %0 = call i64 @"(reflect.rawType).elem"(i64 %t, i8* undef), !dbg !3987

traceback:
C:\Users\marti\scoop\apps\tinygo\current\src\reflect\type.go:385:15:
  %0 = call i64 @"(reflect.rawType).elem"(i64 %t, i8* undef), !dbg !3987
C:\Users\marti\scoop\apps\tinygo\current\src\reflect\type.go:384:18:
  %ret = call %runtime._interface @"(reflect.rawType).Elem"(i64 %unpack.int, i8* %1), !dbg !3983
C:\Users\marti\go\pkg\mod\gopkg.in\yaml.v2@v2.4.0:
  %6 = call %runtime._interface @"interface:{Align:func:{}{basic:int},AssignableTo:func:{named:reflect.Type}{basic:bool},Bits:func:{}{basic:int},ChanDir:func:{}{named:reflect.ChanDir},Comparable:func:{}{basic:bool},ConvertibleTo:func:{named:reflect.Type}{basic:bool},Elem:func:{}{named:reflect.Type},Field:func:{basic:int}{named:reflect.StructField},FieldAlign:func:{}{basic:int},FieldByName:func:{basic:string}{named:reflect.StructField,basic:bool},Implements:func:{named:reflect.Type}{basic:bool},In:func:{basic:int}{named:reflect.Type},IsVariadic:func:{}{basic:bool},Key:func:{}{named:reflect.Type},Kind:func:{}{named:reflect.Kind},Len:func:{}{basic:int},MethodByName:func:{basic:string}{named:reflect.Method,basic:bool},Name:func:{}{basic:string},NumField:func:{}{basic:int},NumIn:func:{}{basic:int},NumMethod:func:{}{basic:int},NumOut:func:{}{basic:int},Out:func:{basic:int}{named:reflect.Type},PkgPath:func:{}{basic:string},Size:func:{}{basic:uintptr},String:func:{}{basic:string}}.Elem$invoke"(i8* %invoke.func.value, i64 %invoke.func.typecode, i8* undef), !dbg !3984

[mengqiy]

It is triggered by gopkg.in/yaml.v2 aka https://github.com/go-yaml/yaml

The issue I have is triggered by this library too. We are using a forked version of https://github.com/go-yaml/yaml. But I believe there is same issue with gopkg.in/yaml.v2 and gopkg.in/yaml.v3.

[mengqiy]

Just tried with the new release v0.24.0 and got a similar error.

$ tinygo build -o set-ns.wasm -target wasi ./main.go
# sigs.k8s.io/kustomize/kyaml/internal/forked/github.com/go-yaml/yaml                                                                                                                                                                                                                
/usr/local/lib/tinygo/src/reflect/type.go:385:15: (reflect.Type).Elem() called on map type                                                
  <badref> = callfatal error: unexpected signal during runtime execution                                                                                                                                                                                                             
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4f86cb9]
                                                                                                                                          
runtime stack:                                                                                                                            
runtime.throw({0x53b25d1?, 0x6583748?})                                                                                                   
        /opt/hostedtoolcache/go/1.18.1/x64/src/runtime/panic.go:992 +0x71                                                                 
runtime.sigpanic()                                                                                                                        
        /opt/hostedtoolcache/go/1.18.1/x64/src/runtime/signal_unix.go:802 +0x3a9
                                                                                                                                          
goroutine 1 [syscall]:                                                                                                                    
runtime.cgocall(0xe5e0f0, 0xc00cc774b8)                                                                                                   
        /opt/hostedtoolcache/go/1.18.1/x64/src/runtime/cgocall.go:157 +0x5c fp=0xc00cc77490 sp=0xc00cc77458 pc=0xaff23c
tinygo.org/x/go-llvm._Cfunc_LLVMDumpValue(0x7fb40c0a9c40)
        _cgo_gotypes.go:5196 +0x45 fp=0xc00cc774b8 sp=0xc00cc77490 pc=0xd15985
tinygo.org/x/go-llvm.Value.Dump.func1({0x1?})
        /home/runner/go/pkg/mod/tinygo.org/x/go-llvm@v0.0.0-20220626113704-45f1e2dbf887/ir.go:712 +0x3f fp=0xc00cc774f0 sp=0xc00cc774b8 pc=0xd26f1f
tinygo.org/x/go-llvm.Value.Dump({0xc00d4397a0?})
        /home/runner/go/pkg/mod/tinygo.org/x/go-llvm@v0.0.0-20220626113704-45f1e2dbf887/ir.go:712 +0x19 fp=0xc00cc77508 sp=0xc00cc774f0 pc=0xd26eb9
main.printCompilerError(0x53bb058, {0x541c980?, 0xc00ba8f730?})
        /home/runner/work/tinygo/tinygo/main.go:1170 +0x4d8 fp=0xc00cc777c0 sp=0xc00cc77508 pc=0xe3bed8
main.handleCompilerError({0x541c980?, 0xc00ba8f730?})
        /home/runner/work/tinygo/tinygo/main.go:1203 +0x37 fp=0xc00cc777e8 sp=0xc00cc777c0 pc=0xe3c7f7
main.main()
        /home/runner/work/tinygo/tinygo/main.go:1444 +0x3b9a fp=0xc00cc77f80 sp=0xc00cc777e8 pc=0xe40b3a
runtime.main()
        /opt/hostedtoolcache/go/1.18.1/x64/src/runtime/proc.go:250 +0x212 fp=0xc00cc77fe0 sp=0xc00cc77f80 pc=0xb316d2
runtime.goexit()
        /opt/hostedtoolcache/go/1.18.1/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00cc77fe8 sp=0xc00cc77fe0 pc=0xb5eca1

[dkegel-fastly]

This is not unexpected.
Perhaps the warning about reflect in https://tinygo.org/docs/reference/lang-support/ should be stronger.
It is hard to predict when tinygo's reflect will support maps better.
#2640 might be a step towards a fix, but it's a big job.

https://tinygo.org/docs/reference/lang-support/stdlib/ shows many standard library packages fail to pass tests due to reflect being incomplete in tinygo.
#2566 is related.

And running https://github.com/go-yaml/yaml's tests in tinygo fails with:

../../go/pkg/mod/gopkg.in/check.v1@v0.0.0-20161208181325-20d25e280405/check.go:56:30: receiver.Method undefined (type reflect.Value has no field or method Method)
../../go/pkg/mod/gopkg.in/check.v1@v0.0.0-20161208181325-20d25e280405/check.go:56:57: receiver.Type().Method undefined (type reflect.Type has no field or method Method)

[dgryski]

As for the specific panic in this case, the we do not encode the map's full type into a side-table in the binary that would allow us to get the key and element types.

[dgryski]

The go-yaml/yaml tests won't pass because they rely on Method functions, which we don't support for reflect. However basic yaml support should be working on the last few PRs are merged. You can try it now by checking out this branch: https://github.com/dgryski/tinygo/tree/dgryski/reflect-all-fixes-3

Running the go-yaml/yaml example from https://github.com/go-yaml/yaml#example gives:

~/go/src/github.com/dgryski/bug/yml $ tinygo run main.go
--- t:
{Easy! {2 [3 4]}}

--- t dump:
a: Easy!
b:
    c: 2
    d: [3, 4]


--- m:
map[a:Easy! b:map[c:2 d:[3 4]]]

--- m dump:
a: Easy!
b:
    c: 2
    d:
        - 3
        - 4

[daveshanley]

Thank you for working on this. I want to use tinygo to compile my app, but it heavily depends on that yaml library. Thank you for all your effort. I've been watching this thread, waiting to see this.

[deadprogram]

This is part of the v0.28 release so now closing this issue. Thanks!