[Question]tinygo build wasm error when use protobuf #1879

zhenjunMa · 2021-05-13T02:55:18Z

Question

when i use struct compiled by protobuf, and use tinygo build to a wasm file, i got an error.

How to reproduce

model.proto

syntax = "proto3";

package model;

message Request {
	string name = 1;
}

build command
protoc -I. --go_out=. model.proto

hello.go

package main

import (
	"TinyGoProtoBuf/model"
)

func main() {
	_ = &model.Request{
		Name: "a",
	}
}

build command:
tinygo build -o wasm.wasm -target wasm ./hello.go

error:

i have uploaded the total project: https://github.com/zhenjunMa/TinyGoProtoBuf

any idea about how to solve this? or tinygo can not support protobuf now.

The text was updated successfully, but these errors were encountered:

zhenjunMa · 2021-05-13T06:04:21Z

hello.go

package main

import (
	"encoding/json"
	"fmt"
)

type request struct {
	name string `json:"name"`
}

func main() {
	req := &request{
		name: "hello",
	}

	b, _ := json.Marshal(req)

	fmt.Println(len(b))

}

i tried to use json like above, but still got an error:

so, if i can use any other way to serialize a struct ?

dengliangjun · 2021-05-17T02:46:50Z

I met those compiled errors as the same with yours
You should update tinygo version to 0.18.
However, Maybe you will meet new errors as the same with mine.
i.e.
xxxx/protobuf/proto/clone.go : AddpendSlice not declared by package reflect
xxxx/protobuf/proto/discard.go : t.FieldByName undefined ( type reflect.Type has no field or method FieldByName)
xxxx......
xxxx.....

If you solve this problem, Please tell me. we can solve this problem jointly.
email: dengliangjun@skycto.com

aykevl · 2021-05-19T11:58:48Z

Some of these issues are fixed in the latest release. Can you try again with the latest release?

dengliangjun · 2021-05-19T12:21:59Z

Some of these issues are fixed in the latest release. Can you try again with the latest release?

Hi, I need your help, My Env :
tinygo=0.18.0
go=1.13
target=wasm
my source code like ：
struct Order {
Name string ‘json:name’
Price int 'json:price'
}
bytes:=[b'{',b‘}’]] // empty json "{}"
json.Unmarshal(bytes,&order)

get a runtime error:
panic: unimplemented : (reflect.Type).Name()

Whether My go version is too low.
My source code can run within 'go build'

zhenjunMa · 2021-05-20T03:09:55Z

@aykevl thank you for you reply.

as @dengliangjun said, after i upgrade tinygo to 1.18.0

when i used protobuf to serialize a struct, i got an error:

github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

when i used json to serialize a struct, i got an another error:

panic: unimplemented: (reflect.Type).Name()

the difference is for protobuf, the error is happened at compiled time, but for json, the error is happened at runtime 😂

dengliangjun · 2021-05-20T03:17:26Z

the same errors as you, me too! I want to solve/contribute this issue. Whether there is anyone else can direct me. 来自skycto.com企业邮箱------------------------------------------------------------------ ***@***.***> 日　期：2021年05月20日 11:10:12 ***@***.***> ***@***.***>; ***@***.***> 主　题：Re: [tinygo-org/tinygo] [Question]tinygo build wasm error when use protobuf (#1879) @aykevl thank you for you reply. as @deadprogram said, after i upgrade tinygo to 1.18.0 when i used protobuf to serialize a struct, i got an error: github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect when i used json to serialize a struct, i got an another error: panic: unimplemented: (reflect.Type).Name() the difference is for protobuf, the error is happened at compiled time, but for json, the error is happened at runtime 😂 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

hamidrezakks · 2021-05-22T14:31:45Z

Well I had the same problem, i highly suggest to use go go proto instead.
https://github.com/gogo/protobuf

Make sure you remove extra methods from generated .pb.go files and just keep marshal and unmarshal methods

aykevl · 2021-05-22T15:29:40Z

when i used json to serialize a struct, i got an another error:

panic: unimplemented: (reflect.Type).Name()

Yes, I'm aware of this issue. It is something that I want to fix in the reflect package. Unfortunately there is no easy fix.

aykevl · 2021-05-22T16:41:03Z

when i used protobuf to serialize a struct, i got an error:

github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

This should be fixed with this PR: #1902

dengliangjun · 2021-05-24T04:09:02Z

Well I had the same problem, i highly suggest to use go go proto instead.
https://github.com/gogo/protobuf

Make sure you remove extra methods from generated .pb.go files and just keep marshal and unmarshal methods
thanks for your suggestions and help.
have you test your approach?
can you share your code within gogo/protobuf, which can be compiled with tinygo.

zhenjunMa · 2021-05-25T04:05:43Z

when i used protobuf to serialize a struct, i got an error:
github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

This should be fixed with this PR: #1902

@aykevl thank you for your attention about this issue.

I have tried to compile the latest dev branch which maybe fixed my question, the compiled result is below:

It's my bad that I didn't describe the problem clearly before, so let me redo this.

use tinyGo 0.18.0 to compile a go file that used protobuf to serialize a struct, the full error stack is:

➜  wasm git:(wasm) ✗ tinygo build -o wasm.wasm -scheduler=none -target=wasi -wasm-abi=generic -tags 'abi_010' wasm.go
# github.com/golang/protobuf/proto
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/clone.go:222:20: AppendSlice not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/pointer_unsafe.go:287:17: NewAt not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field o

use tinyGo 0.19.0-dev to compile the same go file, the full error stack is:

➜  wasm git:(wasm) ✗ tinygo build -o wasm.wasm -scheduler=none -target=wasi -wasm-abi=generic -tags 'abi_010' wasm.go
# github.com/golang/protobuf/proto
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/pointer_unsafe.go:287:17: NewAt not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field o

PR: #1902 really solve the AppendSlice problem, but still can't compile protobuf to wasm.

@hamidrezakks

I also tried to use gogoprotobuf, but no matter 0.18.0 or 0.19.0-dev, it can't work yet, the error is same with above.

could you share your code?

hamidrezakks · 2021-05-30T07:34:11Z

Hi @zhenjunMa

Here is one of our protos

// MessageEnvelope
// This type of message will be used to contain another ProtoBuffer Message inside
message MessageEnvelope {
    int64 Constructor = 1;
    fixed64 RequestID = 2;
    bytes Message = 4;
    bytes Auth = 8;
    repeated KeyValue Header = 10;
}

the generated gogo file has many functions just keep these

func (m *MessageEnvelope) Marshal() (dAtA []byte, err error)
func (m *MessageEnvelope) MarshalTo(dAtA []byte) (int, error)
func (m *MessageEnvelope) MarshalToSizedBuffer(dAtA []byte) (int, error)

and the main struct which is

type MessageEnvelope struct {
	Constructor int64       `protobuf:"varint,1,opt,name=Constructor,proto3" json:"Constructor,omitempty"`
	RequestID   uint64      `protobuf:"fixed64,2,opt,name=RequestID,proto3" json:"RequestID,omitempty"`
	Message     []byte      `protobuf:"bytes,4,opt,name=Message,proto3" json:"Message,omitempty"`
	Auth        []byte      `protobuf:"bytes,8,opt,name=Auth,proto3" json:"Auth,omitempty"`
	Header      []*KeyValue `protobuf:"bytes,10,rep,name=Header,proto3" json:"Header,omitempty"`
}

in my case.

for generating proto.bp.go files use this command
protoc -I=$GOPATH/src -I=./msg --gogofaster_out=./msg ./msg/*.proto

alanpoon · 2021-06-03T03:22:57Z

Here is my script to aggressively strip lines https://gist.github.com/alanpoon/541ceea2129412b5a5ae046ad90fccd6
to make it compile into wasm.

aykevl · 2021-06-13T13:03:47Z

I am working on refactoring the reflect package to be more powerful. This is still a work in progress, but once completed I think it will be feasible to implement the remaining features necessary for protobuf.

fredgoya · 2022-09-23T17:35:41Z

Is this (the reflect package work and the protobuf support) still WIP? I'd love for the proxy-wasm-go issue to move forward but it's depending on this ticket.

andresperezl · 2023-07-19T14:54:28Z

Using tinygo 0.28.1 to compile a wasm/wasi binary that uses protobuf, and using wasmedge to run it, I see the following error:

panic: unimplemented: reflect.New()

Which was strange as reflect.New() seems to be implemented. But looking for that text in the code further down below I found it at reflect.NewAt():

func NewAt(typ Type, p unsafe.Pointer) Value {
	panic("unimplemented: reflect.New()")
}

And looking at the protobuf-go code you can find a call to it here: https://github.com/protocolbuffers/protobuf-go/blob/master/internal/impl/pointer_unsafe.go#L78

deadprogram added the wasm WebAssembly label May 14, 2021

aykevl mentioned this issue May 22, 2021

Implement reflect.AppendSlice #1902

Merged

mathetake mentioned this issue Jun 15, 2021

gRPC callouts support tetratelabs/proxy-wasm-go-sdk#14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question]tinygo build wasm error when use protobuf #1879

[Question]tinygo build wasm error when use protobuf #1879

zhenjunMa commented May 13, 2021

zhenjunMa commented May 13, 2021

dengliangjun commented May 17, 2021

aykevl commented May 19, 2021

dengliangjun commented May 19, 2021

zhenjunMa commented May 20, 2021 •

edited

Loading

dengliangjun commented May 20, 2021 via email

hamidrezakks commented May 22, 2021 •

edited

Loading

aykevl commented May 22, 2021

aykevl commented May 22, 2021

dengliangjun commented May 24, 2021

zhenjunMa commented May 25, 2021

hamidrezakks commented May 30, 2021 •

edited

Loading

alanpoon commented Jun 3, 2021 •

edited

Loading

aykevl commented Jun 13, 2021

fredgoya commented Sep 23, 2022

andresperezl commented Jul 19, 2023

[Question]tinygo build wasm error when use protobuf #1879

[Question]tinygo build wasm error when use protobuf #1879

Comments

zhenjunMa commented May 13, 2021

Question

How to reproduce

zhenjunMa commented May 13, 2021

dengliangjun commented May 17, 2021

aykevl commented May 19, 2021

dengliangjun commented May 19, 2021

zhenjunMa commented May 20, 2021 • edited Loading

dengliangjun commented May 20, 2021 via email

hamidrezakks commented May 22, 2021 • edited Loading

aykevl commented May 22, 2021

aykevl commented May 22, 2021

dengliangjun commented May 24, 2021

zhenjunMa commented May 25, 2021

hamidrezakks commented May 30, 2021 • edited Loading

alanpoon commented Jun 3, 2021 • edited Loading

aykevl commented Jun 13, 2021

fredgoya commented Sep 23, 2022

andresperezl commented Jul 19, 2023

zhenjunMa commented May 20, 2021 •

edited

Loading

hamidrezakks commented May 22, 2021 •

edited

Loading

hamidrezakks commented May 30, 2021 •

edited

Loading

alanpoon commented Jun 3, 2021 •

edited

Loading