Vendoring gopherjs code causes generated js to fail

[benechols-wf]

I have a piece of go that imports gopherjs

package main

import (
    "github.com/gopherjs/gopherjs/js"
)

func main() {
    js.Global.Set("mypackage", map[string]interface{}{
        "DoSomething":         DoSomething,
    })

    println("gopher bindings loaded")
}

func DoSomething() {
  print("done!")
}

I then vendor gopherjs and try to do a gopherjs build. the command succeeds but the generated code fails as soon as it tries to use 'js.Global.Set' because of nil pointer.

If I then delete the vendor/ directory and rebuild everything works fine.

Is there something else I should be vendoring in order for this to work? Or what is the expected workflow?

[dmitshur]

Thanks for reporting.

I'm guessing it happens because the github.com/gopherjs/gopherjs/js import path is special (just like the "C" import path that cgo uses), and we're probably not taking into account that it can change when gopherjs is vendored to be example.com/user/repo/vendor/github.com/gopherjs/gopherjs/js.

[benechols-wf]

Quick question related to this -- if I vendor the stdlib overrides(gopherjs/compiler/natives/...) into vendor/github.com/gopherjs/gopherjs will those be used instead of looking for them in $GOPATH ?

[neelance]

It will always use those form the GOPATH. Vendoring in this context not possible on a theoretical level, unless we come up with some special vendoring rule. E.g. when GopherJS wants to compile net/http, then this resolves to the normal package in GOROOT. There is no vendor directory in this context.

And yes, the github.com/gopherjs/gopherjs/js import path is special and if the package is vendored then it is not recognized any more. I am not sure if we should fix this, because it is mandatory that the versions of the js package and the used GopherJS compiler are the same. Vendoring the js package could make it go out of sync, which is dangerous.

[benechols-wf]

I want to vendor all of gopherjs and my build process will build gopherjs from the vendored version. Then there will never be mismatched versions :)

[benechols-wf]

I guess our build process could take the vendored version and copy it back into the GOPATH during the build but that seems bad to me.

[mpl]

@neelance As @benechols-wf notes, one hits that bug even when one wants to simply vendor the whole of github.com/gopherjs/gopherjs into a project (which should be a pretty common use case), which would not make github.com/gopherjs/gopherjs/js out of sync wrt to github.com/gopherjs/gopherjs.

So it should definitely be fixed imho.

[neelance]

This will probably still run into #462. Working on it.

[mpl]

@neelance jsyk, I was hoping f627518 and 2e16c68 had fixed this , so I've just retried, and it still does not seem to work for me.

Layout is like this:

$HOME/project/build-env/
    src/
        app/ui/gopherjs/main.go
        app/vendor/github.com/gopherjs/gopherjs
        app/vendor/github.com/gopherjs/gopherjs/js
    pkg/
    bin/

GOPATH=$HOME/project/build-env/

using gopherjs as:

$HOME/project/build-env/bin/gopherjs build -m -o $HOME/project/build-env/app/ui/gopherjs.js app/ui/gopherjs

and getting this error:

exit status 1, cannot find package "github.com/gopherjs/gopherjs/js" in any of:
    /home/mpl/go1/src/vendor/github.com/gopherjs/gopherjs/js (vendor tree)
    /home/mpl/go1/src/github.com/gopherjs/gopherjs/js (from $GOROOT)
    /home/mpl/project/build-env/src/github.com/gopherjs/gopherjs/js (from $GOPATH)
exit status 1

So it seems like it's still not looking for github.com/gopherjs/gopherjs/js in the vendor dir of my project.

[nmiyake]

I ran into this as well and was interested in fixing this so made a fix locally. It can still use some iteration, but wanted to verify that the general approach is acceptable.

The main issue seems to be resolving the github.com/gopherjs/gopherjs import path to the desired vendored import path when performing imports from a source path that is outside the project (as is done for the standard library generation). My proposed fix takes the following form:

• Determine if a project wants to use a vendored version of gopherjs -- if so, determine the import path to the vendored version
• In the importWithSrcDir function in build/build.go, if the requested import path starts with github.com/gopherjs/gopherjs, update that path with the vendored version determined above

This approach solves all of the import resolution. The last issue is that the natives version of runtime/runtime.go has a string constant that is explicitly set to github.com/gopherjs/gopherjs/js. However, build.go already has code that adds in the natives files and reads through the AST, so as long as we're okay with some tight coupling, we can update that code to have special logic that, when processing runtime/runtime.go, updates the value of the import path there to be the vendored one as well (it requires minor refactoring of runtime/runtime.go to extract this value to a constant, but that is trivial).

I tried this approach locally on one of my own projects that use gopherjs (made these changes to gopherjs, vendored it entirely, then generated code by running against the vendored version) and everything seemed to work -- the import paths were correct and the Javascript code worked.

The part of the solution that needs the most work right now is determining when gopherjs is vendored and setting the vendor path at the correct level -- for my prototype I just used a package-global variable for storage and set the value in the code that currently determines whether or not a package is vendored, but a real solution would need to do this more elegantly (and not quite sure where that logic should go).

PR for this is at #678 -- feedback very much appreciated.

[dmitshur]

Thanks a lot for looking at this @nmiyake and sending a PR. It's very valuable for us to know the cause of the issue and what it takes to resolve it, so thanks for doing the investigation. I'll be able to review the PR shortly.

[hanleym]

@shurcooL Any update on this?

[dmitshur]

There are no updates from my side. My last comment on the PR said:

Unfortunately, I don't have an opportunity to review it soon.

That continues to be true, unfortunately.

[nathanhack]

@shurcooL I'm sure you're still busy, but I was curious if there was any update for this issue?

[dmitshur]

Now that GopherJS 1.10-1 is out, I'll be prioritizing resolving this issue in the coming weeks. I think it's becoming more important to be able to vendor GopherJS, and be able to use fixed versions of it (rather than always counting on the latest).

The PR just needs to be reviewed and tested.

[bradfitz]

Thanks!

[nathanhack]

@shurcooL That would be fantastic! Thank you.

[nmiyake]

Great! I should be able to iterate on or answer any questions about the PR once it gets a chance to be reviewed (I see that there are some merge conflicts now -- I'll take a look at fixing those after receiving initial feedback).

[bradfitz]

@shurcooL, this is holding up Perkeep getting rid of its hacky make.go wrapper that builds a temp dir parallel GOPATH to work around pre-Go 1.10 cmd/go bugs. We want to just use Go 1.10 without GOPATH tempdir hackery.

Can we incentivize you to fix this? We have some budget (https://opencollective.com/perkeep) and could pay for your time.

[dmitshur]

Can we incentivize you to fix this?

You already have! I will make this issue my priority for the coming weekend.

this is holding up Perkeep getting rid of its hacky make.go wrapper that builds a temp dir parallel GOPATH to work around pre-Go 1.10 cmd/go bugs. We want to just use Go 1.10 without GOPATH tempdir hackery.

You should be able to use Go 1.10 even before this is resolved. I'll want to chime in on that issue when I'm working on this.

We have some budget (https://opencollective.com/perkeep) and could pay for your time.

Thank you very much, but I'd be worried that I only did a part of all the work required to resolve this issue (I've never been paid to work on an open source issue before, so I'm not sure how to deal with it). Maybe you'd want to buy me a coffee instead. But this weekend is still the soonest I can get to it, because I'll need a large continuous time block.

[dmitshur]

I've created PR #787 that should resolve this issue.

In my testing, I was able to vendor the entire GopherJS project in another Go project, and then used it to successfully build Go code (including Go code that imports github.com/gopherjs/gopherjs/js package). This was done without the (normal, non-vendored) GopherJS repository in GOPATH.

I think everything should work, but it's possible I'm not aware of some other ways people are vendoring GopherJS. Please feel free to test the PR and let me know if vendoring doesn't work for you.