Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Go module support #941

Merged
merged 5 commits into from
May 27, 2020
Merged

Go module support #941

merged 5 commits into from
May 27, 2020

Conversation

aykevl
Copy link
Member

@aykevl aykevl commented Mar 5, 2020 •
edited

This is more of a mockup than anything serious. I want to see whether creating a new GOROOT per build is feasible. Right now creating such a GOROOT costs about 4ms on my system (after a warmed-up cache) so is not a significant overhead per build.

I'm not sure whether the current approach (with symlinks) is going to be viable on Windows.

The first commit adds the go list command, which might be enough to support TinyGo in IDEs such as VS Code.

@bradleypeabody bradleypeabody mentioned this pull request Apr 13, 2020
Closed
@deadprogram deadprogram mentioned this pull request Apr 20, 2020
Closed
@aykevl aykevl force-pushed the mod branch 15 times, most recently from b98a76c to 7301dfc Compare May 8, 2020 13:48
@aykevl aykevl changed the title WIP: experimenting with Go modules support Go module support May 8, 2020
@aykevl
Copy link
Member Author

aykevl commented May 8, 2020 •
edited

  • Implemented loading packages using x/tools/go/packages.
  • Implemented a workaround for Windows systems, for when developer mode is not enabled.
  • Fixed error messages to use the real paths (currently always absolute paths, this might need fixing).

This PR is now ready for review.

The first two commits get things ready, they are not directly related to Go module support. The third switches to a custom GOROOT which is cached between invocations. The fourth actually replaces the custom loader with (indirectly) go list, which is Go module aware.

@aykevl aykevl mentioned this pull request May 8, 2020
Closed
@aykevl aykevl force-pushed the mod branch 2 times, most recently from 4ae3f73 to f7b2dc0 Compare May 15, 2020 21:01
@aykevl aykevl mentioned this pull request May 16, 2020
Closed
@deadprogram
Copy link
Member

Hi @aykevl this PR has a merge conflict to resolve, please.

@aykevl
Copy link
Member Author

aykevl commented May 20, 2020

Fixed the merge conflict.

Additionally, by adding the TinyGo version to the cache key it should be very unlikely that the wrong cache key is used in releases. During development, it may still be necessary to run tinygo clean with some changes (such as changes to the syscall package).

@deadprogram
Copy link
Member

Yet another merge conflict @aykevl can you please resolve? Thanks.

@aykevl
builder: move Go version code to goenv package
d9e7a1b 
This is necessary to avoid a circular dependency in the loader (which
soon will need to read the Go version) and because it seems like a
better place anyway.
@aykevl
Copy link
Member Author

aykevl commented May 22, 2020

Done. This conflict was probably introduced with #1125.

@deadprogram
Copy link
Member

Seems like something went wrong with the linux build? https://app.circleci.com/pipelines/github/tinygo-org/tinygo/2972/workflows/e8dc6e29-f0e7-4ad7-90a9-0ab20ec9b77c/jobs/13574/steps

@deadprogram
Copy link
Member

Here is the specific problem:

fpm -f -s dir -t deb -n tinygo -v  -m '@tinygo-org' --description='TinyGo is a Go compiler for small places.' --license='BSD 3-Clause' --url=https://tinygo.org/ --deb-changelog CHANGELOG.md -p build/release.deb -C ./build/release-deb
 
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
 
Require just the needed backports instead, or 'backports/latest'.
 
All flags should be before the first argument (stray flags found: ["--description=TinyGo is a Go compiler for small places.", "--license=BSD 3-Clause", "--url=https://tinygo.org/", "--deb-changelog", "-p", "-C"] {:level=>:warn}
 
Invalid package configuration: Cannot package the path './@tinygo-org', does it exist? {:level=>:error}
 
Makefile:353: recipe for target 'deb' failed
 
make: *** [deb] Error 1
 
make: *** Waiting for unfinished jobs....

@aykevl
main: move TinyGo version to goenv
0319368 
This is needed to make it available to more packages, for caching
purposes.

For caching, the version itself may not be enough during development.
But for regular releases, the version provides some protection against
accidentally using a cache entry that is invalid in a newer version.
@aykevl
main: fix test subcommand
4e264dd 
It was broken for quite some time without anybody noticing...
@aykevl
loader: merge roots from both Go and TinyGo in a cached directory
95ec745 
This commit changes the way that packages are looked up. Instead of
working around the loader package by modifying the GOROOT variable for
specific packages, create a new GOROOT using symlinks. This GOROOT is
cached for the specified configuration (Go version, underlying GOROOT
path, TinyGo path, whether to override the syscall package).

This will also enable go module support in the future.

Windows is a bit harder to support, because it only allows the creation
of symlinks when developer mode is enabled. This is worked around by
using symlinks and if that fails, using directory junctions or hardlinks
instead. This should work in the vast majority of cases. The only case
it doesn't work, is if developer mode is disabled and TinyGo, the Go
toolchain, and the cache directory are not all on the same filesystem.
If this is a problem, it is still possible to improve the code by using
file copies instead.

As a side effect, this also makes diagnostics use a relative file path
only when the file is not in GOROOT or in TINYGOROOT.
@aykevl
loader: load packages using Go modules
ba9741f 
This commit replaces the existing ad-hoc package loader with a package
loader that uses the x/tools/go/packages package to find all
to-be-loaded packages.
@aykevl
Copy link
Member Author

aykevl commented May 23, 2020

I hope I really fixed it this time.

@aykevl
Copy link
Member Author

aykevl commented May 23, 2020

Finally the tests pass! (Although TinyHCI appears to be stuck?)

@deadprogram
Copy link
Member

All the tests are passing now.

@deadprogram
Copy link
Member

I am just wondering how to test this exactly. @aykevl can you please explain?

@aykevl
Copy link
Member Author

aykevl commented May 26, 2020

Module support itself is not directly tested. But because all compilations (both the tests in main_test.go and the smoke tests) use the new system that also supports Go modules, that might be enough. For example: the list of Go files to compile is provided by the Go command (go list inside x/tools/packages) which will transparently use Go modules if it finds a go.mod file.

@deadprogram
Copy link
Member

Well, it does appear to be working as expected in all cases. I am going to merge, and if any new issues appear as a result, we can address them. Great work on this @aykevl

@deadprogram deadprogram merged commit 4ca2d3f into dev May 27, 2020
@deadprogram deadprogram deleted the mod branch May 27, 2020 11:09
@aykevl aykevl mentioned this pull request May 28, 2020
Open
@bradleypeabody bradleypeabody mentioned this pull request Jun 27, 2020
Closed
@niaow niaow added this to the v0.14 milestone Jun 27, 2020
@aykevl aykevl mentioned this pull request Jul 20, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.14
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@aykevl @deadprogram @niaow