Support Go1.12 #900

hajimehoshi · 2019-03-03T18:42:13Z

This PR fixes implementation and tests in order to support Go 1.12.

Fixes #887

PTAL

neelance · 2019-03-03T20:12:51Z

circle.yml

-    - run: go tool vet *.go # Go package in root directory.
-    - run: for d in */; do echo $d; done | grep -v tests/ | grep -v third_party/ | xargs go tool vet # All subdirectories except "tests", "third_party".
+    - run: go vet *.go # Go package in root directory.
+    - run: for d in */; do echo ./$d...; done | grep -v ./doc | grep -v ./test | grep -v ./node | xargs go vet # All subdirectories except "doc", "tests", "node*".


Please explain this change.

go tool vet is deprecated and go vet is the substitute. (Error)

It looked like go vet required the relative paths for local packages. I thought this line tries to go vet against part of the local directories. Is that correct?

This change makes sense to me, as @hajimehoshi described above.

A few questions/suggestions:

I suspect the go tool vet *.go line can be simplified even further to just go vet, or go vet ., to run vet on the Go package with the import path github.com/gopherjs/gopherjs. It was *.go previously because go tool vet did not accept import paths, only files and directories.

Does doc directory need to be skipped? It has just 2 .md files, so the import path pattern ./doc/... should not be matching any packages.

Done.

go vet ./doc/... fails with the exit code non-zero, and I thought I needed to avoid.

Would go vet ./... work btw?

hajimehoshi@Hajimes-MacBook-Pro ~/go/src/github.com/gopherjs/gopherjs $ go vet ./doc/... go: warning: "./doc/..." matched no packages no packages to vet hajimehoshi@Hajimes-MacBook-Pro ~/go/src/github.com/gopherjs/gopherjs $ echo $? 1

Would go vet ./... work btw?

I don't think it would, because the intention is to skip "tests" directory from being vetted.

compiler/natives/src/syscall/syscall_unix.go

compiler/version_check.go

neelance · 2019-03-03T20:22:15Z

compiler/natives/src/reflect/reflect.go

+		kv = iter.last
+	} else {
+		// Compare the index and the size of the actual key set, and check if the iterator is already exhausted.
+		if iter.i >= js.Global.Call("$keys", iter.m).Length() {


Calling $keys again does not look right at first glance. Which bug is this supposed to fix?

reflect.TestMapIterDelete1

Value tries to access mapiterkeys at the second and later Next call (code, and iter.keys can be already not synced with $keys by deleting keys in a loop.

On second thought, my fix might not work with the below code:

iter := ValueOf(m).MapRange() for iter.Next() { delete(m, iter.Key()) } if len(m) != 0 { panic("not reached") }

Let me think...

hajimehoshi

Thank you!

hajimehoshi · 2019-03-04T01:21:26Z

circle.yml

-    - run: go tool vet *.go # Go package in root directory.
-    - run: for d in */; do echo $d; done | grep -v tests/ | grep -v third_party/ | xargs go tool vet # All subdirectories except "tests", "third_party".
+    - run: go vet *.go # Go package in root directory.
+    - run: for d in */; do echo ./$d...; done | grep -v ./doc | grep -v ./test | grep -v ./node | xargs go vet # All subdirectories except "doc", "tests", "node*".


go tool vet is deprecated and go vet is the substitute. (Error)

It looked like go vet required the relative paths for local packages. I thought this line tries to go vet against part of the local directories. Is that correct?

hajimehoshi · 2019-03-04T01:27:44Z

compiler/natives/src/reflect/reflect.go

+		kv = iter.last
+	} else {
+		// Compare the index and the size of the actual key set, and check if the iterator is already exhausted.
+		if iter.i >= js.Global.Call("$keys", iter.m).Length() {


reflect.TestMapIterDelete1

Value tries to access mapiterkeys at the second and later Next call (code, and iter.keys can be already not synced with $keys by deleting keys in a loop.

On second thought, my fix might not work with the below code:

iter := ValueOf(m).MapRange() for iter.Next() { delete(m, iter.Key()) } if len(m) != 0 { panic("not reached") }

Let me think...

compiler/natives/src/syscall/syscall_unix.go

compiler/version_check.go

hajimehoshi · 2019-03-04T02:14:15Z

I found another problem: the test doesn't work well on macOS (e.g. go run -tags=gopherjsdev . test --short reflect takes forever). I'll fix this later before merging this.

hajimehoshi · 2019-03-05T14:21:10Z

In the current situation:

hajimehoshi@Hajimes-MacBook-Pro ~/go/src/github.com/gopherjs/gopherjs
$ go build . && env GOOS=linux ./gopherjs test -v github.com/gopherjs/gopherjs/tests/main
gopherjs: Source maps disabled. Install source-map-support module for nice stack traces. See https://github.com/gopherjs/gopherjs#gopherjs-run-gopherjs-test.                                                                             
=== RUN   TestNotRunMain
--- PASS: TestNotRunMain (0.00s)
PASS
fatal error: all goroutines are asleep - deadlock!
FAIL    github.com/gopherjs/gopherjs/tests/main 0.268s
hajimehoshi@Hajimes-MacBook-Pro ~/go/src/github.com/gopherjs/gopherjs

$ go build . && env GOOS=darwin ./gopherjs test -v github.com/gopherjs/gopherjs/tests/main
gopherjs: Source maps disabled. Install source-map-support module for nice stack traces. See https://github.com/gopherjs/gopherjs#gopherjs-run-gopherjs-test.                                                                             
warning: system calls not available, see https://github.com/gopherjs/gopherjs/blob/master/doc/syscalls.md
fatal error: all goroutines are asleep - deadlock!
FAIL    github.com/gopherjs/gopherjs/tests/main 0.246s

Now I'm investigating why the tests don't work on Darwin.

EDIT: This was a syscall issue. Fixed!

dmitshur · 2019-03-05T15:01:16Z

Did you install source-map-support module and syscall module? The ones mentioned at https://github.com/gopherjs/gopherjs#gopherjs-run-gopherjs-test. They used to be more optional but by now I think they're mandatory for some tests.

hajimehoshi · 2019-03-05T15:24:09Z

Did you install source-map-support module and syscall module?

No, but I think the cause was that the write implementation was not used on Darwin and I fixed syscall infrastructure for Darwin.

hajimehoshi · 2019-03-05T15:36:08Z

Now all the known issues have been solved. Please take another look. Thank you!

myitcv · 2019-03-08T18:47:37Z

Just adding some notes based on us applying roughly the same patch to myitcv#41:

I think the *_js.go files need to be renamed and build constraints used
Because the CircleCI config file specifies the standard library tests to run (as opposed to a set negation approach) this PR is not running tests in some of the new standard library packages, crucially internal/fmtsort, so the fact the tests are passing for this PR is (I think) currently a false positive

$ diff -wu 1.11 1.12
--- 1.11        2019-03-08 18:46:10.181963351 +0000
+++ 1.12        2019-03-08 18:46:02.570274117 +0000
@@ -88,6 +88,8 @@
 index/suffixarray
 internal/bytealg
 internal/cpu
+internal/fmtsort
+internal/goroot
 internal/nettrace
 internal/poll
 internal/race
@@ -99,6 +101,27 @@
 internal/testenv
 internal/testlog
 internal/trace
+internal/x/crypto/chacha20poly1305
+internal/x/crypto/cryptobyte
+internal/x/crypto/cryptobyte/asn1
+internal/x/crypto/curve25519
+internal/x/crypto/hkdf
+internal/x/crypto/internal/chacha20
+internal/x/crypto/poly1305
+internal/x/net/dns/dnsmessage
+internal/x/net/http/httpguts
+internal/x/net/http/httpproxy
+internal/x/net/http2/hpack
+internal/x/net/idna
+internal/x/net/internal/nettest
+internal/x/net/nettest
+internal/x/text/secure
+internal/x/text/secure/bidirule
+internal/x/text/transform
+internal/x/text/unicode
+internal/x/text/unicode/bidi
+internal/x/text/unicode/norm
+internal/xcoff
 io
 io/ioutil
 log
@@ -143,6 +166,7 @@
 runtime/cgo
 runtime/debug
 runtime/internal/atomic
+runtime/internal/math
 runtime/internal/sys
 runtime/pprof
 runtime/pprof/internal/profile
@@ -167,22 +191,3 @@
 unicode/utf16
 unicode/utf8
 unsafe
-vendor/golang_org/x/crypto/chacha20poly1305
-vendor/golang_org/x/crypto/cryptobyte
-vendor/golang_org/x/crypto/cryptobyte/asn1
-vendor/golang_org/x/crypto/curve25519
-vendor/golang_org/x/crypto/internal/chacha20
-vendor/golang_org/x/crypto/poly1305
-vendor/golang_org/x/net/dns/dnsmessage
-vendor/golang_org/x/net/http/httpguts
-vendor/golang_org/x/net/http/httpproxy
-vendor/golang_org/x/net/http2/hpack
-vendor/golang_org/x/net/idna
-vendor/golang_org/x/net/internal/nettest
-vendor/golang_org/x/net/nettest
-vendor/golang_org/x/text/secure
-vendor/golang_org/x/text/secure/bidirule
-vendor/golang_org/x/text/transform
-vendor/golang_org/x/text/unicode
-vendor/golang_org/x/text/unicode/bidi
-vendor/golang_org/x/text/unicode/norm

hajimehoshi · 2019-03-09T06:19:09Z

@myitcv Thank you! For the above diff, the original master's circle.yml doesn't include a lot of tests like internal/cpu. Would it be possible to decide to care such packages in another PR if needed?

I think the *_js.go files need to be renamed and build constraints used

Done.

myitcv · 2019-03-09T09:27:20Z

@hajimehoshi

For the above diff, the original master's circle.yml doesn't include a lot of tests like internal/cpu. Would it be possible to decide to care such packages in another PR if needed?

I'd suggest switching to the negation-based approach. We can even do that ahead of the Go 1.12 changes.

It's a simple, easy to understand change that means we won't have to manually keep track of new packages. Packages like internal/fmtsort which, per myitcv#41, seem to have caught an issue.

hajimehoshi · 2019-03-09T09:43:39Z

I'd suggest switching to the negation-based approach. We can even do that ahead of the Go 1.12 changes.

A blacklist instead of the current whitelist makes sense.

myitcv · 2019-03-09T14:20:25Z

.std_test_pkg_exclusions

+image/color/palette
+image/internal/imageutil
+internal/cpu
+internal/fmtsort


A better starting point for this blacklist is the list of packages that were not tested as part of the 1.11 cycle (hence why I suggested we could/should do that as part of a separate, simple PR).

That would then highlight that you're including internal/fmtsort in this file, i.e. it would be an additional package we're ignoring for Go 1.12.

I still think we should understand why this package in particular is failing, because it's fundamental to the sorting changes in the fmt package and other places:

Package fmtsort provides a general stable ordering mechanism for maps, on behalf of the fmt and text/template packages.

Sure. As I merged your PR, I'll be investigating why internal/fmtsort tests failed. I agree fmtsort is so fundamental that we need to care.

The root cause is that getting pointer values from a slice is now counterintuitive

https://gopherjs.github.io/playground/#/jvnZoFsINA

Ha, nice, my guess is that conversion an integer from a pointer (in reflect?) causes this issue

https://gopherjs.github.io/playground/#/fN4zm6Wj4W

https://github.com/gopherjs/gopherjs/blob/master/compiler/expressions.go#L982 I think this is the culprit

I made the tests pass with these fixes:

For pointers and channels, the current GopherJS cannot treat them well for sorting. Skip this.

GopherJS cannot treat nil key in a map well. Skip this.

Using a black list is a more effective mechanism for expressing those standard library packages we don't want to test because it means we don't need to manually check whether new standard library packages have been added that would otherwise need to be added to the white list. See the discussion in gopherjs#900 for more context.

So that we can then see the true diff that the Go 1.12 changes in gopherjs#900 will introduce.

…902) Using a black list is a more effective mechanism for expressing those standard library packages we don't want to test because it means we don't need to manually check whether new standard library packages have been added that would otherwise need to be added to the white list. See the discussion in #900 for more context.

So that we can then see the true diff that the Go 1.12 changes in gopherjs#900 will introduce.

So that we can then see the true diff that the Go 1.12 changes in #900 will introduce.

hajimehoshi · 2019-03-18T02:51:37Z

Friendly ping

dmitshur

Thank you very much for working on this @hajimehoshi, you did a great job implementing support for Go 1.12!

I've reviewed this and did not find any issues or significant opportunities for improvement. I left a few comments about some minor things.

I've also tested it with various GopherJS programs on macOS and they all ran without problems. The generated file sizes have increased by about 5-7% (depending on which standard library packages are imported), which is unfortunate but within normal range of new Go releases.

compiler/natives/src/reflect/reflect.go

compiler/natives/fs_vfsdata.go

compiler/version_check.go

hajimehoshi · 2019-03-27T14:07:38Z

@dmitshur Thank you for reviewing! I appreciate your careful work.

Should we squash the commits, or not? My preference is to squash them, but I'd like to obey the policy of GopherJS.

dmitshur · 2019-03-28T03:27:29Z

When you ask if we should squash the commits, can you clarify what exactly you mean? I can't answer your question directly because I'm not sure what you meant, so I'll describe this in full.

When merging this PR, we should use the "create a merge commit" merge strategy:

When creating the merge commit, we can follow the same pattern as in previous merge commits, e.g., 0210a2f. You'll have to edit the subject and body fields accordingly. E.g.:

We should not use the "squash and merge" merge strategy. Doing so would squash all commits in this PR into a single commit. Such a commit would be massive, it would include all generated code, and it's not viable to meaningfully describe the changes that went into adding Go 1.12 support.

However, before doing the merge, it is completely acceptable and in fact preferable to re-write the PR history to clean it up and squash redundant commits, so that it consists of clean logical well-described changes. At the very least:

all the generated commits can be squashed into a single commit at the end that re-generates all data; see third-last commit in PR Go 1.11 support. #853 that added Go 1.11 support as an example of this

If any of the other commits are redundant or WIP (e.g., where you tried one thing, but ended up overwriting it a future commit) and can be squashed into a single logical commit with a descriptive commit message, that's great too.

Of course, re-writing the history involves a force push to this PR branch, so be careful, and do a diff of the final result against the starting point (i.e., the latest commit is a02f42a right now) to ensure the end result is unchanged.

Finally, feel free to look over the previous PRs to get an idea of how it was done historically. I've linked them all in #900 (comment).

Cleaning up the PR history and writing more descriptive commit messages is great, but optional. Feel free to do as much or as little before merging this PR. Thanks again @hajimehoshi!

hajimehoshi · 2019-03-28T16:15:25Z

We should not use the "squash and merge" merge strategy.

What I wanted to ask is whether we should use this strategy. Sorry for my unclear question. Thanks!

On Darwin, syscall was instroduced at golang/go@a3b0144. This change renames syscall to avoid collision.

`go tool vet` is abandoned. Use `go vet` instead.

The implementation is mock.

Still work in progress.

hajimehoshi · 2019-03-28T16:45:37Z

I removes some redundant commits, but I couldn't integrate go-generating into one commit. That's not perfect, but I believe that's acceptable.

Done on darwin/amd64 environment. Regenerated with: go generate github.com/gopherjs/gopherjs.github.io/playground Follows gopherjs/gopherjs#900.

hajimehoshi requested review from dmitshur, myitcv and neelance March 3, 2019 18:42

neelance reviewed Mar 3, 2019

View reviewed changes

hajimehoshi commented Mar 4, 2019

View reviewed changes

hajimehoshi requested a review from neelance March 5, 2019 17:06

myitcv mentioned this pull request Mar 7, 2019

compiler: upgrade to Go 1.12 myitcv/gopherjs#41

Merged

AnikHasibul approved these changes Mar 8, 2019

View reviewed changes

myitcv reviewed Mar 9, 2019

View reviewed changes

myitcv mentioned this pull request Mar 9, 2019

ci: use black list instead of white list for standard library tests #902

Merged

myitcv added a commit to myitcv/gopherjs that referenced this pull request Mar 9, 2019

ci: upgrade to Go 1.11.5

ecac41d

So that we can then see the true diff that the Go 1.12 changes in gopherjs#900 will introduce.

myitcv mentioned this pull request Mar 9, 2019

ci: upgrade to Go 1.11.5 #903

Merged

myitcv added a commit to myitcv/gopherjs that referenced this pull request Mar 9, 2019

ci: upgrade to Go 1.11.5

cd658a9

So that we can then see the true diff that the Go 1.12 changes in gopherjs#900 will introduce.

myitcv added a commit that referenced this pull request Mar 9, 2019

ci: upgrade to Go 1.11.5 (#903)

847fc94

So that we can then see the true diff that the Go 1.12 changes in #900 will introduce.

dmitshur approved these changes Mar 27, 2019

View reviewed changes

compiler/natives/src/reflect/reflect.go Show resolved Hide resolved

compiler/natives/fs_vfsdata.go Outdated Show resolved Hide resolved

compiler/natives/fs_vfsdata.go Outdated Show resolved Hide resolved

compiler/version_check.go Outdated Show resolved Hide resolved

hajimehoshi added 24 commits March 29, 2019 01:33

compiler/natives/src/syscall: Add Syscall18

80a8d4e

compiler/natives/src/syscall: Rename syscall

9576d72

On Darwin, syscall was instroduced at golang/go@a3b0144. This change renames syscall to avoid collision.

Update circle.yml

756c79f

`go tool vet` is abandoned. Use `go vet` instead.

compiler/natives/src/internal/syscall/unix: Add fstatatTrap

96c95ab

go generate

6b3d4d1

compiler/natives/src/syscall: Define syscall functions for Darwin

f65c39d

The implementation is mock.

compiler/natives/src/crypto/rand: Add func batched

9793ddf

compiler/natives/src: Fix some test failures

c0c4111

Still work in progress.

compiler/natives/src: Fix test failures

3a29d97

compiler/natives/src/reflect: Fix failure tests

250c084

tests: Add new failure tests in Go 1.12

57d7708

Address on reviews

cf99cfe

Address on reviews

158447e

Fix circle.yml

db381e3

Fix testing issues on macOS

35239cf

reflect: Reimplement mapiter functions

2ea661b

Remove filename suffixes _js

4711f47

Add .std_test_pkg_exlusions

a2f6a71

Test internal/fmtsort

9da6e14

Add more testable standard libs

3844e18

Fix .std_test_pkg_exclusions

e34c676

go generate

18df411

compiler/natives/src/reflect: Add comments

8471480

compiler: Update version to 1.12-1

09ef6f4

hajimehoshi force-pushed the go1.12 branch from a02f42a to 09ef6f4 Compare March 28, 2019 16:44

hajimehoshi merged commit bb26745 into master Mar 28, 2019

hajimehoshi deleted the go1.12 branch March 28, 2019 17:07

dmitshur mentioned this pull request Mar 29, 2019

Support for Go 1.12 #887

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Go1.12 #900

Support Go1.12 #900

hajimehoshi commented Mar 3, 2019

neelance Mar 3, 2019

hajimehoshi Mar 4, 2019

dmitshur Mar 4, 2019

hajimehoshi Mar 5, 2019

hajimehoshi Mar 5, 2019 •

edited

dmitshur Mar 9, 2019

neelance Mar 3, 2019

hajimehoshi Mar 4, 2019

hajimehoshi Mar 5, 2019

hajimehoshi left a comment

hajimehoshi Mar 4, 2019

hajimehoshi Mar 4, 2019

hajimehoshi commented Mar 4, 2019

hajimehoshi commented Mar 5, 2019 •

edited

dmitshur commented Mar 5, 2019

hajimehoshi commented Mar 5, 2019

hajimehoshi commented Mar 5, 2019

myitcv commented Mar 8, 2019

hajimehoshi commented Mar 9, 2019

myitcv commented Mar 9, 2019

hajimehoshi commented Mar 9, 2019

myitcv Mar 9, 2019

hajimehoshi Mar 9, 2019

hajimehoshi Mar 9, 2019

hajimehoshi Mar 9, 2019

hajimehoshi Mar 9, 2019

hajimehoshi Mar 9, 2019 •

edited

hajimehoshi commented Mar 18, 2019

dmitshur left a comment

hajimehoshi commented Mar 27, 2019

dmitshur commented Mar 28, 2019 •

edited

hajimehoshi commented Mar 28, 2019

hajimehoshi commented Mar 28, 2019

Support Go1.12 #900

Support Go1.12 #900

Conversation

hajimehoshi commented Mar 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hajimehoshi Mar 5, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hajimehoshi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hajimehoshi commented Mar 4, 2019

hajimehoshi commented Mar 5, 2019 • edited

dmitshur commented Mar 5, 2019

hajimehoshi commented Mar 5, 2019

hajimehoshi commented Mar 5, 2019

myitcv commented Mar 8, 2019

hajimehoshi commented Mar 9, 2019

myitcv commented Mar 9, 2019

hajimehoshi commented Mar 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hajimehoshi Mar 9, 2019 • edited

Choose a reason for hiding this comment

hajimehoshi commented Mar 18, 2019

dmitshur left a comment

Choose a reason for hiding this comment

hajimehoshi commented Mar 27, 2019

dmitshur commented Mar 28, 2019 • edited

hajimehoshi commented Mar 28, 2019

hajimehoshi commented Mar 28, 2019

hajimehoshi Mar 5, 2019 •

edited

hajimehoshi commented Mar 5, 2019 •

edited

hajimehoshi Mar 9, 2019 •

edited

dmitshur commented Mar 28, 2019 •

edited