Build failing - process terminated by signal 15 (timeout?)

[liamawhite]

When building this workspace I get the following error:

bazel build //...
ERROR: /Users/liam/dev/go/src/istio.io/manager/platform/kube/BUILD:48:1: no such package '@io_k8s_ingress//core/pkg/ingress/annotations/class': failed to generate BUILD files for k8s.io/ingress: Process terminated by signal 15 and referenced by '//platform/kube:go_default_test'.
ERROR: Analysis of target '//platform/kube:go_default_test' failed; build aborted.
INFO: Elapsed time: 614.350s

I am seeing this problem on an OSX laptop and a separate Ubuntu 16.04 laptop on multiple networks (work, home and mobile hotspot). I believe signal 15 is a timeout which would explain why no other developers are seeing this issue.

Even if I have the dependencies that bazel is complaining about (k8s.io/ingress) pulled down locally in my gopath it still fails. Which is what I think is equivalent to what this work around is suggesting.

To reproduce just run bazel build //... in the repo root. As previously stated this will probably work for most people.

Environment info

• Operating System: macOS Sierra 10.12.5 and Ubuntu 16.04

• Bazel version: release 0.4.5-homebrew

cc @kyessenov

[laszlocsomor]

INFO: Elapsed time: 614.350s

That too could be the sign of a timeout.

I also see a timeout, with bazel-0.5.0, having checked out https://github.com/istio/manager/tree/21f4cb412eab119f34c81ceec37bce3b71dc26bc:

  $ bazel fetch //...
ERROR: /usr/local/google/laszlocsomor/gitroot/istio_mgr/platform/kube/BUILD:3:1: no such package '@io_k8s_ingress//core/pkg/ingress/status': failed to generate BUILD files for k8s.io/ingress: Process terminated by signal 15 and referenced by '//platform/kube:go_default_library'.
ERROR: /usr/local/google/laszlocsomor/gitroot/istio_mgr/platform/kube/BUILD:3:1: no such package '@io_k8s_ingress//core/pkg/ingress/store': failed to generate BUILD files for k8s.io/ingress: Process terminated by signal 15 and referenced by '//platform/kube:go_default_library'.
ERROR: /usr/local/google/laszlocsomor/gitroot/istio_mgr/platform/kube/BUILD:3:1: no such package '@io_k8s_ingress//core/pkg/ingress/status': failed to generate BUILD files for k8s.io/ingress: Process terminated by signal 15 and referenced by '//platform/kube:go_default_library'.
ERROR: Evaluation of query "deps(//...)" failed: errors were encountered while computing transitive closure.

Looking at $(bazel info output_base)/java.log I see this line:

170531 14:21:52.703:W 1601 [com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run] Aborting evaluation due to com.google.devtools.build.lib.rules.repository.RepositoryFunction$RepositoryFunctionException: com.google.devtools.build.lib.syntax.EvalExceptionWithStackTrace: failed to generate BUILD files for k8s.io/ingress: Process terminated by signal 15 while evaluating REPOSITORY_DIRECTORY:@io_k8s_ingress

meaning it's the @io_k8s_ingress new_go_repository rule failing.

@ianthehat, @pmbethe09 (I hope you're the right people): Any ideas?

[pmbethe09]

The WORKSPACE lists:
new_go_repository(
name = "io_k8s_ingress",
commit = "7f3763590a681011eedc4b14a80a97240dea644c",
importpath = "k8s.io/ingress",
)

which exists and has those paths, so I am not sure what is happening.

[liamawhite]

@laszlocsomor @pmbethe09 anything I can do to help debug?

[pmbethe09]

I checked out https://github.com/istio/pilot at HEAD which appears to be: 11dc95967a0e11c723922e51e07cb1b9935973e0
Using bazel @ HEAD I get the same issue at first, but when I run again, it seems to setting on this problem:
ERROR: missing input file '//platform/kube:config'.
ERROR: /usr/local/google/home/pbethe/golang/src/github.com/istio/pilot/platform/kube/BUILD:48:1: //platform/kube:go_default_test: missing input file '//platform/kube:config'.
ERROR: /usr/local/google/home/pbethe/golang/src/github.com/istio/pilot/platform/kube/BUILD:42:1: //platform/kube:kubeconfig: missing input file '//platform/kube:config'.
ERROR: /usr/local/google/home/pbethe/golang/src/github.com/istio/pilot/cmd/istioctl/BUILD:38:1: //cmd/istioctl:go_default_test: missing input file '//platform/kube:config'.

If I do a fresh checkout, 'touch platform/kube/config' and build, it takes a long time, but eventually succeeds.

[liamawhite]

@pmbethe09 so thats expected as part of the build process. Some of the integration tests require a kubernetes cluster to run against, which is what that file enables.

Is there any way to extend my timeout? I could try changing it to an hour to try to get round the issue.

[pmbethe09]

I tried running gazelle in k8s.io/ingress and had a crash -- and then I realized my gazelle was over 2 months old, so I updated to gazelle from 0.4.4 and it worked,

I notice that you are using rules_go 0.4.3, anyway you can update to 0.4.4 which has some performance improvements to gazelle?

[liamawhite]

@pmbethe09 just updated as follows:

git_repository(
    name = "io_bazel_rules_go",
    commit = "4c9a52aba0b59511c5646af88d2f93a9c0193647",  # May 5 2017 (0.4.4)
    remote = "https://github.com/bazelbuild/rules_go.git",
)

still getting the same error though

Scratch that, I didn't bazel clean I'm getting a different error. I will investigate further to see if this resolves the issue.

Edit: still getting the same error.

[pmbethe09]

Strange -- I can't repeat the issue but I am running with bazel @ HEAD.

[liamawhite]

@pmbethe09 are there any logs I can retrieve to help diagnose?

[pmbethe09]

@jayconrod any ideas whats going on here?

[jayconrod]

I think the problem is Gazelle is going off into the weeds because of the performance issue mentioned in bazelbuild/rules_go#353. When run in external mode (the default), Gazelle would do network fetches for every unqualified import for packages outside of go_prefix. Gazelle now caches these fetches and is significantly faster.

When I tried to build this workspace in Darwin, it got stuck, but I could see Gazelle in Activity Monitor.

After updating rules_go to the tip of master, I got the error below for a missing dependency after about a minute. This was with Bazel 0.4.5-homebrew.

ERROR: /private/var/tmp/_bazel_jayconrod/c814da7ca84eec0185ceb04620ef6ea3/external/com_github_spf13_cobra/BUILD.bazel:5:1: no such package '@com_github_inconshreveable_mousetrap//': error loading package 'external': The repository named 'com_github_inconshreveable_mousetrap' could not be resolved and referenced by '@com_github_spf13_cobra//:go_default_library'.
ERROR: Analysis of target '//cmd/version:go_default_library' failed; build aborted.
INFO: Elapsed time: 66.581s

We haven't done a release since the fix for this issue landed, but we plan to do one as soon as some other Gazelle improvements are submitted.

[liamawhite]

Good new and bad news.

Good news: upgrading to the latest commit of bazel_rules_go seems to have fixed this issue.
Bad news: upgrading to the latest commit of bazel_rules_go seems to have caused an issue. #3132

Thanks for all the help on this issue!