website: tip.golang.org is down

[andybons]

Fails with this message on all pages:

builder.Init: build cache is disabled by GOCACHE=off, but required as of Go 1.12

exit status 1

/cc @bradfitz @bcmills

[bradfitz]

Bryan, can you fix this soon?

/cc @dmitshur

[bradfitz]

I see no mention of GOCACHE in x/tools or x/net. We're not setting this explicitly anywhere that I can see.

[bradfitz]

GOCACHE doesn't appear to be set in that container:

bradfitz@gdev:~$ kubectl exec -ti tipgodoc-deployment-5c777d44fd-6g7b7 /bin/bash
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# ps afx
    PID TTY      STAT   TIME COMMAND
  54205 ?        Ss     0:00 /bin/bash
  54212 ?        R+     0:00  \_ ps afx
      1 ?        Ssl    0:20 /go/bin/tip --autocert=tip.golang.org,beta.golang.org --autocert-bucket=golang-tip-autocert
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# cat /proc/1/environ  | grep GOCACHE
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# 

[kevinburke]

Likely related to not being able to detect the home directory, see the issue I linked

On Fri, Dec 14, 2018 at 14:07 Brad Fitzpatrick ***@***.***> wrote: I see no mention of GOCACHE in x/tools or x/net. We're not setting this explicitly anywhere that I can see. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#29251 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOSI6chk7fLZ6YQ2-TnzKnj20ZCCYHCks5u47CogaJpZM4ZS3IB> .

-- Kevin Burke phone: 925.271.7005 | kev.inburke.com

[bradfitz]

@kevinburke, ah, great, thanks!

[gopherbot]

Change https://golang.org/cl/154181 mentions this issue: tip: fix, update tip.golang.org

[bradfitz]

I got it kinda back alive, but now the godoc processes are failing after a few minutes with:

process in /build/tip/b exited: signal: killed

I added some more debugging,

                go func() {
                        // TODO(adg,bradfitz): be smarter about dead processes
                        if err := cmd.Wait(); err != nil {
                                log.Printf("process in %v exited: %v (%T)", dir, err, err)
                                if ee, ok := err.(*exec.ExitError); ok {
                                        if ws, ok := ee.ProcessState.Sys().(syscall.WaitStatus); ok {
                                                log.Printf("  ... ws = %v, stopped=%v, exited=%v, exit=%v", ws, ws.Stopped(), ws.Exited(), ws.ExitStatus())
                                                if ws.Signaled() {
                                                        log.Printf("  ... stop signal: %v", ws.StopSignal())
                                                }
                                        }
                                }
                        }
                }()

And now see:

2018/12/14 16:09:55 process in /build/tip/b exited: signal: killed (*exec.ExitError)
2018/12/14 16:09:55   ... ws = 9, stopped=false, exited=false, exit=-1
2018/12/14 16:09:55   ... stop signal: signal -1

What is stop signal -1 !? @ianlancetaylor would be my usual person to ask about that. @bcmills, do you know?

[bradfitz]

Nevermind. I was misusing StopSignal. I thought it was something else. WaitStatus of 9 just means it got SIGKILLed.

[bradfitz]

As expected, OOM:

[55181.043263] Memory cgroup out of memory: Kill process 158615 (godoc) score 1861 or sacrifice child
[55181.052447] Killed process 158615 (godoc) total-vm:5326320kB, anon-rss:4174076kB, file-rss:1072kB, shmem-rss:0kB
[56028.461663] godoc invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=864
[56028.472283] godoc cpuset=0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c mems_allowed=0
[56028.481937] CPU: 0 PID: 167205 Comm: godoc Not tainted 4.14.65+ #1
[56028.488424] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[56028.497766] Call Trace:
[56028.500342]  dump_stack+0x63/0x82
[56028.503776]  dump_header+0x9f/0x22a
[56028.507386]  ? mem_cgroup_scan_tasks+0xaf/0xd0
[56028.511964]  oom_kill_process+0x232/0x430
[56028.516098]  out_of_memory+0x100/0x4c0
[56028.519968]  mem_cgroup_out_of_memory+0x3f/0x60
[56028.524629]  mem_cgroup_oom_synchronize+0x2dd/0x300
[56028.529625]  ? unlock_page_memcg+0x50/0x50
[56028.533846]  pagefault_out_of_memory+0x25/0x56
[56028.538421]  __do_page_fault+0x478/0x4e0
[56028.545848]  ? page_fault+0x2f/0x50
[56028.549549]  page_fault+0x45/0x50
[56028.552989] RIP: fee4c0:          (null)
[56028.557042] RSP: 70000000:000000c001d623a8 EFLAGS: 0a38b3d2
[56028.557132] Task in /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c killed as a result of limit of /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a
[56028.584011] memory: usage 4194304kB, limit 4194304kB, failcnt 38477
[56028.590440] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[56028.597736] kmem: usage 14084kB, limit 9007199254740988kB, failcnt 0
[56028.604292] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[56028.629615] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/f04d13c3e17abb2424036943fd9ad241ca2708e91cdb8a621d48212e23b3a72d: cache:0KB rss:40KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:40KB inactive_file:0KB active_file:0KB unevictable:0KB
[56028.659538] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c: cache:104KB rss:4179924KB rss_huge:1441792KB shmem:0KB mapped_file:48KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:4179924KB inactive_file:100KB active_file:4KB unevictable:0KB
[56028.691290] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[56028.700052] [151592]     0 151592      256        1       4       2        0          -998 pause
[56028.709049] [151679]     0 151679   138583     3513      38       4        0           864 tip
[56028.717989] [167201]     0 167201  1384927  1043295    2249       8        0           864 godoc
[56028.727279] Memory cgroup out of memory: Kill process 167201 (godoc) score 1860 or sacrifice child
[56028.736654] Killed process 167201 (godoc) total-vm:5539708kB, anon-rss:4172504kB, file-rss:676kB, shmem-rss:0kB
[56028.926586] oom_reaper: reaped process 167201 (godoc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Something between Go 1.9 and Go 1.11 uses more memory I guess. Yay.

Will increase it a bit for now.