Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/internal/testdir: Test/rangegen.go failures #67556

Open
gopherbot opened this issue May 21, 2024 · 16 comments
Open

cmd/internal/testdir: Test/rangegen.go failures #67556

gopherbot opened this issue May 21, 2024 · 16 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@gopherbot
Copy link
Contributor 

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"

Issue created automatically to collect these failures.

Example (log):

=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
        
--- FAIL: Test/rangegen.go (54.26s)

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 21, 2024
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-04-09 04:07 gotip-linux-ppc64_power8 go@9f3f4c64 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
        
--- FAIL: Test/rangegen.go (54.26s)

— watchflakes

@pmur
Copy link
Contributor

pmur commented May 23, 2024

The most recent ppc64le power9 failures indicate OOM. The ppc64le containers are limited to 10GB today (including tmpfs for LUCI stuff). The rangegen test is probably the most memory intensive test.

Memory cgroup out of memory: Killed process 2631390 (compile) total-vm:3672064kB, anon-rss:2462528kB, file-rss:128kB, shmem-rss:16704kB, UID:2000 pgtables:350kB oom_score_adj:0

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-06-04 15:28 gotip-linux-ppc64le_power9 go@535f81e7 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (75.88s)
2024-06-04 16:19 gotip-linux-ppc64le_power9 go@b5a86178 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (69.98s)
2024-06-05 19:56 gotip-linux-ppc64le_power9 go@e53d10dc cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (67.58s)
2024-06-21 12:36 gotip-linux-ppc64le_power9 go@20b79fd5 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (58.92s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-17 15:08 gotip-linux-ppc64le_power9 go@fc51e502 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (65.10s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-22 15:45 gotip-linux-ppc64le_power9 go@f0de94ff cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (65.94s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-23 17:16 gotip-linux-ppc64le_power9 go@fe87b586 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (66.05s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-25 14:38 gotip-linux-ppc64le_power9 go@e76353d5 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (63.98s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-24 18:55 go1.23-linux-ppc64le_power9 release-branch.go1.23@559c7759 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (73.00s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-30 17:16 gotip-linux-ppc64le_power9 go@c7227bc7 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (57.99s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-31 14:26 gotip-linux-ppc64le_power9 go@eea2e929 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (73.02s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-07-31 21:23 gotip-linux-ppc64le_power9 go@31e692be cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (73.01s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-08-01 02:53 gotip-linux-ppc64le_power9 go@bd85a3b1 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (76.35s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-08-01 19:13 gotip-linux-ppc64le_power9 go@2caf638e cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (55.61s)

— watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-08-02 00:29 gotip-linux-ppc64le_power9 go@11dbbaff cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (53.98s)
2024-08-02 14:14 gotip-linux-ppc64le_power9 go@9abd1144 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (63.39s)

— watchflakes

@pmur
Copy link
Contributor

pmur commented Aug 5, 2024

I've readjusted the power9 builders to 4 instances, and reallocated the memory (about 3G more to the remaining buildlets). I suspect the underlying hardware is overcommitted, and that is the cause of the spurious slowdowns we see only on P9.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
2024-08-05 13:12 gotip-linux-ppc64le_power9 go@d465aee0 cmd/internal/testdir.Test/rangegen.go (log) 
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:145: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed
        
--- FAIL: Test/rangegen.go (55.73s)

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gopherbot @pmur