Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/bisect: "failed to copy test binary to VM" errors #1250

Open
dvyukov opened this issue Jun 24, 2019 · 1 comment
Open

pkg/bisect: "failed to copy test binary to VM" errors #1250

dvyukov opened this issue Jun 24, 2019 · 1 comment

Comments

@dvyukov
Copy link
Collaborator

dvyukov commented Jun 24, 2019

See e.g. the following bisection log:
https://syzkaller.appspot.com/x/bisect.txt?x=10393a89a00000
for this bug:
https://syzkaller.appspot.com/bug?extid=4a81789411f3de69b84e
It all started well, but then at some point started suffering from massive "failed to copy test binary to VM" errors during instance testing. Debug logs for these runs are empty (no output at all). We also see similar errors during image testing (lost connection, connection timed out).
I don't understand what could cause this on our side. Could be a GCE bug, but hard to prove...
This issue is about figuring out what happens and why.

testing commit 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.11
testing commit a351e9b9fc24e982ec2f0e76379a49826036da12 with gcc (GCC) 7.3.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #9: crashed: INFO: rcu detected stall in ext4_file_write_iter
testing release v4.10
testing commit c470abd4fde40ea6a0846a2beab642a578c0b8cd with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor990482663" "root@10.128.0.67:./syz-executor990482663"]: exit status 1
Connection timed out during banner exchange
lost connection

run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor216456474" "root@10.128.15.205:./syz-executor216456474"]: exit status 1
Connection timed out during banner exchange
lost connection

run #6: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor009762225" "root@10.128.15.200:./syz-executor009762225"]: exit status 1
ssh: connect to host 10.128.15.200 port 22: Connection timed out
lost connection

run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor994225500" "root@10.128.15.206:./syz-executor994225500"]: exit status 1
Connection timed out during banner exchange
lost connection

run #8: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor725134091" "root@10.128.10.61:./syz-executor725134091"]: exit status 1
Connection timed out during banner exchange
lost connection

run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor062433006" "root@10.128.15.211:./syz-executor062433006"]: exit status 1
Connection timed out during banner exchange
lost connection

testing release v4.9
testing commit 69973b830859bc6529a7a0468ba0d80ee5117826 with gcc (GCC) 5.5.0
run #0: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor208047471" "root@10.128.0.125:./syz-executor208047471"]: exit status 1
Connection timed out during banner exchange
lost connection

run #1: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor602329592" "root@10.128.10.45:./syz-executor602329592"]: exit status 1
Connection timed out during banner exchange
lost connection

run #2: OK
run #3: OK
run #4: OK
run #5: OK
run #6: OK
run #7: OK
run #8: OK
run #9: OK
# git bisect start v4.10 v4.9
Bisecting: 7099 revisions left to test after this (roughly 13 steps)
[f4000cd99750065d5177555c0a805c97174d1b9f] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
testing commit f4000cd99750065d5177555c0a805c97174d1b9f with gcc (GCC) 5.5.0
all runs: boot failed: can't ssh into the instance
# git bisect skip f4000cd99750065d5177555c0a805c97174d1b9f
Bisecting: 7099 revisions left to test after this (roughly 13 steps)
[ab1effc09519f3bb4b84dd6d8276cedf07b17a1b] staging: ks7010: Add blank line after declarations
testing commit ab1effc09519f3bb4b84dd6d8276cedf07b17a1b with gcc (GCC) 5.5.0
run #0: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor660042412" "root@10.128.15.201:./syz-executor660042412"]: exit status 1
Connection timed out during banner exchange
lost connection

run #1: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor920693950" "root@10.128.0.254:./syz-executor920693950"]: exit status 1
Connection timed out during banner exchange
lost connection

run #2: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor765371141" "root@10.128.15.200:./syz-executor765371141"]: exit status 1
ssh: connect to host 10.128.15.200 port 22: Connection timed out
lost connection

run #3: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor910776703" "root@10.128.15.202:./syz-executor910776703"]: exit status 1
Connection timed out during banner exchange
lost connection

run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor392968201" "root@10.128.15.203:./syz-executor392968201"]: exit status 1
Connection timed out during banner exchange
lost connection

run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor277896660" "root@10.128.1.6:./syz-executor277896660"]: exit status 1
ssh: connect to host 10.128.1.6 port 22: Connection timed out
lost connection

run #6: OK
run #7: OK
run #8: OK
run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor808612131" "root@10.128.15.205:./syz-executor808612131"]: exit status 1
Connection timed out during banner exchange
lost connection

# git bisect skip ab1effc09519f3bb4b84dd6d8276cedf07b17a1b
Bisecting: 7099 revisions left to test after this (roughly 13 steps)
[ae7871be189cb41184f1e05742b4a99e2c59774d] swiotlb: Convert swiotlb_force from int to enum
testing commit ae7871be189cb41184f1e05742b4a99e2c59774d with gcc (GCC) 5.5.0
run #0: basic kernel testing failed: failed to copy test binary to VM: timedout ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/syzkaller/jobs/linux/gopath/src/github.com/google/syzkaller/bin/linux_amd64/syz-fuzzer" "root@10.128.1.48:./syz-fuzzer"]
Warning: Permanently added '10.128.1.48' (ECDSA) to the list of known hosts.

run #1: boot failed: can't ssh into the instance
run #2: boot failed: can't ssh into the instance
run #3: boot failed: can't ssh into the instance
run #4: boot failed: can't ssh into the instance
run #5: boot failed: can't ssh into the instance
run #6: boot failed: can't ssh into the instance
run #7: boot failed: can't ssh into the instance
run #8: boot failed: can't ssh into the instance
run #9: boot failed: can't ssh into the instance
# git bisect skip ae7871be189cb41184f1e05742b4a99e2c59774d
Bisecting: 7099 revisions left to test after this (roughly 13 steps)
[4a2ce27bb5f291c38d8e49ef16899828289e4d3d] i40e: properly cleanup on allocation failure in i40e_sync_vsi_filters
testing commit 4a2ce27bb5f291c38d8e49ef16899828289e4d3d with gcc (GCC) 5.5.0
run #0: crashed: BUG: soft lockup in corrupted
run #1: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor126824008" "root@10.128.0.30:./syz-executor126824008"]: exit status 1
Connection timed out during banner exchange
lost connection

run #2: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor808254727" "root@10.128.0.216:./syz-executor808254727"]: exit status 1
Connection timed out during banner exchange
lost connection

run #3: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor362597562" "root@10.128.0.217:./syz-executor362597562"]: exit status 1
ssh: connect to host 10.128.0.217 port 22: Connection timed out
lost connection

run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor380104188" "root@10.128.0.220:./syz-executor380104188"]: exit status 1
ssh: connect to host 10.128.0.220 port 22: Connection timed out
lost connection

run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor679095083" "root@10.128.0.206:./syz-executor679095083"]: exit status 1
Connection timed out during banner exchange
lost connection

run #6: OK
run #7: OK
run #8: OK
run #9: OK
# git bisect bad 4a2ce27bb5f291c38d8e49ef16899828289e4d3d
Bisecting: 181 revisions left to test after this (roughly 8 steps)
[4fe77d82ef80c77031c9c6f8554cd0dee2aa423a] skbedit: allow the user to specify bitmask for mark
testing commit 4fe77d82ef80c77031c9c6f8554cd0dee2aa423a with gcc (GCC) 5.5.0
run #0: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor381513634" "root@10.128.10.47:./syz-executor381513634"]: exit status 1
Connection timed out during banner exchange
lost connection

run #1: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor577946404" "root@10.128.10.63:./syz-executor577946404"]: exit status 1
Connection timed out during banner exchange
lost connection

run #2: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor142568600" "root@10.128.10.57:./syz-executor142568600"]: exit status 1
Connection timed out during banner exchange
lost connection

run #3: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor227848471" "root@10.128.10.58:./syz-executor227848471"]: exit status 1
Connection timed out during banner exchange
lost connection

run #4: OK
run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor201491338" "root@10.128.0.196:./syz-executor201491338"]: exit status 1
Connection timed out during banner exchange
lost connection

run #6: OK
run #7: OK
run #8: OK
run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor185613153" "root@10.128.10.49:./syz-executor185613153"]: exit status 1
ssh: connect to host 10.128.10.49 port 22: Connection timed out
lost connection
@dvyukov
Copy link
Collaborator Author

dvyukov commented Jun 24, 2019

Related to #1051

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dvyukov