Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qemu dump new cases development #2457

Merged
merged 1 commit into from Nov 24, 2020

Conversation

leidwang
Copy link
Contributor

dump_guest_core.py:qemu dump new cases development
1.Use gdb check core dump file
2.Use crash check vmcore file

ID:1888531, 1888532

Signed-off-by: Leidong Wang leidwang@redhat.com

@leidwang leidwang force-pushed the dump_guest_core branch 3 times, most recently from b98c05a to 27e5f75 Compare October 15, 2020 07:32
@leidwang
Copy link
Contributor Author

(1/2) Host_RHEL.m8.u3.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.x86_64.io-github-autotest-qemu.dump_guest_core.on.q35: PASS (234.56 s)
(2/2) Host_RHEL.m8.u3.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.x86_64.io-github-autotest-qemu.dump_guest_core.off.q35: PASS (74.86 s)

@leidwang
Copy link
Contributor Author

@MiriamDeng Could you please help review this patch?Thanks!

@MiriamDeng
Copy link
Contributor

I can only see the job's name for x86 but it is supported on ppc too.

% gdb_command_file, shell=True)
process.run("echo \"set height 0\" >> %s"
% gdb_command_file, shell=True)
process.run("echo \"dump-guest-memory /tmp/vmcore X86_64\" >> %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this step only support x86 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not only for x86,since it is not specified in test case.
Could you help confirm this on ppc?Thanks!

@leidwang
Copy link
Contributor Author

And the title of the patch also need to updated, for example,
spapr_vty_multi_backends: New case to test guest with multi backends of pci-serial

Those info seems should described in commit.

@leidwang leidwang force-pushed the dump_guest_core branch 4 times, most recently from 6de42df to c87cd0b Compare October 23, 2020 02:39
@leidwang
Copy link
Contributor Author

@MiriamDeng Could you help review this patch again?Thanks!

@leidwang
Copy link
Contributor Author

@vivianQizhu Could you help review this patch? Thanks!

@MiriamDeng
Copy link
Contributor

The first job got failures on ppc, could you check ?
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/var/core.81687: No such file or directory.
Traceback (most recent call last):
File "/usr/share/qemu-kvm/dump-guest-memory.py", line 20, in
UINTPTR_T = gdb.lookup_type("uintptr_t")
gdb.error: No type named uintptr_t.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/share/qemu-kvm/dump-guest-memory.py", line 22, in
raise gdb.GdbError("Symbols must be loaded prior to sourcing dump-guest-memory.\n"
gdb.GdbError: Symbols must be loaded prior to sourcing dump-guest-memory.
Symbols may be loaded by 'attach'ing a QEMU process id or by 'load'ing a QEMU binary.
/home/gdb_command:3: Error in sourced command file:
Undefined command: "dump-guest-memory". Try "help".

(3/4) Host_RHEL.m8.u3.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.ppc64le.io-github-autotest-qemu.dump_guest_core.on: FAIL: Vmcore corrupt (261.97 s)

@leidwang
Copy link
Contributor Author

leidwang commented Oct 27, 2020

The first job got failures on ppc, could you check ?
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/var/core.81687: No such file or directory.
Traceback (most recent call last):
File "/usr/share/qemu-kvm/dump-guest-memory.py", line 20, in
UINTPTR_T = gdb.lookup_type("uintptr_t")
gdb.error: No type named uintptr_t.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/share/qemu-kvm/dump-guest-memory.py", line 22, in
raise gdb.GdbError("Symbols must be loaded prior to sourcing dump-guest-memory.\n"
gdb.GdbError: Symbols must be loaded prior to sourcing dump-guest-memory.
Symbols may be loaded by 'attach'ing a QEMU process id or by 'load'ing a QEMU binary.
/home/gdb_command:3: Error in sourced command file:
Undefined command: "dump-guest-memory". Try "help".

(3/4) Host_RHEL.m8.u3.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.ppc64le.io-github-autotest-qemu.dump_guest_core.on: FAIL: Vmcore corrupt (261.97 s)

This seems to only be encountered on ppc, could you please help to confirm?

% gdb_command_file, shell=True)
process.run("echo \"set height 0\" >> %s"
% gdb_command_file, shell=True)
process.run("echo \"dump-guest-memory /tmp/vmcore X86_64\" >> %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had better provide parameter for "x86_64" since the value should be ''ppc64le" on ppc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, if confirmed this value will change based on architecture changes, then I will change it. Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use "ppc64-le" for this command on ppc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok,thanks!

@MiriamDeng
Copy link
Contributor

And I also realized we had an existed bz 1785126, so I wonder if the patch fails or not on both x86 and ppc finally, thanks a lot.

@leidwang
Copy link
Contributor Author

leidwang commented Oct 28, 2020

And I also realized we had an existed bz 1785126, so I wonder if the patch fails or not on both x86 and ppc finally, thanks a lot.

Yes, I know this bz. We can discuss privately.

@MiriamDeng
Copy link
Contributor

Hi Leiwang,
Could you check it again ?
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-5.1.0-14.module+el8.3.0+8438+644aff69.ppc64le.debug...done.
done.

warning: Ignoring non-absolute filename: <linux-vdso64.so.1>
Missing separate debuginfo for linux-vdso64.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/79/0ecd20cfe9f0902efba483bf5059a53b3b008f
Missing separate debuginfo for /lib64/libstdc++.so.6
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/42/4a39f2ca21cc948063d8a54cdcf9dee7efd7e6
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/ed/f099bde78abe84c9a381d3ddbc13d40cf8680e
Missing separate debuginfo for /lib64/libstdc++.so.6
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/42/4a39f2ca21cc948063d8a54cdcf9dee7efd7e6.debug
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/ed/f099bde78abe84c9a381d3ddbc13d40cf8680e.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/power9/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -sandbox on -machine pseries -nod'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fff8340f1f0 in ppoll () from /lib64/power9/libc.so.6
[Current thread is 1 (Thread 0x7fff82c28760 (LWP 232609))]
/home/gdb_command:3: Error in sourced command file:
No valid arch type specified.
Currently supported types:
aarch64-be, aarch64-le, X86_64, 386, s390, ppc64-be, ppc64-le

Thanks

@leidwang
Copy link
Contributor Author

Hi Leiwang,
Could you check it again ?
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-5.1.0-14.module+el8.3.0+8438+644aff69.ppc64le.debug...done.
done.

warning: Ignoring non-absolute filename: <linux-vdso64.so.1>
Missing separate debuginfo for linux-vdso64.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/79/0ecd20cfe9f0902efba483bf5059a53b3b008f
Missing separate debuginfo for /lib64/libstdc++.so.6
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/42/4a39f2ca21cc948063d8a54cdcf9dee7efd7e6
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/ed/f099bde78abe84c9a381d3ddbc13d40cf8680e
Missing separate debuginfo for /lib64/libstdc++.so.6
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/42/4a39f2ca21cc948063d8a54cdcf9dee7efd7e6.debug
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/ed/f099bde78abe84c9a381d3ddbc13d40cf8680e.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/power9/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -sandbox on -machine pseries -nod'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fff8340f1f0 in ppoll () from /lib64/power9/libc.so.6
[Current thread is 1 (Thread 0x7fff82c28760 (LWP 232609))]
/home/gdb_command:3: Error in sourced command file:
No valid arch type specified.
Currently supported types:
aarch64-be, aarch64-le, X86_64, 386, s390, ppc64-be, ppc64-le

Thanks

This error only appears in ppc, maybe we need to discuss this error later.

@MiriamDeng
Copy link
Contributor

MiriamDeng commented Nov 2, 2020

I didn't meet the issue by manual last time, It's fine for me to discuss it later, thanks.

@leidwang leidwang force-pushed the dump_guest_core branch 2 times, most recently from f860a7c to ad738dd Compare November 4, 2020 06:27
@CongLi
Copy link
Member

CongLi commented Nov 9, 2020

@MiriamDeng Could you please review this patch again?

@MiriamDeng
Copy link
Contributor

MiriamDeng commented Nov 10, 2020

Could you help to check the patch again ? Thanks

  1. The core file generated under /var/* but not /tmp/vmcore on ppc
    2.Again met the old problem we discussed before
    and need to install extra packages
    qemu-kvm-core-debuginfo-5.1.0-14.module+el8.3.0+8438+644aff69.ppc64le.rpm,
    qemu-kvm-debuginfo-5.1.0-14.module+el8.3.0+8438+644aff69.ppc64le,
    qemu-kvm-debugsource-5.1.0-14.module+el8.3.0+8438+644aff69.ppc64le

@MiriamDeng
Copy link
Contributor

Finally it passed on ppc
(2/3) Host_RHEL.m8.u4.product_rhel.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.4.0.ppc64le.io-github-autotest-qemu.dump_guest_core.on: PASS (131.20 s)
(3/3) Host_RHEL.m8.u4.product_rhel.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.4.0.ppc64le.io-github-autotest-qemu.dump_guest_core.off: PASS (104.84 s)

@MiriamDeng
Copy link
Contributor

LGTM

virt_test_type = qemu
start_vm = no
pre_command = "ulimit -c unlimited"
pre_command1 = "echo "/var/core.%p" > /proc/sys/kernel/core_pattern"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pre_command1 = "echo "/var/core.%p" > /proc/sys/kernel/core_pattern"
pre_command + = " && echo "/var/core.%p" > /proc/sys/kernel/core_pattern"

gdb_command_file = "/home/gdb_command"
crash_script = "/home/crash.cmd"
vmcore_file = "/tmp/vmcore"
extra_params += " -machine dump-guest-core=%s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might move this line down to the bottom and extra_params += " -machine dump-guest-core=${dump_guest_core}"

pre_command = params.get("pre_command")
pre_command1 = params.get("pre_command1")
crash_script = params.get("crash_script")
vmcore_file = params.get("vmcore_file")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

params['xxx'] instead of params.get() when the params can not be None.

vmcore_file = params.get("vmcore_file")

process.run(pre_command, shell=True)
process.run(pre_command1, shell=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env_process will help you run the command when you define pre_command.

process.run(pre_command, shell=True)
process.run(pre_command1, shell=True)

params["extra_params"] %= dump_guest_core
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you remove this one, and pre command does not need you run it by yourself, then it should be fine to remove start_vm=no in cfg and directly start vm with those prepared configurations.

process.run("echo \"set height 0\" >> %s"
% gdb_command_file, shell=True)
if params['vm_arch_name'] == 'x86_64':
process.run("echo \"dump-guest-memory /tmp/vmcore X86_64\" >> %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vmcore_file instead of /tmp/vmcore

process.run("echo \"dump-guest-memory /tmp/vmcore X86_64\" >> %s"
% gdb_command_file, shell=True)
if params['vm_arch_name'] == 'ppc64le':
process.run("echo \"dump-guest-memory /tmp/vmcore ppc64-le\" >> %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

"""
Use gdb to check core dump file
"""
process.run("echo \"source /usr/share/qemu-kvm/dump-guest-memory.py\" > %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better define a variable for this path.

process.run("echo \"dump-guest-memory /tmp/vmcore ppc64-le\" >> %s"
% gdb_command_file, shell=True)
process.run("echo bt >> %s" % gdb_command_file, shell=True)
process.run("echo quit >> %s" % gdb_command_file, shell=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

echo -e "bt\nquit" >>...

process.run("echo bt > %s" % crash_script, shell=True)
process.run("echo task 0 >> %s" % crash_script, shell=True)
process.run("echo task 1 >> %s" % crash_script, shell=True)
process.run("echo quit >> %s" % crash_script, shell=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, echo -e with \n.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much,updated the code that you pointed out.
Could you please review this patch again?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process.run('echo -e "bt\\ntask 0\\ntask 1\\nquit" > %s' % crash_script, shell=True)

if params['vm_arch_name'] == 'ppc64le':
process.run("echo \"dump-guest-memory %s ppc64-le\" >> %s"
% (vmcore_file, gdb_command_file), shell=True)
process.run("echo -e 'bt\nquit' >> %s" % gdb_command_file, shell=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

        arch = params['vm_arch_name']
        arch = 'ppc64-le' if params['vm_arch_name'] == 'ppc64le' else arch
        command = ('echo -e "source %s\\nset height 0\\ndump-guest-memory'
                   ' %s %s\\nbt\\nquit" >> %s' % (dump_guest_memory_file,
                                                  vmcore_file, arch,
                                                  gdb_command_file))
        process.run(command, shell=True)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the code.Thanks!

@leidwang leidwang force-pushed the dump_guest_core branch 2 times, most recently from f6dafd2 to aa0c698 Compare November 16, 2020 06:02
extra_params += " -machine dump-guest-core=${dump_guest_core}"
- off:
dump_guest_core = off
extra_params += " -machine dump-guest-core=${dump_guest_core}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    variants:
        - on:
            dump_guest_core = on
        - off:
            dump_guest_core = off
    extra_params += " -machine dump-guest-core=${dump_guest_core}"

env_process.preprocess_vm)
vm = env.get_vm(params["main_vm"])
session = vm.wait_for_login()
host_kernel_version = process.getoutput("uname -r").strip()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommend to move this line inside check_env() to make the function more integrated.

Copy link
Contributor Author

@leidwang leidwang Nov 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since check_vmcore_file() also references host_kernel_version, it cannot be moved to check_env()


qemu_id = vm.get_pid()
gdb_command %= qemu_id
core_file = '/var/core.' + str(qemu_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a fixed variable defined before gdb_command, who should reference it, or /var/core. should be separately defined before pre_command, where reference it too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the code.Thanks

@leidwang leidwang force-pushed the dump_guest_core branch 2 times, most recently from 27458ac to 9ff23ba Compare November 17, 2020 09:53
@leidwang
Copy link
Contributor Author

leidwang commented Nov 19, 2020

@vivianQizhu Could you please review this patch again?Thanks

Copy link
Contributor

@vivianQizhu vivianQizhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure you test it before submit the code.

% crash_script, shell=True)
crash_cmd = "crash -i %s /usr/lib/debug/lib/modules/%s/vmlinux %s"
crash_cmd %= (crash_script, host_kernel_version, vmcore_file)
status, output = process.getstatusoutput(crash_cmd, timeout=60)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If status is useless here could be output = process.getoutput(crash_cmd, timeout=60)

host_kernel_version = process.getoutput("uname -r").strip()
env_process.process(test, params, env,
env_process.preprocess_image,
env_process.preprocess_vm)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line could be dropped.

utils_package.package_install(packages)

trigger_core_dump_command = params["trigger_core_dump_command"]
core_file = params["core_file"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where you define core_file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, it is because i test this on my host, then modify the code on my laptop and push it to github.I will be careful next time.
Thanks!

@leidwang leidwang force-pushed the dump_guest_core branch 2 times, most recently from b96c3e9 to 0d12329 Compare November 20, 2020 05:36
@leidwang
Copy link
Contributor Author

JOB ID : 27df29f3a00ade5c2a1bc92251c98058bce38da0
JOB LOG : /root/avocado/job-results/job-2020-11-20T00.31-27df29f/job.log
(1/1) Host_RHEL.m8.u3.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.x86_64.io-github-autotest-qemu.dump_guest_core.on.q35: PASS (212.97 s)

Copy link
Contributor

@vivianQizhu vivianQizhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Otherwise.

crash_script = "/home/crash.cmd"
vmcore_file = "/tmp/vmcore"
core_file = "/var/core.%s"
gdb_command = "gdb %s --command=${gdb_command_file}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean:

core_file = "/var/core"
pre_command += "&& echo "${core_file}.%p" > /proc/sys/kernel/core_pattern"
gdb_command = "gdb ${core_file}.%s --command=${gdb_command_file}"

vmcore_file = "/tmp/vmcore"
core_file = "/var/core.%s"
gdb_command = "gdb %s --command=${gdb_command_file}"
crash_cmd = "crash -i ${crash_script} /usr/lib/debug/lib/modules/%s/vmlinux %s"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here: crash_cmd = "crash -i ${crash_script} /usr/lib/debug/lib/modules/%s/vmlinux ${vmcore_file}"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for your help, the code has been updated.Will update test result later.

@leidwang
Copy link
Contributor Author

JOB ID : c406b1473b242278aa8a8bb9cd02094beeaf3e04
JOB LOG : /root/avocado/job-results/job-2020-11-20T02.58-c406b14/job.log
(1/1) Host_RHEL.m8.u3.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.3.0.x86_64.io-github-autotest-qemu.dump_guest_core.on.q35: PASS (242.70 s)

@vivianQizhu
Copy link
Contributor

@MiriamDeng Request for your re-review since the code changes, thanks.

@MiriamDeng
Copy link
Contributor

It looks good to me from ppc side. Thanks

@MiriamDeng
Copy link
Contributor

JOB LOG : /root/avocado/job-results/job-2020-11-23T23.21-f887006/job.log
(1/2) type_specific.io-github-autotest-qemu.dump_guest_core.on: PASS (139.82 s)
(2/2) type_specific.io-github-autotest-qemu.dump_guest_core.off: PASS (131.57 s)

@leidwang leidwang force-pushed the dump_guest_core branch 2 times, most recently from 5f23eda to 82e81e0 Compare November 24, 2020 05:28
    1.Use gdb check core dump file
    2.Use crash check vmcore file

Signed-off-by: Leidong Wang <leidwang@redhat.com>
@vivianQizhu
Copy link
Contributor

Thanks @leidwang @MiriamDeng

@vivianQizhu vivianQizhu merged commit d6ca032 into autotest:master Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants