Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

my replay just end without error #141

Closed
zshu1 opened this issue Feb 22, 2017 · 11 comments
Closed

my replay just end without error #141

zshu1 opened this issue Feb 22, 2017 · 11 comments

Comments

@zshu1
Copy link

zshu1 commented Feb 22, 2017 •
edited
Loading

I run a worm in windows-7-32bit and use qemu-system-i386.

zhanshu@cougar:~/tools/panda/build-panda/i386-softmmu$
sudo ./qemu-system-i386 -m 1024 ~/images/win7_test2/snapshot1.img -monitor stdio
QEMU 2.7.90 monitor - type 'help' for more information
(qemu) begin_record worm3
(qemu) writing snapshot: ./worm3-rr-snp
opening nondet log for write : ./worm3-rr-nondet.log
end_record
(qemu) Time taken was: 127 seconds.

and now I get a record. Then I tried to replay it

loading snapshot
... done.
opening nondet log for read : ./worm3-rr-nondet.log
total_instr in replay: 10283332009
worm3: 102956137 ( 1.00%) instrs. 2.12 sec. 1.00 GB ram.
worm3: 205990156 ( 2.00%) instrs. 5.32 sec. 1.06 GB ram.
worm3: 309241486 ( 3.01%) instrs. 8.33 sec. 1.10 GB ram.
worm3: 413319106 ( 4.02%) instrs. 10.96 sec. 1.14 GB ram.
worm3: 515996469 ( 5.02%) instrs. 12.62 sec. 1.16 GB ram.
worm3: 617719878 ( 6.01%) instrs. 15.11 sec. 1.19 GB ram.
worm3: 720020510 ( 7.00%) instrs. 17.33 sec. 1.21 GB ram.
worm3: 823464399 ( 8.01%) instrs. 19.47 sec. 1.21 GB ram.
worm3: 926965247 ( 9.01%) instrs. 21.24 sec. 1.22 GB ram.
worm3: 1029143515 ( 10.01%) instrs. 23.48 sec. 1.23 GB ram.
worm3: 1131566991 ( 11.00%) instrs. 25.72 sec. 1.25 GB ram.
worm3: 1235750808 ( 12.02%) instrs. 27.67 sec. 1.26 GB ram.
worm3: 1337278536 ( 13.00%) instrs. 29.52 sec. 1.26 GB ram.
worm3: 1440135058 ( 14.00%) instrs. 31.86 sec. 1.28 GB ram.
worm3: 1542594997 ( 15.00%) instrs. 34.13 sec. 1.29 GB ram.
worm3: 1645383436 ( 16.00%) instrs. 36.55 sec. 1.30 GB ram.
worm3: 1748346459 ( 17.00%) instrs. 39.93 sec. 1.30 GB ram.
worm3: 1854578935 ( 18.03%) instrs. 41.74 sec. 1.30 GB ram.
worm3: 1953890351 ( 19.00%) instrs. 43.74 sec. 1.30 GB ram.
worm3: 2057476564 ( 20.01%) instrs. 46.31 sec. 1.30 GB ram.
worm3: 2160670806 ( 21.01%) instrs. 48.65 sec. 1.30 GB ram.
worm3: 2263983991 ( 22.02%) instrs. 51.17 sec. 1.30 GB ram.
worm3: 2365620588 ( 23.00%) instrs. 52.69 sec. 1.30 GB ram.
worm3: 2468589477 ( 24.01%) instrs. 55.08 sec. 1.31 GB ram.
worm3: 2571340806 ( 25.00%) instrs. 56.90 sec. 1.31 GB ram.
worm3: 2673877359 ( 26.00%) instrs. 59.45 sec. 1.32 GB ram.
worm3: 2776806819 ( 27.00%) instrs. 61.56 sec. 1.32 GB ram.
worm3: 2879404659 ( 28.00%) instrs. 64.55 sec. 1.32 GB ram.
worm3: 2988035451 ( 29.06%) instrs. 66.06 sec. 1.32 GB ram.
worm3: 3088603572 ( 30.04%) instrs. 67.36 sec. 1.32 GB ram.
worm3: 3188632935 ( 31.01%) instrs. 68.60 sec. 1.32 GB ram.
worm3: 3291608687 ( 32.01%) instrs. 69.66 sec. 1.32 GB ram.
worm3: 3393729640 ( 33.00%) instrs. 70.87 sec. 1.32 GB ram.
worm3: 3496342117 ( 34.00%) instrs. 71.95 sec. 1.32 GB ram.
worm3: 3599554273 ( 35.00%) instrs. 73.27 sec. 1.32 GB ram.
worm3: 3703856201 ( 36.02%) instrs. 74.53 sec. 1.32 GB ram.
worm3: 3804918764 ( 37.00%) instrs. 76.25 sec. 1.32 GB ram.
worm3: 3907799223 ( 38.00%) instrs. 77.13 sec. 1.32 GB ram.
worm3: 4011084817 ( 39.01%) instrs. 77.91 sec. 1.32 GB ram.
worm3: 4114099553 ( 40.01%) instrs. 78.79 sec. 1.32 GB ram.
worm3: 4216347969 ( 41.00%) instrs. 80.40 sec. 1.32 GB ram.
worm3: 4319149187 ( 42.00%) instrs. 83.32 sec. 1.32 GB ram.
worm3: 4422859328 ( 43.01%) instrs. 86.46 sec. 1.32 GB ram.
worm3: 4524981729 ( 44.00%) instrs. 88.99 sec. 1.32 GB ram.
worm3: 4628065399 ( 45.01%) instrs. 91.80 sec. 1.32 GB ram.
zhanshu@cougar:~/tools/panda/build-panda/i386-softmmu$

then it finished. at 45%.
I use snapshot of QEMU to run the record , does it matters ?
@phulin
Copy link
Contributor

phulin commented Feb 23, 2017 via email

@moyix
Copy link
Collaborator

moyix commented Feb 23, 2017

You may also want to try running PANDA under gdb and setting a breakpoint on the _exit function. When PANDA exits, you can use bt to get a backtrace and find out what caused it to exit.

@zshu1
Copy link
Author

zshu1 commented Feb 23, 2017 •
edited
Loading

thank you all for replying. I tried gdb to catch the crash point and get information bellow:

worm3: 4524981729 ( 44.00%) instrs. 87.95 sec. 1.32 GB ram.
worm3: 4628065399 ( 45.01%) instrs. 90.50 sec. 1.32 GB ram.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd1c65700 (LWP 4160)]
0x0000555555a142ff in memory_region_del_subregion (mr=0x0,
subregion=0x55555721ec70) at /home/zhanshu/tools/panda/panda/memory.c:2082
2082 QTAILQ_REMOVE(&mr->subregions, subregion, subregions_link);
(gdb)

and the backtrace stack here:

(gdb) bt
#0 0x0000555555a142ff in memory_region_del_subregion (mr=0x0,
subregion=0x55555721ec70) at /home/zhanshu/tools/panda/panda/memory.c:2082
#1 0x00005555559a746b in rr_replay_skipped_calls_internal (
call_site=RR_CALLSITE_STW_INTERNAL)
at /home/zhanshu/tools/panda/panda/panda/src/rr/rr_log.c:1164
#2 0x00005555559c5544 in rr_replay_skipped_calls ()
at /home/zhanshu/tools/panda/panda/panda/include/panda/rr/rr_log_all.h:457
#3 address_space_stw_internal (endian=DEVICE_NATIVE_ENDIAN, result=0x0,
attrs=..., val=256, addr=, as=)
at /home/zhanshu/tools/panda/panda/exec.c:3717
#4 address_space_stw (as=, addr=, val=256,
attrs=..., result=result@entry=0x0)
at /home/zhanshu/tools/panda/panda/exec.c:3755
#5 0x0000555555ad824e in helper_outw (env=,
port=, data=)
at /home/zhanshu/tools/panda/panda/target-i386/misc_helper.c:61
#6 0x00007fffda8d2b3c in ?? ()
#7 0x000000001fbbd000 in ?? ()
#8 0x00000005fffffe98 in ?? ()
#9 0x00005555571b2750 in ?? ()
#10 0xa2587152d5f2c600 in ?? ()
#11 0x000000009615e000 in ?? ()
#12 0x00005555571b2750 in ?? ()
---Type to continue, or q to quit---
#13 0x0000000000000005 in ?? ()
#14 0x00005555571ba9e0 in ?? ()
#15 0x00007fff808a8690 in ?? ()
#16 0x0000555555a162c3 in tlb_set_page_with_attrs (cpu=0x100, vaddr=964,
paddr=140736860072368, attrs=..., prot=1460353440, mmu_idx=1437418800,
size=0) at /home/zhanshu/tools/panda/panda/cputlb.c:381
#17 0x0000000000000000 in ?? ()

I want to add -g flag to PANDA then I can try to debug by myself but after reading makefile I can't find CFLAGS where it is defined so can't add symbol to gdb when compile. Could someone also help me on that?

@moyix
Copy link
Collaborator

moyix commented Feb 24, 2017

OK – it looks like this is a known bug (related to handling I/O memory map changes). It usually only manifests during system boot, though. Does this happen consistently? And are you able to share the malware sample that triggers the crash?

@zshu1
Copy link
Author

zshu1 commented Feb 27, 2017 via email •
edited
Loading

@moyix
Copy link
Collaborator

moyix commented Feb 27, 2017

Sending the malware is best – to debug we will need to reproduce the recording and replay process.

@zshu1
Copy link
Author

zshu1 commented Feb 27, 2017 via email

@moyix
Copy link
Collaborator

moyix commented Feb 27, 2017

Could you send it to brendandg@nyu.edu ? If that doesn't work I can try to set up a place for you to upload it directly.

@zshu1
Copy link
Author

zshu1 commented Feb 27, 2017 via email

@moyix
Copy link
Collaborator

moyix commented Mar 1, 2017

FYI, I never received this so I think some mail filter along the way deleted it. Maybe you can host it somewhere for download?

@zshu1
Copy link
Author

zshu1 commented Mar 13, 2017

I can't reproduce this bug. I running 5 times and 2 of them works well.

@zshu1 zshu1 closed this as completed Mar 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@moyix @phulin @zshu1