Found one, now what?

[sokoow]

Hey, sorry to be thick, but this is the first time I'm doing this ... So after 3 days of fuzzing 4.6.5, I got this single crash:

start up afl forkserver!
Input from outputs/M0/crashes/id:000000,sig:00,src:002140,op:ext_AO,pos:4 at time 1471587966.490247
test running in pid 29239
call 28
arg 0: argNum 40001d
arg 1: argNum 6400000707000000
arg 2: argNum 1d00640000000064
arg 3: argVec64 cafbf0 - size 0
arg 4: argNum 1d00000064
arg 5: argBuflen 0
read 44 bytes, parse result 0 nrecs 1
syscall 28 (40001d, 6400000707000000, 1d00640000000064, cafbf0, 1d00000064, 0)
[   18.875920] Injecting memory failure for page 0x35f2 at 0x40001d
[   18.875920] MCE 0x35f2: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f3 at 0x40101d
[   18.875920] MCE 0x35f3: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f4 at 0x40201d
[   18.875920] MCE 0x35f4: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f5 at 0x40301d
[   18.875920] MCE 0x35f5: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f6 at 0x40401d
[   18.875920] MCE 0x35f6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f7 at 0x40501d
[   18.875920] MCE 0x35f7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f8 at 0x40601d
[   18.875920] MCE 0x35f8: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35f9 at 0x40701d
[   18.875920] MCE 0x35f9: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35fa at 0x40801d
[   18.875920] MCE 0x35fa: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35fb at 0x40901d
[   18.875920] MCE 0x35fb: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35fc at 0x40a01d
[   18.875920] MCE 0x35fc: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35fd at 0x40b01d
[   18.875920] MCE 0x35fd: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35fe at 0x40c01d
[   18.875920] MCE 0x35fe: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x35ff at 0x40d01d
[   18.875920] MCE 0x35ff: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2f6 at 0x40e01d
[   18.875920] MCE 0x2f6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2f7 at 0x40f01d
[   18.875920] MCE 0x2f7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2f8 at 0x41001d
[   18.875920] MCE 0x2f8: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2f9 at 0x41101d
[   18.875920] MCE 0x2f9: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2fa at 0x41201d
[   18.875920] MCE 0x2fa: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2fb at 0x41301d
[   18.875920] MCE 0x2fb: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2fc at 0x41401d
[   18.875920] MCE 0x2fc: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2fd at 0x41501d
[   18.875920] MCE 0x2fd: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2fe at 0x41601d
[   18.875920] MCE 0x2fe: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x2ff at 0x41701d
[   18.875920] MCE 0x2ff: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x300 at 0x41801d
[   18.875920] MCE 0x300: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x301 at 0x41901d
[   18.875920] MCE 0x301: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x302 at 0x41a01d
[   18.875920] MCE 0x302: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x303 at 0x41b01d
[   18.875920] MCE 0x303: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x304 at 0x41c01d
[   18.875920] MCE 0x304: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x305 at 0x41d01d
[   18.875920] MCE 0x305: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x306 at 0x41e01d
[   18.875920] MCE 0x306: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x307 at 0x41f01d
[   18.875920] MCE 0x307: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x308 at 0x42001d
[   18.875920] MCE 0x308: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x309 at 0x42101d
[   18.875920] MCE 0x309: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30a at 0x42201d
[   18.875920] MCE 0x30a: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c0 at 0x42301d
[   18.875920] MCE 0x39c0: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c1 at 0x42401d
[   18.875920] MCE 0x39c1: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c2 at 0x42501d
[   18.875920] MCE 0x39c2: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c3 at 0x42601d
[   18.875920] MCE 0x39c3: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c4 at 0x42701d
[   18.875920] MCE 0x39c4: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c5 at 0x42801d
[   18.875920] MCE 0x39c5: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c6 at 0x42901d
[   18.875920] MCE 0x39c6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c7 at 0x42a01d
[   18.875920] MCE 0x39c7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c8 at 0x42b01d
[   18.875920] MCE 0x39c8: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39c9 at 0x42c01d
[   18.875920] MCE 0x39c9: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ca at 0x42d01d
[   18.875920] MCE 0x39ca: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39cb at 0x42e01d
[   18.875920] MCE 0x39cb: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39cc at 0x42f01d
[   18.875920] MCE 0x39cc: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39cd at 0x43001d
[   18.875920] MCE 0x39cd: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ce at 0x43101d
[   18.875920] MCE 0x39ce: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39cf at 0x43201d
[   18.875920] MCE 0x39cf: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d0 at 0x43301d
[   18.875920] MCE 0x39d0: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d1 at 0x43401d
[   18.875920] MCE 0x39d1: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d2 at 0x43501d
[   18.875920] MCE 0x39d2: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d3 at 0x43601d
[   18.875920] MCE 0x39d3: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d4 at 0x43701d
[   18.875920] MCE 0x39d4: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d5 at 0x43801d
[   18.875920] MCE 0x39d5: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d6 at 0x43901d
[   18.875920] MCE 0x39d6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d7 at 0x43a01d
[   18.875920] MCE 0x39d7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d8 at 0x43b01d
[   18.875920] MCE 0x39d8: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39d9 at 0x43c01d
[   18.875920] MCE 0x39d9: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39da at 0x43d01d
[   18.875920] MCE 0x39da: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39db at 0x43e01d
[   18.875920] MCE 0x39db: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39dc at 0x43f01d
[   18.875920] MCE 0x39dc: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39dd at 0x44001d
[   18.875920] MCE 0x39dd: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39de at 0x44101d
[   18.875920] MCE 0x39de: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30b at 0x44201d
[   18.875920] MCE 0x30b: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30c at 0x44301d
[   18.875920] MCE 0x30c: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30d at 0x44401d
[   18.875920] MCE 0x30d: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30e at 0x44501d
[   18.875920] MCE 0x30e: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x30f at 0x44601d
[   18.875920] MCE 0x30f: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x310 at 0x44701d
[   18.875920] MCE 0x310: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x311 at 0x44801d
[   18.875920] MCE 0x311: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x312 at 0x44901d
[   18.875920] MCE 0x312: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x313 at 0x44a01d
[   18.875920] MCE 0x313: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x314 at 0x44b01d
[   18.875920] MCE 0x314: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x315 at 0x44c01d
[   18.875920] MCE 0x315: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x316 at 0x44d01d
[   18.875920] MCE 0x316: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x317 at 0x44e01d
[   18.875920] MCE 0x317: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x318 at 0x44f01d
[   18.875920] MCE 0x318: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x319 at 0x45001d
[   18.875920] MCE 0x319: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31a at 0x45101d
[   18.875920] MCE 0x31a: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31b at 0x45201d
[   18.875920] MCE 0x31b: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31c at 0x45301d
[   18.875920] MCE 0x31c: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31d at 0x45401d
[   18.875920] MCE 0x31d: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31e at 0x45501d
[   18.875920] MCE 0x31e: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x31f at 0x45601d
[   18.875920] MCE 0x31f: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39df at 0x45701d
[   18.875920] MCE 0x39df: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e0 at 0x45801d
[   18.875920] MCE 0x39e0: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e1 at 0x45901d
[   18.875920] MCE 0x39e1: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e2 at 0x45a01d
[   18.875920] MCE 0x39e2: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e3 at 0x45b01d
[   18.875920] MCE 0x39e3: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e4 at 0x45c01d
[   18.875920] MCE 0x39e4: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e5 at 0x45d01d
[   18.875920] MCE 0x39e5: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e6 at 0x45e01d
[   18.875920] MCE 0x39e6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e7 at 0x45f01d
[   18.875920] MCE 0x39e7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e8 at 0x46001d
[   18.875920] MCE 0x39e8: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39e9 at 0x46101d
[   18.875920] MCE 0x39e9: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ea at 0x46201d
[   18.875920] MCE 0x39ea: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39eb at 0x46301d
[   18.875920] MCE 0x39eb: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ec at 0x46401d
[   18.875920] MCE 0x39ec: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ed at 0x46501d
[   18.875920] MCE 0x39ed: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ee at 0x46601d
[   18.875920] MCE 0x39ee: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39ef at 0x46701d
[   18.875920] MCE 0x39ef: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f0 at 0x46801d
[   18.875920] MCE 0x39f0: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f1 at 0x46901d
[   18.875920] MCE 0x39f1: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f2 at 0x46a01d
[   18.875920] MCE 0x39f2: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f3 at 0x46b01d
[   18.875920] MCE 0x39f3: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f4 at 0x46c01d
[   18.875920] MCE 0x39f4: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f5 at 0x46d01d
[   18.875920] MCE 0x39f5: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f6 at 0x46e01d
[   18.875920] MCE 0x39f6: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f7 at 0x46f01d
[   18.875920] MCE 0x39f7: recovery action for dirty LRU page: Recovered
[   18.875920] Injecting memory failure for page 0x39f8 at 0x47001d
[   18.875920] MCE 0x39f8: dirty LRU page still referenced by 1 users
[   18.875920] MCE 0x39f8: recovery action for dirty LRU page: Failed
[   18.875920] MCE: Killing driver:123 due to hardware memory corruption fault at 444ca9
[   18.875920] MCE: Killing driver:1 due to hardware memory corruption fault at 441c8a
[   18.875920] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000007
[   18.875920] 
[   18.875920] CPU: 0 PID: 1 Comm: driver Not tainted 4.6.5-rt9 #1
[   18.875920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014
[   18.875920]  0000000000000086 000000008ac765a6 ffff880002d93c30 ffffffff81400776
[   18.875920]  ffffffff81c84be0 ffff880002d93cc8 ffff880002d93cb8 ffffffff811a12ae
[   18.875920]  ffffffff00000010 ffff880002d93cc8 ffff880002d93c60 000000008ac765a6
[   18.875920] Call Trace:
[   18.875920]  [<ffffffff81400776>] dump_stack+0x63/0x8d
[   18.875920]  [<ffffffff811a12ae>] panic+0xde/0x220
[   18.875920]  [<ffffffff810882b7>] do_exit+0xb77/0xb80
[   18.875920]  [<ffffffff81088350>] do_group_exit+0x50/0xd0
[   18.875920]  [<ffffffff81094dc2>] get_signal+0x282/0x680
[   18.875920]  [<ffffffff810aaeea>] ? migrate_enable+0x7a/0x130
[   18.875920]  [<ffffffff8102f5b7>] do_signal+0x37/0x770
[   18.875920]  [<ffffffff810e3699>] ? vprintk_default+0x29/0x40
[   18.875920]  [<ffffffff811a15e6>] ? printk+0x57/0x73
[   18.875920]  [<ffffffff810e3699>] ? vprintk_default+0x29/0x40
[   18.875920]  [<ffffffff8106cbd1>] ? mm_fault_error+0x141/0x190
[   18.875920]  [<ffffffff8106d110>] ? __do_page_fault+0x4f0/0x540
[   18.875920]  [<ffffffff810032b4>] exit_to_usermode_loop+0xf4/0x150
[   18.875920]  [<ffffffff81003d98>] prepare_exit_to_usermode+0x38/0x40
[   18.875920]  [<ffffffff818537ef>] retint_user+0x8/0x10
[   18.875920] Kernel Offset: disabled

timeout
test ended with status 9
5964 edges

fork server ended with status 200
boot time:  29.70
test time:  2.02
total time: 31.72
tests:     1
execs/sec: 0.49

I can provide you with the crash input file also. Now, questions:

1. It's madvise right ? How do I know whether this is kernel side or qemu side of things ?
2. Is there any way to re-run this on bare hardware/VM outside of qemu?
3. Does it look legit ?

[sokoow]

Oh, so it's as simple as running this on bare metal right?

#include <sys/mman.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>


int main(void) {

        madvise((void*)0x40001d, 0x6400000707000000, (int)0x1d00640000000064);

        return 0;
}

and I got:

# ./a.out 
Bus error
# ./a.out 
bash: ./a.out: cannot execute binary file: Exec format error

dmesg:

[90727.774210] Injecting memory failure for page 0x39185 at 0x40001d
[90727.774387] MCE 0x39185: recovery action for dirty LRU page: Recovered
[90727.774424] MCE: Killing a.out:19423 due to hardware memory corruption fault at 400543
[90761.017705] JBD2: Detected IO errors while flushing file data on dm-0-8
[91341.599164] Injecting memory failure for page 0x1914bb at 0x40001d
[91341.599348] MCE 0x1914bb: recovery action for dirty LRU page: Recovered
[91341.599389] MCE: Killing a.out:19975 due to hardware memory corruption fault at 400543

[timnewsham]

Oh, so it's as simple as running this on bare metal right?

You could also have taken the input file and run "./driver -tv < filename" on bare metal or inside the emulated environment. There are some notes on debugging in the readme that comes with the linux fuzzer.
We've run across this particular crash ourselves and observed that this one is caused by madvise() triggering a fake memory corruption issue (it’s a feature of madvise that I wasn’t aware of!) that is causing /bin/driver to get killed (not just the driver, but also the parent of the driver which is acting as a watchdog!). If you look at the root template's "init" file, after the driver runs, init finishes and when init dies, the kernel panics. That’s why you see error messages when running it on bare metal but you don't get the panic (because init didn’t get killed and didn’t exit).

This is the madvise flag that is being used:

   MADV_HWPOISON (Since Linux 2.6.32)
          Poison  a  page and handle it like a hardware memory corruption.
          This operation is available only for privileged  (CAP_SYS_ADMIN)
          processes.   This  operation  may  result in the calling process
          receiving a SIGBUS and the page being unmapped.  This feature is
          intended for testing of memory error-handling code; it is avail‐
          able only if the kernel was configured with  CONFIG_MEMORY_FAIL‐
          URE.

It requires a capability in the original namespace that only the "real" root has.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[sokoow]

What's actually interesting is the behavior of C program, so this:

[90761.017705] JBD2: Detected IO errors while flushing file data on dm-0-

It actually manages to damage the executable on disk, I can't run it anymore even after a reboot. So is this the case that it poisons page-cache somehow, and then that page gets written back to disk?