Witherspoon QEMU kernel oops: Unable to handle kernel paging request

[geissonator]

I'm starting to get QEMU testing up and running in CI. I've seen this hit once so far, it was not related to the change under test. Lots of other tests have passed fine.

This was testing a master build based on a317e38 using the witherspoon QEMU option.

https://openpower.xyz/job/openbmc-test-qemu-ci/1304/console

21:34:59 [   52.490000] Unable to handle kernel paging request at virtual address 12005452
21:34:59 [   52.490000] pgd = 8d088000
21:34:59 [   52.490000] [12005452] *pgd=00000000
21:34:59 [   52.490000] Internal error: Oops: 5 [#1] ARM
21:34:59 [   52.490000] CPU: 0 PID: 1057 Comm: (agetty) Not tainted 4.10.17-d7ccda2b787f8589c65f3943d854ee0ea9b69c50 #1
21:34:59 [   52.490000] Hardware name: ASpeed SoC
21:34:59 [   52.490000] task: 8d170120 task.stack: 8d140000
21:34:59 [   52.490000] PC is at unlink_anon_vmas+0x68/0x1f4
21:34:59 [   52.490000] LR is at 0x100
21:34:59 [   52.490000] pc : [<800f1560>]    lr : [<00000100>]    psr: 60000113
21:34:59 [   52.490000] sp : 8d141d88  ip : 00000200  fp : 8d141dbc
21:34:59 [   52.490000] r10: 8d00e380  r9 : 8d253d10  r8 : 806362c0
21:34:59 [   52.490000] r7 : 8d253d4c  r6 : 8de521c0  r5 : 1200544a  r4 : 1200544a
21:34:59 [   52.490000] r3 : 00000002  r2 : 8de521c0  r1 : 8d24dc60  r0 : 8d24dc60
21:34:59 [   52.490000] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
21:34:59 [   52.490000] Control: 00c5387d  Table: 8d088008  DAC: 00000055
21:34:59 [   52.490000] Process (agetty) (pid: 1057, stack limit = 0x8d140188)
21:34:59 [   52.490000] Stack: (0x8d141d88 to 0x8d142000)
21:34:59 [   52.490000] 1d80:                   800eac2c 800eb424 8d141dbc 8d253d10 8d253d68 76980000
21:34:59 [   52.490000] 1da0: 8d141df0 00002000 00000000 8d141ea8 8d141dec 8d141dc0 800e5e90 800f1504
21:34:59 [   52.490000] 1dc0: 76980000 8ddb23e0 8d27b478 8ddb23e0 8d141e18 8ddb2720 8ddb23e0 8d19fa00
21:34:59 [   52.490000] 1de0: 8d141e54 8d141df0 800ecd74 800e5e14 8ddb23e0 00000001 00000000 00000000
21:34:59 [   52.490000] 1e00: ffffffff 8001954c 00000000 00000383 00000400 8d275000 8d170120 8d233f00
21:34:59 [   52.490000] 1e20: 8ddb2720 8d0f11b4 8ddb2720 8ddb2878 8d141e6c 8d141e40 80017b90 8ddb23e0
21:34:59 [   52.490000] 1e40: 00000000 8d233f00 8d141e6c 8d141e58 800191e8 800ecc84 8ddb23e0 8d170120
21:34:59 [   52.490000] 1e60: 8d141ea4 8d141e70 80104c6c 800191b4 8d141ea4 8d141e80 8014ab5c 8d0f1180
21:34:59 [   52.490000] 1e80: 8d233f00 00000034 00000001 8d0f11b4 8d19fa00 8d141ea8 8d141f24 8d141ea8
21:34:59 [   52.490000] 1ea0: 8014c1dc 80104878 8d0f11b4 00000034 8ef889a0 80015238 8d141edc 8d141ec8
21:34:59 [   52.490000] 1ec0: 80015238 800dd170 00000000 80103bbc 8d141f44 80109060 8d099000 8d2d0d80
21:34:59 [   52.490000] 1ee0: 8d1e7e60 8d169e20 8d233f00 00000000 8d141f0c 8d141f00 80109060 8d233f00
21:34:59 [   52.490000] 1f00: 805ef540 805ec6ac fffffff8 8d233f00 00000000 00000421 8d141f44 8d141f28
21:34:59 [   52.490000] 1f20: 80105298 8014bf50 8d099000 8d170120 56b1d8e0 56b1d318 8d141f84 8d141f48
21:34:59 [   52.490000] 1f40: 80105720 80105258 8ddb2754 00000421 56b1d318 00000000 0000000b 56b1d8e0
21:34:59 [   52.490000] 1f60: 56b1d318 56b1d8e0 0000000b 8000a6e4 8d140000 00000000 8d141fa4 8d141f88
21:34:59 [   52.490000] 1f80: 80105c24 80105324 00000000 76e83cd0 00000010 56b1d318 00000000 8d141fa8
21:34:59 [   52.490000] 1fa0: 8000a540 80105bf0 00000010 56b1d318 56aa5b98 56b1d8e0 56b1d318 56aa7cd8
21:34:59 [   52.490000] 1fc0: 00000010 56b1d318 56b1d8e0 0000000b 7e96d750 56b22798 56aa66a8 7e96d684
21:34:59 [   52.490000] 1fe0: 54bb673c 7e96d45c 54af5224 76bfc69c 60000010 56aa5b98 00000000 00000000
21:34:59 [   52.490000] [<800f1560>] (unlink_anon_vmas) from [<800e5e90>] (free_pgtables+0x88/0xdc)
21:34:59 [   52.490000] [<800e5e90>] (free_pgtables) from [<800ecd74>] (exit_mmap+0xfc/0x1f8)
21:34:59 [   52.490000] [<800ecd74>] (exit_mmap) from [<800191e8>] (mmput+0x40/0xc4)
21:34:59 [   52.490000] [<800191e8>] (mmput) from [<80104c6c>] (flush_old_exec+0x400/0x564)
21:34:59 [   52.490000] [<80104c6c>] (flush_old_exec) from [<8014c1dc>] (load_elf_binary+0x298/0x10c8)
21:34:59 [   52.490000] [<8014c1dc>] (load_elf_binary) from [<80105298>] (search_binary_handler+0x4c/0xcc)
21:34:59 [   52.490000] [<80105298>] (search_binary_handler) from [<80105720>] (do_execveat_common+0x408/0x654)
21:34:59 [   52.490000] [<80105720>] (do_execveat_common) from [<80105c24>] (SyS_execve+0x40/0x48)
21:34:59 [   52.490000] [<80105c24>] (SyS_execve) from [<8000a540>] (ret_fast_syscall+0x0/0x34)
21:34:59 [   52.490000] Code: e585e008 e585c00c eb0020d5 e1a05004 (e5b43008) 
21:34:59 [   52.670000] ---[ end trace 20c787743ffc52cf ]---

[geissonator]

Hit another two here https://openpower.xyz/job/openbmc-test-qemu-ci/1307/console and here https://openpower.xyz/job/openbmc-test-qemu-ci/1312/console

FYI @cyrilbur-ibm @amboar

[amboar]

I doubt @cyrilbur-ibm is interested, though @legoater might be :)

[legoater]

This is the same issue reported in openbmc/qemu#9. We have a couple like this one for the 4.10 kernel. I wonder if this is fixed in 4.13.

[mdmillerii]

One of the logs linked by @geissonator was already gone when I looked, and the second is now gone, but it was also at exit / unlink_anon_vmas when agetty was executing, with a similar faulting address 12005456 from memory.

[geissonator]

Was kind of hoping this had gone away with our move to 4.13 but we still hit it with a recent master test:

16:56:35 [ 82.646346] Unable to handle kernel paging request at virtual address 12005456
16:56:35 [ 82.648254] pgd = 8cab8000
16:56:35 [ 82.648562] [12005456] *pgd=00000000
16:56:35 [ 82.649510] Internal error: Oops: 805 [#1] ARM
16:56:35 [ 82.650466] CPU: 0 PID: 919 Comm: env Not tainted 4.13.16-d86bf903d3fe4d19b3b11c92b19b736ee07edc5d #1
16:56:35 [ 82.651063] Hardware name: Generic DT based system
16:56:35 [ 82.651446] task: 8caa0aa0 task.stack: 8caac000
16:56:35 [ 82.652245] PC is at unlink_anon_vmas+0xbc/0x1f4
16:56:35 [ 82.652610] LR is at 0x12005452
16:56:35 [ 82.652818] pc : [<802044a8>] lr : [<12005452>] psr: a0000013
16:56:35 [ 82.653167] sp : 8caadc90 ip : 00000100 fp : 8caadcc4
16:56:35 [ 82.653458] r10: 8caae380 r9 : 809642a8 r8 : 8caaf420
16:56:35 [ 82.653759] r7 : 8caaf45c r6 : 8caae380 r5 : 8caa1160 r4 : 1200544a
16:56:35 [ 82.654118] r3 : 8caa5634 r2 : 00000200 r1 : 8caa1160 r0 : 8e400500
16:56:35 [ 82.654525] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
16:56:35 [ 82.654957] Control: 00c5387d Table: 8cab8008 DAC: 00000051
16:56:35 [ 82.655312] Process env (pid: 919, stack limit = 0x8caac188)
16:56:35 [ 82.655677] Stack: (0x8caadc90 to 0x8caae000)
16:56:35 [ 82.656154] dc80: 801fcf60 801fd778 8caadcc4 8caaf420
16:56:35 [ 82.656831] dca0: 8caa4630 00010000 8caadcfc 00002000 00000000 00000001 8caadcf4 8caadcc8
16:56:35 [ 82.657462] dcc0: 801f8160 802043f8 00000000 801d81fc 8caaff20 8c96c720 80903008 8c96c720
16:56:35 [ 82.658112] dce0: 8c990000 8ca68aa0 8caadd6c 8caadcf8 801ff28c 801f80e4 00000000 8c96c720
16:56:35 [ 82.658757] dd00: 00000001 00000000 00000000 ffffffff 8ca92288 8e125738 00000139 00000400
16:56:35 [ 82.659410] dd20: 8a8a1000 00000001 8caadd64 80121868 8dd6db80 8dd6dcd8 8caadd84 8caadd48
16:56:35 [ 82.660105] dd40: 80111334 42e1a036 00000000 8c96c720 00000000 8dd6db80 8c990000 8ca68aa0
16:56:35 [ 82.660849] dd60: 8caadd84 8caadd70 801134f8 801ff1bc 8c96c720 8caa0aa0 8caaddbc 8caadd88
16:56:35 [ 82.661526] dd80: 80218974 801134bc 8caaddbc 8caadd98 80262944 8a816f00 80903008 8c990000
16:56:35 [ 82.662216] dda0: 8caaddc0 8dedea00 00000034 00000001 8caade54 8caaddc0 80263fd4 80218708
16:56:35 [ 82.662899] ddc0: 8a816f34 00000034 00000001 00000080 00000397 8c990000 8caade14 8caadde8
16:56:35 [ 82.663585] dde0: 802111a4 80254428 00000000 00000000 802e6cc8 ffffe000 80903008 7f000000
16:56:35 [ 82.664258] de00: 00000000 8cb3e020 8ca92280 8a892320 8021830c 802110b4 00000080 00000000
16:56:35 [ 82.664982] de20: 8e142918 42e1a036 8c990000 8c990000 8092556c 809226a4 fffffff8 7ebc1e68
16:56:35 [ 82.665690] de40: 00000397 8c990000 8caade74 8caade58 80219134 80263d38 8c990000 80903008
16:56:35 [ 82.666346] de60: 809226a4 8dd6db80 8caadf1c 8caade78 802628b0 802190f4 00000000 8c99000f
16:56:35 [ 82.667032] de80: 8c990002 7273752f 6e69622f 766e652f 00002000 8caadf08 00000000 00000000
16:56:35 [ 82.667700] dea0: 8dd6dbb4 00000000 8caa0aa0 8010e1a8 80903018 8010e5ac 8caadedc 8caadec8
16:56:35 [ 82.668396] dec0: 8010e5ac 801ede44 00000000 8021d14c 8de3b000 8b40d1f8 00000000 8c990000
16:56:35 [ 82.669091] dee0: 7ebc1e68 7ebc1e70 8caadf04 8caadef8 8021d14c 8021cf74 8caadf14 8caadf08
16:56:35 [ 82.669804] df00: 8021d1cc 42e1a036 8c990000 80925550 8caadf3c 8caadf20 80219134 80262668
16:56:35 [ 82.670541] df20: 8de3b000 80903008 8caa0aa0 8dd6db80 8caadf84 8caadf40 80219708 802190f4
16:56:35 [ 82.671243] df40: 00000000 8dd6dbb4 00000397 00000000 00000000 42e1a036 0000000b 7ebc1e68
16:56:35 [ 82.671896] df60: 7ebc1e70 7ebc1e70 0000000b 80102804 8caac000 00000000 8caadfa4 8caadf88
16:56:35 [ 82.672592] df80: 80219b20 802191c4 00000000 80000010 7ebc1f82 7ebc1e68 00000000 8caadfa8
16:56:35 [ 82.673279] dfa0: 80102640 80219aec 7ebc1f82 7ebc1e68 7ebc1c20 7ebc1e68 7ebc1e70 00000000
16:56:35 [ 82.673961] dfc0: 7ebc1f82 7ebc1e68 7ebc1e70 0000000b 00000013 0000003d 7ebc1c20 7ebc1ca4
16:56:35 [ 82.674715] dfe0: 00000010 7ebc1c1c 48afc6a0 48afc00c 80000010 7ebc1c20 00000000 00000000
16:56:35 [ 82.676115] [<802044a8>] (unlink_anon_vmas) from [<801f8160>] (free_pgtables+0x88/0xdc)
16:56:35 [ 82.676659] [<801f8160>] (free_pgtables) from [<801ff28c>] (exit_mmap+0xdc/0x13c)
16:56:35 [ 82.677149] [<801ff28c>] (exit_mmap) from [<801134f8>] (mmput+0x48/0xc4)
16:56:35 [ 82.677560] [<801134f8>] (mmput) from [<80218974>] (flush_old_exec+0x278/0x56c)
16:56:35 [ 82.678023] [<80218974>] (flush_old_exec) from [<80263fd4>] (load_elf_binary+0x2a8/0x13dc)
16:56:35 [ 82.678522] [<80263fd4>] (load_elf_binary) from [<80219134>] (search_binary_handler+0x4c/0xd0)
16:56:35 [ 82.679053] [<80219134>] (search_binary_handler) from [<802628b0>] (load_script+0x254/0x274)
16:56:35 [ 82.679584] [<802628b0>] (load_script) from [<80219134>] (search_binary_handler+0x4c/0xd0)
16:56:35 [ 82.680088] [<80219134>] (search_binary_handler) from [<80219708>] (do_execveat_common+0x550/0x6ac)
16:56:35 [ 82.680662] [<80219708>] (do_execveat_common) from [<80219b20>] (SyS_execve+0x40/0x48)
16:56:35 [ 82.681177] [<80219b20>] (SyS_execve) from [<80102640>] (ret_fast_syscall+0x0/0x3c)
16:56:35 [ 82.681941] Code: 0affffe4 e595300c e595e008 e5990004 (e58e3004)
16:56:35 [ 82.683171] ---[ end trace 9f4e8b1440f9b6cc ]---
16:56:35 [ 82.683641] Kernel panic - not syncing: Fatal exception

[mdmillerii]

similar fault address but now env instead of agetty.

[stale]

This issue has been automatically marked as stale because no activity has occurred in the last 6 months. It will be closed if no activity occurs in the next 30 days. If this issue should not be closed please add a comment. Thank you for your understanding and contributions.

[amboar]

Seen this recently @geissonator ?

[stale]

This issue has been automatically marked as stale because no activity has occurred in the last 6 months. It will be closed if no activity occurs in the next 30 days. If this issue should not be closed please add a comment. Thank you for your understanding and contributions.

[stale]

This issue has been closed because no activity has occurred in the last 7 months. Please reopen if this issue should not have been closed. Thank you for your contributions.