New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rkt run triggers kernel panic when using btrfs #1110

Closed
crawford opened this Issue Feb 9, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@crawford
Member

crawford commented Feb 9, 2016

When rkt-running an image on CoreOS 949.0.0 with a btrfs root, I'm seeing the following kernel panic:

[77569275.470475] BTRFS error (device xvda9): could not find root 8
[77569275.482284] BTRFS error (device xvda9): could not find root 8
[77569275.551296] BUG: unable to handle kernel NULL pointer dereference at 0000000000000334
[77569275.551314] IP: [<ffffffffc0115369>] btrfs_sync_file+0xc9/0x8b0 [btrfs]
[77569275.551333] PGD 0 
[77569275.551338] Oops: 0002 [#1] SMP 
[77569275.551344] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack veth overlay nls_ascii nls_cp437 vfat fat xenfs xen_privcmd ext4 crc16 mbcache jbd2 btrfs xor lzo_compress raid6_pq crc32c_intel jitterentropy_rng hmac drbg aesni_intel aes_x86_64 glue_helper lrw xen_netfront gf128mul xen_blkfront ablk_helper cryptd sch_fq_codel ip_tables autofs4
[77569275.551405] CPU: 0 PID: 876 Comm: systemd-journal Not tainted 4.4.0-coreos-r2 #2
[77569275.551416] task: ffff880021718000 ti: ffff88002196c000 task.ti: ffff88002196c000
[77569275.551426] RIP: e030:[<ffffffffc0115369>]  [<ffffffffc0115369>] btrfs_sync_file+0xc9/0x8b0 [btrfs]
[77569275.551445] RSP: e02b:ffff88002196fe48  EFLAGS: 00010246
[77569275.551451] RAX: ffff880021718000 RBX: ffff880021979700 RCX: ffffffff81a6af40
[77569275.551461] RDX: 0000000080000000 RSI: 0000000000000000 RDI: ffff880022c77268
[77569275.551470] RBP: ffff88002196fec8 R08: dcb5ed0a024ea1ae R09: 0000000000000001
[77569275.551479] R10: 00005648d14425c0 R11: 0000000000000246 R12: ffff880022c77268
[77569275.551489] R13: ffff880022c771c0 R14: 0000000000000000 R15: 0000000000000000
[77569275.551505] FS:  00007f02c420a840(0000) GS:ffff880026200000(0000) knlGS:0000000000000000
[77569275.551514] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[77569275.551521] CR2: 0000000000000334 CR3: 000000002197e000 CR4: 0000000000002660
[77569275.551530] Stack:
[77569275.551536]  ffff88002196fee8 ffff88002196fed8 8000000000000000 ffff880022c156c0
[77569275.551549]  7fffffffffffffff 0000000000000000 0000000000000000 0000000000000000
[77569275.551561]  0000000000000000 0000000000000000 000000008fa52c02 ffff880021979700
[77569275.551573] Call Trace:
[77569275.551585]  [<ffffffff8120279d>] vfs_fsync_range+0x3d/0xb0
[77569275.551598]  [<ffffffff810038bf>] ? syscall_trace_enter_phase1+0x11f/0x140
[77569275.551606]  [<ffffffff8120286d>] do_fsync+0x3d/0x70
[77569275.551613]  [<ffffffff81202b00>] SyS_fsync+0x10/0x20
[77569275.551623]  [<ffffffff81553f2e>] entry_SYSCALL_64_fastpath+0x12/0x71
[77569275.551630] Code: ff d1 49 8b 0e 48 85 c9 75 e8 eb 9e 48 8b 45 a0 48 2b 45 a8 4d 8d a5 a8 00 00 00 4c 89 e7 48 83 c0 01 48 89 45 90 e8 f7 c7 43 c1 <3e> 41 ff 87 34 03 00 00 49 8b 85 70 ff ff ff 48 c1 e8 07 83 e0 
[77569275.551693] RIP  [<ffffffffc0115369>] btrfs_sync_file+0xc9/0x8b0 [btrfs]
[77569275.551706]  RSP <ffff88002196fe48>
[77569275.551711] CR2: 0000000000000334
[77569275.551728] ---[ end trace 01b15b9e6646da56 ]---
[77569275.551735] Kernel panic - not syncing: Fatal exception
[77569275.551744] Kernel Offset: disabled

Repro steps:

  1. Boot the latest Alpha (949.0.0) with the following Ignition config:
{
    "ignitionVersion": 1,
    "storage": {
        "filesystems": [
            {
                "device": "/dev/disk/by-label/ROOT",
                "format": "btrfs",
                "create": {
                    "force": true,
                    "options": [
                        "--label=ROOT"
                    ]
                }
            }
        ]
    }
}
  1. Run the following command:
    sudo rkt run coreos.com/etcd:v2.2.5
@mischief

This comment has been minimized.

mischief commented Feb 9, 2016

@mischief

This comment has been minimized.

mischief commented Feb 9, 2016

for now you could work around it with rkt run --no-overlay

@yifan-gu

This comment has been minimized.

yifan-gu commented Feb 9, 2016

subscribe

@mischief

This comment has been minimized.

mischief commented Apr 21, 2016

this should be fixed with torvalds/linux@de17e79, and is in the 4.5.2 patch release. coreos/coreos-overlay#1903

@mischief

This comment has been minimized.

mischief commented Apr 21, 2016

confirmed fixed, cannot reproduce. linux 4.5.2 will be in the next alpha.

@mischief

This comment has been minimized.

mischief commented May 18, 2016

should be fixed in CoreOS 1029.0.0+

@mischief mischief closed this May 18, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment