CIFS kernel panic

[chuckleb]

Issue Report

Bug

In 1800.4.0 (and 1828.1.0), accessing a CIFS mount will cause a kernel panic. This was working properly in 1745.7.0. First noticed in stable and tried alpha channel, same problem persists. Fresh install of 1745 works properly.

Container Linux Version

Both 1800.4.0 and 1828.1.0

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1828.1.0
VERSION_ID=1828.1.0
BUILD_ID=2018-07-24-2257
PRETTY_NAME="Container Linux by CoreOS 1828.1.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Windows 2016 Standard Edition - Hyper-V
Version 1607 (OS Build 14393.339)
4GB RAM w/dynamic memory, 8 cores, external network

SMB shares are exported from the Windows Server. These are NTFS- and ReFS-backed shares in a Storage Spaces mirrored environment.

localhost ~ # mount.cifs -V
mount.cifs version: 6.8

localhost ~ # mount | grep cifs

//192.168.10.22/S/ on /mnt/data type cifs (rw,relatime,vers=default,cache=strict,username=docker-data,domain=,uid=500,forceuid,gid=500,forcegid,addr=192.168.10.22,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,nobrl,mfsymlinks,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

fstab

//192.168.10.22/S/	/mnt/data	cifs	mfsymlinks,nolock,username=<username>,password=<password>,uid=500,gid=500,iocharset=utf8,dir_mode=0777,file_mode=0777 0 0

Expected Behavior

Normal access to CIFS shares for reading/writing.

Actual Behavior

Running docker-compose up to start containers results in immediate kernel panic. Writing a large file causes a kernel panic as well.

Reproduction Steps

1. Mount SMB file share and cd into it.
2. dd if=/dev/zero of=testfile bs=1M count=10
   10+0 records in
   10+0 records out
   10485760 bytes (10 MB, 10 MiB) copied, 0.080528 s, 130 MB/s
3. dd if=/dev/zero of=testfile bs=1M count=1000
   kernel panic

Other Information

From line 14990 of attached journalctl

Jul 26 16:51:39 docker kernel: cache_from_obj: Wrong slab cache. cifs_request but object is from xfrm_dst_cache
Jul 26 16:51:39 docker kernel: cache_from_obj: Wrong slab cache. cifs_request but object is from xfrm_dst_cache
Jul 26 16:51:39 docker kernel: ------------[ cut here ]------------
Jul 26 16:51:39 docker kernel: WARNING: CPU: 7 PID: 3074 at ../source/mm/slab.h:377 kmem_cache_free+0x13a/0x1d0
Jul 26 16:51:39 docker kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay nfsv3 nfs_acl nfs cmac lockd grace arc4 ecb sunrpc md4 nls_utf8 cifs ccm fscache nls_ascii nls_cp437 vfat fat hv_utils i2c_piix4 cn psmouse ptp sb_edac evdev i2c_core edac_core pps_core hv_balloon hyperv_fb mousedev button sch_fq_codel ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_verity dm_bufio sr_mod cdrom sd_mod hid_generic crc32c_intel hid_hyperv ata_piix hv_storvsc aesni_intel hv_netvsc hid hyperv_keyboard libata scsi_transport_fc aes_x86_64 crypto_simd cryptd glue_helper scsi_mod hv_vmbus dm_mirror
Jul 26 16:51:39 docker kernel:  dm_region_hash dm_log dm_mod dax
Jul 26 16:51:39 docker kernel: CPU: 7 PID: 3074 Comm: chown Not tainted 4.14.55-coreos #1
Jul 26 16:51:39 docker kernel: Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  04/28/2016
Jul 26 16:51:39 docker kernel: task: ffff92dced201e00 task.stack: ffffb19b47684000
Jul 26 16:51:39 docker kernel: RIP: 0010:kmem_cache_free+0x13a/0x1d0
Jul 26 16:51:39 docker kernel: RSP: 0018:ffffb19b47687b30 EFLAGS: 00010286
Jul 26 16:51:39 docker kernel: RAX: 0000000000000050 RBX: ffff92dceea3a300 RCX: 000000223eaecb49
Jul 26 16:51:39 docker kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000246
Jul 26 16:51:39 docker kernel: RBP: ffff92da1c3c8780 R08: 0000000000000000 R09: 0000000000000050
Jul 26 16:51:39 docker kernel: R10: fffff52d8fbd58c0 R11: 0000000000000000 R12: ffff92dcee555800
Jul 26 16:51:39 docker kernel: R13: ffffb19b47687c10 R14: ffff92dceea3a300 R15: ffffb19b47687c1c
Jul 26 16:51:39 docker kernel: FS:  00007f7ea07b3700(0000) GS:ffff92dcf67c0000(0000) knlGS:0000000000000000
Jul 26 16:51:39 docker kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 16:51:39 docker kernel: CR2: 0000000000daa0c8 CR3: 00000003bb75a004 CR4: 00000000001606e0
Jul 26 16:51:39 docker kernel: Call Trace:
Jul 26 16:51:39 docker kernel:  cifs_symlink+0x561/0x570 [cifs]
Jul 26 16:51:39 docker kernel:  SMB2_logoff+0x5bd/0x840 [cifs]
Jul 26 16:51:39 docker kernel:  SMB2_set_info+0x54/0x80 [cifs]
Jul 26 16:51:39 docker kernel:  SMB2_lease_break+0x32d/0x3d0 [cifs]
Jul 26 16:51:39 docker kernel:  smb2_set_file_info+0x62/0xa0 [cifs]
Jul 26 16:51:39 docker kernel:  cifs_set_file_info+0xb0/0x1b0 [cifs]
Jul 26 16:51:39 docker kernel:  cifs_setattr+0x15f/0xc80 [cifs]
Jul 26 16:51:39 docker kernel:  notify_change+0x2e8/0x430
Jul 26 16:51:39 docker kernel:  chown_common.isra.13+0xec/0x1a0
Jul 26 16:51:39 docker kernel:  SyS_fchownat+0xdc/0x100
Jul 26 16:51:39 docker kernel:  do_syscall_64+0x67/0x120
Jul 26 16:51:39 docker kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Jul 26 16:51:39 docker kernel: RIP: 0033:0x7f7ea02c139a
Jul 26 16:51:39 docker kernel: RSP: 002b:00007fff1cc3b668 EFLAGS: 00000202 ORIG_RAX: 0000000000000104
Jul 26 16:51:39 docker kernel: RAX: ffffffffffffffda RBX: 00007fff1cc3b890 RCX: 00007f7ea02c139a
Jul 26 16:51:39 docker kernel: RDX: 0000000000000000 RSI: 0000000000da8c60 RDI: 00000000ffffff9c
Jul 26 16:51:39 docker kernel: RBP: 0000000000da9e90 R08: 0000000000000000 R09: 00000000ffffffff
Jul 26 16:51:39 docker kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00000000ffffffff
Jul 26 16:51:39 docker kernel: R13: 0000000000da8bd0 R14: 0000000000000000 R15: 0000000000da9f08
Jul 26 16:51:39 docker kernel: Code: c5 0f 84 08 ff ff ff 48 3b a8 d8 00 00 00 74 6a 48 8b 48 60 48 8b 55 60 48 c7 c6 60 02 c3 84 48 c7 c7 38 55 fa 84 e8 e1 4b ed ff <0f> 0b e9 dd fe ff ff 65 8b 05 90 8c e1 7b 89 c0 48 0f a3 05 1e 
Jul 26 16:51:39 docker kernel: ---[ end trace 2b30dea888642a2a ]---
Jul 26 16:51:39 docker kernel: cache_from_obj: Wrong slab cache. cifs_request but object is from xfrm_dst_cache
Jul 26 16:51:39 docker kernel: cache_from_obj: Wrong slab cache. cifs_request but object is from xfrm_dst_cache

coreos.log

[bgilbert]

Thanks for the report. It seems this was introduced by coreos/linux@748144f in 4.14.55. A revert of that commit is already queued for 4.14.59.

1828.1.0 is on the beta channel, which also ships the 4.14 kernel. The alpha channel ships 4.17 and should not have this problem. Could you test the latest alpha and confirm?

[chuckleb]

Yes, the alpha release fixes this problem. I was able to test this by creating very large files using dd as well as start and run docker containers with docker-compose.

Thank you!

[bgilbert]

This should be fixed in beta 1828.2.0 and stable 1800.5.0, due shortly. Thanks for reporting.