Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic - PANIC at zfs_vfsops.c:583:zfs_space_delta_cb() #6474

Closed
YFLOPS opened this issue Aug 7, 2017 · 8 comments
Closed

Panic - PANIC at zfs_vfsops.c:583:zfs_space_delta_cb() #6474

YFLOPS opened this issue Aug 7, 2017 · 8 comments

Comments

@YFLOPS
Copy link

YFLOPS commented Aug 7, 2017

System information

Type Version/Name
Distribution Name CentOS
Distribution Version 7.3.1611
Linux Kernel 3.10.0-514.26.1.el7.x86_64
Architecture x86_64
ZFS Version 0.7.0-1
SPL Version 0.7.0-1

Describe the problem you're observing

Lustre MDS - running a loop of mdtest will kpanic the MDS

Describe how to reproduce the problem

Install mdtest, run it in a loop on a client node.
-or-
Run mds-survey on MDT

Include any warning/errors/backtraces from the system logs
@YFLOPS YFLOPS changed the title Panic Panic - PANIC at zfs_vfsops.c:583:zfs_space_delta_cb() Aug 7, 2017
@dinatale2
Copy link
Contributor

@YFLOPS are you able to provide the call stacks that you're seeing?

@YFLOPS
Copy link
Author

YFLOPS commented Aug 7, 2017

Yeah. i provided them before and it showed when I submitted the ticket. Not after the refresh. Odd.

[ 6004.896506] VERIFY3(sa.sa_magic == 0x2F505A) failed (24 == 3100762)
[ 6004.903567] PANIC at zfs_vfsops.c:583:zfs_space_delta_cb()
[ 6004.909720] Showing stack for process 130238
[ 6004.909724] CPU: 5 PID: 130238 Comm: mdt01_006 Tainted: P           OE  ------------   3.10.0-514.26.1.el7.x86_64 #1
[ 6004.909725] Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.4.2 01/09/2017
[ 6004.909726]  ffffffffa18ad43b 0000000028956b86 ffff8826264fb420 ffffffff81687233
[ 6004.909729]  ffff8826264fb430 ffffffffa027c234 ffff8826264fb5b8 ffffffffa027c309
[ 6004.909730]  000000030000000e ffff882c00000030 ffff8826264fb5c8 ffff8826264fb568
[ 6004.909732] Call Trace:
[ 6004.909749]  [<ffffffff81687233>] dump_stack+0x19/0x1b
[ 6004.909766]  [<ffffffffa027c234>] spl_dumpstack+0x44/0x50 [spl]
[ 6004.909771]  [<ffffffffa027c309>] spl_panic+0xc9/0x110 [spl]
[ 6004.909776]  [<ffffffff811de800>] ? kmem_cache_open+0x4c0/0x4d0
[ 6004.909780]  [<ffffffffa0278319>] ? spl_kmem_cache_alloc+0x99/0x150 [spl]
[ 6004.909786]  [<ffffffff8168aa82>] ? mutex_lock+0x12/0x2f
[ 6004.909814]  [<ffffffffa174d191>] ? dbuf_find+0x141/0x150 [zfs]
[ 6004.909819]  [<ffffffffa0278319>] ? spl_kmem_cache_alloc+0x99/0x150 [spl]
[ 6004.909821]  [<ffffffff8168aa82>] ? mutex_lock+0x12/0x2f
[ 6004.909828]  [<ffffffff810eacce>] ? getrawmonotonic64+0x2e/0xc0
[ 6004.909849]  [<ffffffffa176c40d>] ? dmu_zfetch+0x40d/0x4a0 [zfs]
[ 6004.909854]  [<ffffffffa027851d>] ? spl_kmem_cache_free+0x14d/0x1d0 [spl]
[ 6004.909882]  [<ffffffffa17f1afc>] zfs_space_delta_cb+0x9c/0x200 [zfs]
[ 6004.909897]  [<ffffffffa175f6ae>] dmu_objset_userquota_get_ids+0x13e/0x3e0 [zfs]
[ 6004.909913]  [<ffffffffa176df2e>] dnode_setdirty+0x3e/0x120 [zfs]
[ 6004.909929]  [<ffffffffa176e2f6>] dnode_allocate+0x186/0x220 [zfs]
[ 6004.909943]  [<ffffffffa175bb1e>] dmu_object_alloc_dnsize+0x29e/0x360 [zfs]
[ 6004.909964]  [<ffffffffa17d2b92>] zap_create_flags_dnsize+0x42/0xc0 [zfs]
[ 6004.909973]  [<ffffffffa087f9d8>] __osd_zap_create+0x88/0x100 [osd_zfs]
[ 6004.909978]  [<ffffffffa087fc27>] osd_mkdir+0x97/0x140 [osd_zfs]
[ 6004.909981]  [<ffffffffa087ed12>] osd_create+0x2b2/0x9d0 [osd_zfs]
[ 6004.909986]  [<ffffffffa02771a0>] ? spl_kmem_zalloc+0xc0/0x170 [spl]
[ 6004.910005]  [<ffffffffa112d3b5>] lod_sub_create+0x1f5/0x480 [lod]
[ 6004.910012]  [<ffffffffa1122419>] lod_create+0x69/0x2c0 [lod]
[ 6004.910025]  [<ffffffffa118e9d5>] mdd_create_object_internal+0xb5/0x280 [mdd]
[ 6004.910030]  [<ffffffffa117a355>] mdd_create_object+0x75/0xb80 [mdd]
[ 6004.910034]  [<ffffffffa11805a8>] ? mdd_declare_create+0x578/0xe20 [mdd]
[ 6004.910038]  [<ffffffffa11843ce>] mdd_create+0xd2e/0x1330 [mdd]
[ 6004.910056]  [<ffffffffa1073d56>] mdt_create+0x846/0xbb0 [mdt]
[ 6004.910087]  [<ffffffffa0c880e4>] ? lprocfs_stats_lock+0x24/0xd0 [obdclass]
[ 6004.910098]  [<ffffffffa0c87d5d>] ? lprocfs_stats_unlock+0x3d/0x50 [obdclass]
[ 6004.910106]  [<ffffffffa107422b>] mdt_reint_create+0x16b/0x350 [mdt]
[ 6004.910114]  [<ffffffffa1075730>] mdt_reint_rec+0x80/0x210 [mdt]
[ 6004.910120]  [<ffffffffa10572fb>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
[ 6004.910127]  [<ffffffffa1062e37>] mdt_reint+0x67/0x140 [mdt]
[ 6004.910185]  [<ffffffffa0ee3915>] tgt_request_handle+0x915/0x1360 [ptlrpc]
[ 6004.910208]  [<ffffffffa0e8d1b3>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc]
[ 6004.910230]  [<ffffffffa0e8a9a8>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
[ 6004.910235]  [<ffffffff810c54f2>] ? default_wake_function+0x12/0x20
[ 6004.910239]  [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90
[ 6004.910260]  [<ffffffffa0e91190>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc]
[ 6004.910280]  [<ffffffffa0e906f0>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
[ 6004.910284]  [<ffffffff810b0a4f>] kthread+0xcf/0xe0
[ 6004.910286]  [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[ 6004.910289]  [<ffffffff81697858>] ret_from_fork+0x58/0x90
[ 6004.910292]  [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140

@ofaaland
Copy link
Contributor

ofaaland commented Aug 8, 2017 •
edited

@YFLOPS , can you provde the Lustre version as well? Thanks. Also, following ZFS properties for your MDT's dataset: xattr and dnodesize. Thank you.

@YFLOPS
Copy link
Author

YFLOPS commented Aug 8, 2017 

# modinfo zfs
filename:       /lib/modules/3.10.0-514.26.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
version:        0.7.0-1


# zpool get all
NAME     PROPERTY                       VALUE                          SOURCE
mdtpool  size                           11.4T                          -
mdtpool  capacity                       0%                             -
mdtpool  altroot                        -                              default
mdtpool  health                         ONLINE                         -
mdtpool  guid                           5491873849748818681            -
mdtpool  version                        -                              default
mdtpool  bootfs                         -                              default
mdtpool  delegation                     on                             default
mdtpool  autoreplace                    on                             local
mdtpool  cachefile                      -                              default
mdtpool  failmode                       wait                           default
mdtpool  listsnapshots                  off                            default
mdtpool  autoexpand                     off                            default
mdtpool  dedupditto                     0                              default
mdtpool  dedupratio                     1.00x                          -
mdtpool  free                           11.4T                          -
mdtpool  allocated                      1.23G                          -
mdtpool  readonly                       off                            -
mdtpool  ashift                         12                             local
mdtpool  comment                        -                              default
mdtpool  expandsize                     -                              -
mdtpool  freeing                        480K                           -
mdtpool  fragmentation                  4%                             -
mdtpool  leaked                         0                              -
mdtpool  multihost                      off                            default
mdtpool  feature@async_destroy          enabled                        local
mdtpool  feature@empty_bpobj            active                         local
mdtpool  feature@lz4_compress           active                         local
mdtpool  feature@multi_vdev_crash_dump  enabled                        local
mdtpool  feature@spacemap_histogram     active                         local
mdtpool  feature@enabled_txg            active                         local
mdtpool  feature@hole_birth             active                         local
mdtpool  feature@extensible_dataset     active                         local
mdtpool  feature@embedded_data          active                         local
mdtpool  feature@bookmarks              enabled                        local
mdtpool  feature@filesystem_limits      enabled                        local
mdtpool  feature@large_blocks           enabled                        local
mdtpool  feature@large_dnode            active                         local
mdtpool  feature@sha512                 enabled                        local
mdtpool  feature@skein                  enabled                        local
mdtpool  feature@edonr                  enabled                        local
mdtpool  feature@userobj_accounting     active                         local


# zfs get all mdtpool/mdt
NAME         PROPERTY              VALUE                  SOURCE
mdtpool/mdt  type                  filesystem             -
mdtpool/mdt  creation              Fri Aug  4  7:06 2017  -
mdtpool/mdt  used                  404M                   -
mdtpool/mdt  available             2.00T                  -
mdtpool/mdt  referenced            404M                   -
mdtpool/mdt  compressratio         1.00x                  -
mdtpool/mdt  mounted               no                     -
mdtpool/mdt  quota                 2T                     local
mdtpool/mdt  reservation           none                   default
mdtpool/mdt  recordsize            128K                   default
mdtpool/mdt  mountpoint            /mdtpool/mdt           default
mdtpool/mdt  sharenfs              off                    default
mdtpool/mdt  checksum              on                     default
mdtpool/mdt  compression           off                    inherited from mdtpool
mdtpool/mdt  atime                 off                    inherited from mdtpool
mdtpool/mdt  devices               on                     default
mdtpool/mdt  exec                  on                     default
mdtpool/mdt  setuid                on                     default
mdtpool/mdt  readonly              off                    default
mdtpool/mdt  zoned                 off                    default
mdtpool/mdt  snapdir               hidden                 default
mdtpool/mdt  aclinherit            restricted             default
mdtpool/mdt  createtxg             285657                 -
mdtpool/mdt  canmount              off                    local
mdtpool/mdt  xattr                 sa                     local
mdtpool/mdt  copies                1                      default
mdtpool/mdt  version               5                      -
mdtpool/mdt  utf8only              off                    -
mdtpool/mdt  normalization         none                   -
mdtpool/mdt  casesensitivity       sensitive              -
mdtpool/mdt  vscan                 off                    default
mdtpool/mdt  nbmand                off                    default
mdtpool/mdt  sharesmb              off                    default
mdtpool/mdt  refquota              none                   default
mdtpool/mdt  refreservation        none                   default
mdtpool/mdt  guid                  16752771120380356715   -
mdtpool/mdt  primarycache          all                    default
mdtpool/mdt  secondarycache        all                    default
mdtpool/mdt  usedbysnapshots       0B                     -
mdtpool/mdt  usedbydataset         404M                   -
mdtpool/mdt  usedbychildren        0B                     -
mdtpool/mdt  usedbyrefreservation  0B                     -
mdtpool/mdt  logbias               latency                default
mdtpool/mdt  dedup                 off                    default
mdtpool/mdt  mlslabel              none                   default
mdtpool/mdt  sync                  standard               default
mdtpool/mdt  dnodesize             auto                   inherited from mdtpool
mdtpool/mdt  refcompressratio      1.00x                  -
mdtpool/mdt  written               404M                   -
mdtpool/mdt  logicalused           106M                   -
mdtpool/mdt  logicalreferenced     106M                   -
mdtpool/mdt  volmode               default                default
mdtpool/mdt  filesystem_limit      none                   default
mdtpool/mdt  snapshot_limit        none                   default
mdtpool/mdt  filesystem_count      none                   default
mdtpool/mdt  snapshot_count        none                   default
mdtpool/mdt  snapdev               hidden                 default
mdtpool/mdt  acltype               off                    default
mdtpool/mdt  context               none                   default
mdtpool/mdt  fscontext             none                   default
mdtpool/mdt  defcontext            none                   default
mdtpool/mdt  rootcontext           none                   default
mdtpool/mdt  relatime              off                    inherited from mdtpool
mdtpool/mdt  redundant_metadata    all                    default
mdtpool/mdt  overlay               off                    default
mdtpool/mdt  lustre:flags          37                     local
mdtpool/mdt  lustre:svname         test-MDT0000           local
mdtpool/mdt  lustre:version        1                      local
mdtpool/mdt  lustre:index          0                      local
mdtpool/mdt  lustre:fsname         test                   local

@YFLOPS
Copy link
Author

YFLOPS commented Aug 8, 2017

Lustre is 2.10.0-1

@dinatale2
Copy link
Contributor

@YFLOPS Thank you for the extra information. I suspect from the call stack and since you have dnodesize=auto set on mdtpool, that you might be hitting a different side effect of #6414.

We recently committed a fix to the 0.7.0-release branch. I suggest you build new spl and zfs packages from the 0.7.0-release branches. Install those and see if the issue clears up.

@behlendorf
Copy link
Contributor

Yes, this is very likely the same issue. There's a fix for this in recently tagged v0.7.1 release.

@jspiros jspiros mentioned this issue Aug 10, 2017
Open
@behlendorf
Copy link
Contributor

Closing as duplicate of #7117.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@behlendorf @ofaaland @YFLOPS @dinatale2