7968 multi-threaded spa_sync #138
Conversation
| sync_dnodes_task, sda, 0); | ||
| /* callback frees sda */ | ||
| } | ||
| taskq_wait(dmu_objset_pool(os)->dp_sync_taskq); | ||
|
|
||
| list = &DMU_META_DNODE(os)->dn_dirty_records[txgoff]; | ||
| while (dr = list_head(list)) { |
There was a problem hiding this comment.
could you update it to:
while ((dr = list_head(list)) != NULL)
for remove paretnes gcc warning
|
../../common/fs/zfs/dmu_objset.c: In function 'dmu_objset_sync_dnodes': |
|
hello, ahrens, is this effective for one zvol under multi-thread writing? thanks. |
|
@wangdbang This change parallelizes syncing of different objects (dnodes). Each zvol only has one (modified) object, so there should be no performance change when writing one zvol. |
|
@ahrens Thanks. |
|
@zettabot go |
1 similar comment
|
@zettabot go |
|
@zettabot go |
|
@zettabot go |
|
Also reviewed by @behlendorf, @pzakha, and Brad Lewis |
|
@zettabot go |
| @@ -121,11 +121,14 @@ struct objset { | |||
|
|
|||
| /* Protected by os_lock */ | |||
| kmutex_t os_lock; | |||
| list_t os_dirty_dnodes[TXG_SIZE]; | |||
| list_t os_free_dnodes[TXG_SIZE]; | |||
| struct objset *os_origin_mooch_objset; | |||
There was a problem hiding this comment.
Can't seem to find the meaning of this. Maybe a leftover from earlier revisions?
There was a problem hiding this comment.
@skiselkov, Thanks for taking a look. Yeah this was due to mismergeing, I thought I fixed it yesterday but turns out it wasn't included in my push. Will fix!
There was a problem hiding this comment.
Ok, other than this minor nit, as a whole LGTM.
|
@zettabot go |
|
@zettabot go |
|
@zettabot go |
|
I thought I did a commit for the missing multilist_destroy in arc.c, but since they are still missing in IllumOS maybe I was dreaming :) |
spa_sync() iterates over all the dirty dnodes and processes each of them by calling dnode_sync(). If there are many dirty dnodes (e.g. because we created or removed a lot of files), the single thread of spa_sync() calling dnode_sync() can become a bottleneck. Additionally, if many dnodes are dirtied concurrently in open context (e.g. due to concurrent file creation), the os_lock will experience lock contention via dnode_setdirty().
The solution is to track dirty dnodes on a multilist_t, and for spa_sync() to use separate threads to process each of the sublists in the multilist.
On the concurrent file creation microbenchmark, the performance improvement from dnode_setdirty() is up to 7%. Additionally, the wall clock time spent in spa_sync() is reduced to 15%-40% of the single-threaded case. In terms of cost/reward, once the other bottlenecks are addressed, fixing this bug will provide a medium-large performance gain and require a medium amount of effort to implement.
Sponsored by Intel Corp.
Corresponds to ZoL bug openzfs/zfs#4807