RFC: ZIL refactoring, ZIL Kinds, and ZIL-PMEM #12731

problame · 2021-11-07T20:24:26Z

This PR is a large patch series, roughly ordered as followed:

a bunch of preliminary refactorings of the ZIL and related code
ZIL Kinds,
and ZIL-PMEM.

All of this is based on my master's thesis.

I'll give a presentation on all of this at the OpenZFS 2021 Developer Summit this Monday.

Will update the PR with more details afterward.

…ath for '-t installed'

Signed-off-by: Christian Schwarz <me@cschwarz.com>

Was leaking memory, detected by ASAN. Signed-off-by: Christian Schwarz <me@cschwarz.com>

This allows more subsystem's *_init() functions to use zfs_dbgmsg and/or dprintf. Signed-off-by: Christian Schwarz <me@cschwarz.com>

Signed-off-by: Christian Schwarz <me@cschwarz.com>

…m in zil_write_issue Signed-off-by: Christian Schwarz <me@cschwarz.com>

… zil_impl.h

Signed-off-by: Christian Schwarz <me@cschwarz.com>

…t request Signed-off-by: Christian Schwarz <me@cschwarz.com>

We are careful to not change anything, particularly not - the function signatures - the function bodies - the relative order of the LWB-specific functions This makes the diff easy to review: 1. Open the delete-heavy diff for zil.c, and the newly added zil_lwb.c side-by-side. 2. Scroll to the top for both the zil.c diff and zil_lwb.c, then go through the changes to zil.c top to bottom, performing the following check for each function F deleted from zil.c: ``` Search for function's code in zil_lwb.c. It should - be present verbatim (not even whitespace changes) - in the same order relative to the other functions deleted from zil.c ``` 3. Ensure that no extraneous code was added to zil_lwb.c, i.e., we reached the end of zil_lwb.c in step (2).

…lwb.h

…lic zil_* funcs Later, we'll convert the stubs to do dynamic dispatch via a vtable (zilog_vtable_t).

…in zil.c

…og_t In a later commit, we'll introduce the generic struct zilog and update the typedef again.

… be expected by ZIL Kinds

…achine that will be expected by ZIL Kinds

…rt handler function This can be used to implement tests that ensure that certain conditions are covered by an assertion.

WR_INDIRECT is full of subtle opportunities to leak log blocks. Make it optional for a ZIL implementation to support it.

These casts are currently correct because fletcher_4_ctx_t is a union type with zio_cksum_t as the fletcher_4_ctx_t::scaler member. However, they a) make the code unnecessarily hard to follow and b) are brittle with regards to changes to the fletcher_4_ctx_t type (See next patch in this series) Signed-off-by: Christian Schwarz <me@cschwarz.com>

zfs_kfpu_ctx_t enables nested kfpu usage by calling kfpu_begin() on first zfs_kfpu_enter() and kfpu_end() on last kfpu_exit(). Signed-off-by: Christian Schwarz <me@cschwarz.com>

This change adds fletcher_4_native_kfpu_ctx() which allows using fletcher_4_native() in an existing zfs_kfpu_ctx_t. Signed-off-by: Christian Schwarz <me@cschwarz.com>

With the avx512 pmem ops, this cuts down latency because we only need to XSAVE once per entry instead of thrice for header, body bulk, body tail.

…u_counter_stat)

…tart time

liyimeng · 2022-02-08T20:50:53Z

@problame This indeed a great work. But is is a huge PR, do you see any chance for it to get merged in near future?

problame · 2022-02-09T18:55:13Z

I think some of the preliminary ones could be merged independent of the ZIL-PMEM feature.
I think some ZIL redesign work is inevitable in the next few years if we want to keep up with storage performance.
Whether it's going to be the design fromn
my thesis, a variation thereof, or something entirely different is an open question.
I think we didn't have that discussion yet, in the larger community.

liyimeng · 2022-02-10T07:50:01Z

I love zfs because if keep data safe. I am side with you on the performance. Giving the storage technology keep evolving in recent years, especially nvme and pmem, make zfs performance metric look so out-dated. The community need contributions like your one to keep up. Otherwise, it soon become a passed thing :(

Maybe you can break down your PR in small ones, make it get merged step by step? :) Since no one can review such a big change. I don't have competence in doing the work, but I'm happy to help with testing.

problame · 2022-02-13T09:05:18Z

For future reference, all patches up to and including patch make zilog_t* opaque for dsl_{pool,scan}.h are mere refactorings that are a net improvement to the code base in my opinion, even without ZIL Kinds and ZIL PMEM.

unkownerror1 · 2022-11-12T03:28:32Z

@problame Does this PR improve the performance of synchronous writes, for non-PMEM

jumbi77 · 2022-11-12T07:38:53Z

For future reference, all patches up to and including patch make zilog_t* opaque for dsl_{pool,scan}.h are mere refactorings that are a net improvement to the code base in my opinion, even without ZIL Kinds and ZIL PMEM.

That sounds already nice. Is it possible to split out the refactoring/improvement from ZIL Kinds/ZIL PMEM in separate PR? Maybe its easier to review and getting a higher chance to be upstreamed?

Thanks anyway for that impressive work.

problame added 30 commits November 7, 2021 18:51

DO NOT UPSTREAM FIX PROPERLY INSTEAD: scripts/zimport.sh hard-coded p…

705db79

…ath for '-t installed'

DO NOT UPSTREAM devvm script

1bbf7b5

dsl_scan: make zil_scan_arg_t::zsa_zh a const pointer

d6005ac

Signed-off-by: Christian Schwarz <me@cschwarz.com>

fixup "ztest: propagate -o to the zdb child process"

54f74f2

Was leaking memory, detected by ASAN. Signed-off-by: Christian Schwarz <me@cschwarz.com>

vdev_disk_ops: comment to remind about vdev_file_ops for libspl builds

5d5ef1f

kmod init: move zfs_dbgmsg_init to an earlier point in the procedure

f4afa49

This allows more subsystem's *_init() functions to use zfs_dbgmsg and/or dprintf. Signed-off-by: Christian Schwarz <me@cschwarz.com>

linux: comment error code convention during module initialization

6d519e8

zil: comment on P2ROUNDUP(lr_length) for WR_NEED_COPY records

2ae6f86

zil: zil_alloc_lwb: make bp parameter const

36344cc

Signed-off-by: Christian Schwarz <me@cschwarz.com>

zio_rewrite: make bookmark parameter const

ffd2f59

Signed-off-by: Christian Schwarz <me@cschwarz.com>

set -Werror={incompatible-pointer-types,implicit-function-declaration}

732d95d

Signed-off-by: Christian Schwarz <me@cschwarz.com>

zil: comment on how we set the lwb's zil_chain_t::zc_next_blk checksu…

f80525b

…m in zil_write_issue Signed-off-by: Christian Schwarz <me@cschwarz.com>

move zill calls from zvol.c to zvol_log.c, rendering zvol.c free from…

18fd4e4

… zil_impl.h

document objset_t::os_zil_header

63d5e76

zfs_get_data: separate helper functions for WR_{INDIRECT,NEED_COPY}

08ec9c1

zvol_get_data: separate helper functions for WR_{INDIRECT,NEED_COPY}

75e4992

[1/5] zil: factor out physical ZIL traversal into zil_parse_phys

241e36a

Signed-off-by: Christian Schwarz <me@cschwarz.com>

[2/5] dmu_traverse: use zil_parse_phys & only traverse ZIL on explici…

09d8559

…t request Signed-off-by: Christian Schwarz <me@cschwarz.com>

[3/5] dsl_scan: adopt zil_parse_phys API

8c5d348

[4/5] zdb: switch to using zil_parse_phys

0d5b9b9

[5/5] zil: make zil_parse private to zil.c

e38d9fa

make zilog_t* opaque for dsl_{pool,scan}.h

a190087

[ZIL Kinds] move comment on on-disk format into zil_lwb.c

8a2cd3b

[ZIL Kinds] move lwb-specific types, constants, and zilog_t into zil_…

cf0f5da

…lwb.h

[ZIL Kinds] rename zil_* to zillwb_* and leave stubs in place for pub…

4a81ed2

…lic zil_* funcs Later, we'll convert the stubs to do dynamic dispatch via a vtable (zilog_vtable_t).

[ZIL Kinds] rename zil_parse_phys to zillwb_parse_phys & remove stub …

901893f

…in zil.c

[ZIL Kinds] rename zil_chain_t => zillwb_chain_t

02a542f

[ZIL Kinds] rename struct zilog{,_lwb} & typedef struct zilog_lwb zil…

cd172b5

…og_t In a later commit, we'll introduce the generic struct zilog and update the typedef again.

make header consumable by bindgen

e90bcbc

problame added 22 commits November 7, 2021 21:12

[ZIL Kinds] rename LWB-specific kstats

aa8b684

[ZIL Kinds] rename kmem_cache_t instances

7665ff5

[ZIL Kinds] rename zil_block_buckets

fd6be20

[ZIL Kinds] zvol: adjust ZIL usage to fit the state machine that will…

2261936

… be expected by ZIL Kinds

[ZIL Kinds] zfs_create: always call zil_replaying, to fit the state m…

bb6c01c

…achine that will be expected by ZIL Kinds

[ZIL Kinds] implement ZIL kinds

56a40fc

[ZIL-PMEM][independent] spl-kmem: function for allocating aligned memory

b7939ab

[ZIL-PMEM][independent] spl: semaphore and spinlock wrappers

30c0787

[ZIL-PMEM][independent] libspl: add ability to configure a custom abo…

b354bf6

…rt handler function This can be used to implement tests that ensure that certain conditions are covered by an assertion.

[ZIL-PMEM] zil: make it optional for ZIL kinds to support WR_INDIRECT

7a865c2

WR_INDIRECT is full of subtle opportunities to leak log blocks. Make it optional for a ZIL implementation to support it.

[ZIL-PMEM] initial implementation of ZIL-PMEM

15ea926

[2/3] introduce zfs_kfpu_ctx_t, zfs_kfpu_{enter,exit} APIs

64b6830

zfs_kfpu_ctx_t enables nested kfpu usage by calling kfpu_begin() on first zfs_kfpu_enter() and kfpu_end() on last kfpu_exit(). Signed-off-by: Christian Schwarz <me@cschwarz.com>

[3/3] fletcher: add fletcher_4_native_kfpu_ctx

c51b42b

This change adds fletcher_4_native_kfpu_ctx() which allows using fletcher_4_native() in an existing zfs_kfpu_ctx_t. Signed-off-by: Christian Schwarz <me@cschwarz.com>

[ZIL-PMEM] use zfs_kfpu_ctx_t API to reduce number of XSAVEs

093d251

With the avx512 pmem ops, this cuts down latency because we only need to XSAVE once per entry instead of thrice for header, body bulk, body tail.

[1/3] zfs_log: refactor zfs_log_write

f6d3271

[2/3] dmu: add dmu_write_uioandcc_dbuf() and dmu_write_uioandcc_dnode()

7ee25db

[3/3] zfs_log / zvol_log: avoid dmu_read in {zfs,zvol}_log_write

ba722b2

[ZIL-PMEM][independent] add zfs_percpu_counter_stat (GPL only)

7571a93

[ZIL-PMEM] add static stats for the thesis evaluation (uses zfs_percp…

7e27706

…u_counter_stat)

[ZIL-PMEM] implement ITXG bypass mode for ZVOLs

4eea7cf

[ZIL-PMEM] dtrace probe / tracepoint to track delta of task birth & s…

9460f7d

…tart time

problame mentioned this pull request Nov 10, 2021

Intel Optane Memory log device compatibility #6080

Closed

ahrens assigned mmaybee Nov 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: ZIL refactoring, ZIL Kinds, and ZIL-PMEM #12731

RFC: ZIL refactoring, ZIL Kinds, and ZIL-PMEM #12731

problame commented Nov 7, 2021

liyimeng commented Feb 8, 2022 •

edited

problame commented Feb 9, 2022 •

edited

liyimeng commented Feb 10, 2022

problame commented Feb 13, 2022

unkownerror1 commented Nov 12, 2022

jumbi77 commented Nov 12, 2022

RFC: ZIL refactoring, ZIL Kinds, and ZIL-PMEM #12731

Are you sure you want to change the base?

RFC: ZIL refactoring, ZIL Kinds, and ZIL-PMEM #12731

Conversation

problame commented Nov 7, 2021

liyimeng commented Feb 8, 2022 • edited

problame commented Feb 9, 2022 • edited

liyimeng commented Feb 10, 2022

problame commented Feb 13, 2022

unkownerror1 commented Nov 12, 2022

jumbi77 commented Nov 12, 2022

liyimeng commented Feb 8, 2022 •

edited

problame commented Feb 9, 2022 •

edited