-
Notifications
You must be signed in to change notification settings - Fork 419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tokio-epoll-uring: use it on the layer-creating code paths #6378
Commits on Feb 5, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a3c88a8 - Browse repository at this point
Copy the full SHA a3c88a8View commit details -
Configuration menu - View commit details
-
Copy full SHA for c2d8251 - Browse repository at this point
Copy the full SHA c2d8251View commit details -
Configuration menu - View commit details
-
Copy full SHA for a0032f4 - Browse repository at this point
Copy the full SHA a0032f4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0548374 - Browse repository at this point
Copy the full SHA 0548374View commit details -
Configuration menu - View commit details
-
Copy full SHA for e824c84 - Browse repository at this point
Copy the full SHA e824c84View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29b9628 - Browse repository at this point
Copy the full SHA 29b9628View commit details -
Configuration menu - View commit details
-
Copy full SHA for c2e9cf4 - Browse repository at this point
Copy the full SHA c2e9cf4View commit details
Commits on Feb 6, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 4fa0181 - Browse repository at this point
Copy the full SHA 4fa0181View commit details
Commits on Feb 7, 2024
-
Configuration menu - View commit details
-
Copy full SHA for aaccb7a - Browse repository at this point
Copy the full SHA aaccb7aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a4880e - Browse repository at this point
Copy the full SHA 9a4880eView commit details -
refactor(disk_btree): make BlockWriter::write_blk infallible
We can probably get rid of the entire trait, but, that's for another time.
Configuration menu - View commit details
-
Copy full SHA for b5a00b0 - Browse repository at this point
Copy the full SHA b5a00b0View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/btree-b…
…uilder-refactor' into problame/integrate-tokio-epoll-uring/write-path/simplify-write-at-api
Configuration menu - View commit details
-
Copy full SHA for 7ba1949 - Browse repository at this point
Copy the full SHA 7ba1949View commit details -
Configuration menu - View commit details
-
Copy full SHA for 91d3e25 - Browse repository at this point
Copy the full SHA 91d3e25View commit details -
Revert "refactor(disk_btree): make BlockWriter::write_blk infallible"
This reverts commit b5a00b0.
Configuration menu - View commit details
-
Copy full SHA for 6f65648 - Browse repository at this point
Copy the full SHA 6f65648View commit details -
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/refactor-blob-writer
Configuration menu - View commit details
-
Copy full SHA for 85d5fc6 - Browse repository at this point
Copy the full SHA 85d5fc6View commit details -
Configuration menu - View commit details
-
Copy full SHA for edabce6 - Browse repository at this point
Copy the full SHA edabce6View commit details -
Configuration menu - View commit details
-
Copy full SHA for e5d15df - Browse repository at this point
Copy the full SHA e5d15dfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a39457 - Browse repository at this point
Copy the full SHA 2a39457View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4659794 - Browse repository at this point
Copy the full SHA 4659794View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14bdd84 - Browse repository at this point
Copy the full SHA 14bdd84View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4fe3b49 - Browse repository at this point
Copy the full SHA 4fe3b49View commit details -
Configuration menu - View commit details
-
Copy full SHA for a6605a1 - Browse repository at this point
Copy the full SHA a6605a1View commit details -
Configuration menu - View commit details
-
Copy full SHA for c92e8a7 - Browse repository at this point
Copy the full SHA c92e8a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0144e2 - Browse repository at this point
Copy the full SHA b0144e2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54561a8 - Browse repository at this point
Copy the full SHA 54561a8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6c47083 - Browse repository at this point
Copy the full SHA 6c47083View commit details -
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/refactor-blob-writer
Configuration menu - View commit details
-
Copy full SHA for 238296a - Browse repository at this point
Copy the full SHA 238296aView commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/refacto…
…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/refactor-virtual-file
Configuration menu - View commit details
-
Copy full SHA for 207764b - Browse repository at this point
Copy the full SHA 207764bView commit details -
Configuration menu - View commit details
-
Copy full SHA for e5da261 - Browse repository at this point
Copy the full SHA e5da261View commit details -
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/refactor-blob-writer
Configuration menu - View commit details
-
Copy full SHA for 26e51c7 - Browse repository at this point
Copy the full SHA 26e51c7View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/refacto…
…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all
Configuration menu - View commit details
-
Copy full SHA for 6f4d182 - Browse repository at this point
Copy the full SHA 6f4d182View commit details -
Merge remote-tracking branch 'origin/main' into problame/integrate-to…
…kio-epoll-uring/write-path/refactor-blob-writer
Configuration menu - View commit details
-
Copy full SHA for 33f3053 - Browse repository at this point
Copy the full SHA 33f3053View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/refacto…
…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all
Configuration menu - View commit details
-
Copy full SHA for 33261e4 - Browse repository at this point
Copy the full SHA 33261e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 720f633 - Browse repository at this point
Copy the full SHA 720f633View commit details
Commits on Feb 8, 2024
-
Merge remote-tracking branch 'origin/main' into problame/integrate-to…
…kio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for 368f2ac - Browse repository at this point
Copy the full SHA 368f2acView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d8f5d1 - Browse repository at this point
Copy the full SHA 3d8f5d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 183c86c - Browse repository at this point
Copy the full SHA 183c86cView commit details
Commits on Feb 12, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a6d022a - Browse repository at this point
Copy the full SHA a6d022aView commit details -
implement write_at IoEngine function & wire it up; only discovered no…
…w that IoBuf needs to be Send; PITA but feasible
Configuration menu - View commit details
-
Copy full SHA for dff1b9b - Browse repository at this point
Copy the full SHA dff1b9bView commit details -
Merge remote-tracking branch 'origin/problame/integrate-tokio-epoll-u…
…ring/write-path/virtual-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for 64799e6 - Browse repository at this point
Copy the full SHA 64799e6View commit details -
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/virtual-file-write-all
Configuration menu - View commit details
-
Copy full SHA for 8998178 - Browse repository at this point
Copy the full SHA 8998178View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/virtual…
…-file-write-all' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all-at
Configuration menu - View commit details
-
Copy full SHA for 98fe109 - Browse repository at this point
Copy the full SHA 98fe109View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/virtual…
…-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for bfdd86c - Browse repository at this point
Copy the full SHA bfdd86cView commit details
Commits on Feb 13, 2024
-
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/virtual-file-write-all-at
Configuration menu - View commit details
-
Copy full SHA for 87f05ca - Browse repository at this point
Copy the full SHA 87f05caView commit details
Commits on Feb 14, 2024
-
Merge branch 'main' into problame/integrate-tokio-epoll-uring/write-p…
…ath/virtual-file-write-all-at
Configuration menu - View commit details
-
Copy full SHA for ebffde1 - Browse repository at this point
Copy the full SHA ebffde1View commit details -
Configuration menu - View commit details
-
Copy full SHA for ab51748 - Browse repository at this point
Copy the full SHA ab51748View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a55ac7 - Browse repository at this point
Copy the full SHA 6a55ac7View commit details
Commits on Mar 1, 2024
-
Merge remote-tracking branch 'origin/main' into problame/integrate-to…
…kio-epoll-uring/write-path/virtual-file-write-all-at
Configuration menu - View commit details
-
Copy full SHA for 0316adc - Browse repository at this point
Copy the full SHA 0316adcView commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/write-path/virtual…
…-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for 0c50ee9 - Browse repository at this point
Copy the full SHA 0c50ee9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c30090 - Browse repository at this point
Copy the full SHA 2c30090View commit details -
layer file creation: remove redundant fsync()s
The `writer.finish()` methods already fsync the inode, using `VirtualFile::sync_all()`. All that the callers need to do is fsync their directory, i.e., the timeline directory. Note that there's a call in the new compaction code that is apparently dead-at-runtime, so, I couldn't fix up any fsyncs there [Link](https://github.com/neondatabase/neon/blob/502b69b33bbd4ad1b0647e921a9c665249a2cd62/pageserver/src/tenant/timeline/compaction.rs#L204-L211). In the grand scheme of things, layer durability probably doesn't matter anymore because the remote storage is authoritative at all times as of #5198. But, let's not break the discipline in htis commit. part of #6663
Configuration menu - View commit details
-
Copy full SHA for 32358cf - Browse repository at this point
Copy the full SHA 32358cfView commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/layer-write-path-f…
…sync-cleanups' into problame/integrate-tokio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for 4d8b9e3 - Browse repository at this point
Copy the full SHA 4d8b9e3View commit details -
layer file creation: fsync timeline directories using VirtualFile::sy…
…nc_all() Except for the involvement of the VirtualFile fd cache, this is equivalent to what happened before at runtime. Future PR #6378 will implement `VirtualFile::sync_all()` using tokio-epoll-uring if that's configured as the io engine. This PR is preliminary work for that.
Configuration menu - View commit details
-
Copy full SHA for ebf2de5 - Browse repository at this point
Copy the full SHA ebf2de5View commit details -
layer file creation: fatal_err the timeline dir fsync
As pointed out in the comments added in this PR: the in-memory state of the filesystem already has the layer file in its final place. If the fsync fails, but pageserver continues to execute, it's quite easy for subsequent pageserver code to observe the file being there and assume it's durable, where it really isn't. It can happen that we get ENOSPC during the fsync. However, 1. the timeline dir is small (remember, the big layer _file_ has already been synced). Small data means ENOSPC due to delayed allocation races etc are less likely. 2. what elase are we going to do in that case? If we decide to bubble up the error, the file remains on disk. We could try to unlink it and fsync after the unlink. If that fails, we would _definitely_ need to error out. Is it worth the trouble though? Side note: all this logic about not carrying on after fsync failure implies that we `sync` the filesystem successfully before we restart the pageserver. Our systemd unit currently does not do that, but should.
Configuration menu - View commit details
-
Copy full SHA for ca2ed0b - Browse repository at this point
Copy the full SHA ca2ed0bView commit details -
Merge remote-tracking branch 'origin/main' into problame/integrate-to…
…kio-epoll-uring/layer-write-path-fsync-cleanups
Configuration menu - View commit details
-
Copy full SHA for ce251ac - Browse repository at this point
Copy the full SHA ce251acView commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/layer-write-path-f…
…sync-cleanups' into problame/integrate-tokio-epoll-uring/create-layer-fatal-err-on-fsync
Configuration menu - View commit details
-
Copy full SHA for c4f7a19 - Browse repository at this point
Copy the full SHA c4f7a19View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1fe80d7 - Browse repository at this point
Copy the full SHA 1fe80d7View commit details -
Merge remote-tracking branch 'origin/main' into problame/integrate-to…
…kio-epoll-uring/layer-write-path-fsync-cleanups
Configuration menu - View commit details
-
Copy full SHA for 5528b16 - Browse repository at this point
Copy the full SHA 5528b16View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/layer-write-path-f…
…sync-cleanups' into problame/integrate-tokio-epoll-uring/create-layer-fatal-err-on-fsync
Configuration menu - View commit details
-
Copy full SHA for c972d17 - Browse repository at this point
Copy the full SHA c972d17View commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/create-layer-fatal…
…-err-on-fsync' into problame/integrate-tokio-epoll-uring/ioengine-par-fsync
Configuration menu - View commit details
-
Copy full SHA for 7299a0a - Browse repository at this point
Copy the full SHA 7299a0aView commit details -
Merge branch 'problame/integrate-tokio-epoll-uring/ioengine-par-fsync…
…' into problame/integrate-tokio-epoll-uring/more-ops
Configuration menu - View commit details
-
Copy full SHA for 348649d - Browse repository at this point
Copy the full SHA 348649dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0b268fc - Browse repository at this point
Copy the full SHA 0b268fcView commit details
Commits on Mar 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 00be7e1 - Browse repository at this point
Copy the full SHA 00be7e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for ffbe466 - Browse repository at this point
Copy the full SHA ffbe466View commit details -
Revert "[DO NOT MERGE} CI: test with both io engines"
This reverts commit 0548374.
Configuration menu - View commit details
-
Copy full SHA for f471a42 - Browse repository at this point
Copy the full SHA f471a42View commit details -
Configuration menu - View commit details
-
Copy full SHA for bdd5228 - Browse repository at this point
Copy the full SHA bdd5228View commit details