Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter #7484

Conversation

problame
Copy link
Contributor

@problame problame commented Apr 23, 2024

part of #7124

Changes

This PR replaces the EphemeralFile::write_blob-specifc struct Writer
with re-use of owned_buffers_io::write::BufferedWriter.

Further, it restructures the code to cleanly separate

  • the high-level aspect of EphemeralFile's write_blob / read_blk API
  • the page-caching aspect
  • the aspect of IO
    • performing buffered write IO to an underlying VirtualFile
    • serving reads from either the VirtualFile or the buffer if it hasn't been flushed yet
    • the annoying "feature" that reads past the end of the written range are allowed and expected to return zeroed memory, as long as one remains within one PAGE_SZ

Copy link

github-actions bot commented Apr 23, 2024

2796 tests run: 2676 passed, 0 failed, 120 skipped (full report)


Flaky tests (3)

Postgres 15

  • test_download_remote_layers_api: debug

Postgres 14

  • test_crafted_wal_end[last_wal_record_xlog_switch]: debug
  • test_pageserver_restarts_under_worload: debug

Code coverage* (full report)

  • functions: 28.3% (6546 of 23138 functions)
  • lines: 47.0% (46233 of 98432 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
5115aab at 2024-04-26T10:19:19.539Z :recycle:

Copy link
Contributor

@VladLazar VladLazar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

…to problame/larger-buffers-io/refactor-size-tracking-writer
…r' into problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io
…or-ephemeral-file-reuse-buffered-io' into problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io--with-write-path-prewarming
…to problame/larger-buffers-io/refactor-size-tracking-writer
…r' into problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io
…e-buffered-io' into problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io--with-write-path-prewarming
…/refactor-ephemeral-file-reuse-buffered-io--with-write-path-prewarming

refactor(ephemeral_file): bring back pre-warming of PS page cache on write
@problame problame changed the title refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter & drop cache-on-write refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter Apr 25, 2024
Base automatically changed from problame/larger-buffers-io/refactor-size-tracking-writer to main April 26, 2024 09:19
…to problame/larger-buffers-io/refactor-size-tracking-writer
…r' into problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io
@problame problame merged commit dbb0c96 into main Apr 26, 2024
53 checks passed
@problame problame deleted the problame/larger-buffers-io/refactor-ephemeral-file-reuse-buffered-io branch April 26, 2024 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants