`test-images` disks are too slow for pageserver `virtual_file_io_mode=direct-rw`

[problame]

          With the default `direct-rw`, the test-images v16 x86_64 GH action hits the 20min.

Originally posted by @problame in #11868 (comment)

[bayandin]

Observation from https://github.com/neondatabase/neon/actions/runs/14936955880:
Only jobs on ARM timed out

Collecting more details in #11896

[alexanderlaw]

When running performance comparison for two versions (f94248a vs b2705cf), I've discovered that several tests give worse results for the newer version, and one of them is sysbench, more specifically, the oltp_write_only workload:

Best neon-src-pg17-main-2507--.* worse than neon-src-pg17-main-2501--.* by 26.7 percent (66221.19 < 90375.46): sysbench.load_oltp_write_only_ops
Average neon-src-pg17-main-2507--.* worse than neon-src-pg17-main-2501--.* by 22.1 percent (65503.08 < 84120.45): sysbench.load_oltp_write_only_ops

(that's the comparison of results from three runs of each version)

git bisect led me to 79ddc80, where I saw

+virtual_file_io_mode="buffered" # the CI runners where we run the docker compose tests have slow disks

and I confirmed that this setting really improves performance. I run tests on a bare metal Hetzner box EX44 (i5-13500 with two nvme drives combined into md-raid-1):

virtual_file_io_mode="direct-rw"
    queries:                             4194956 (69910.77 per sec.)
    queries:                             3902390 (65035.62 per sec.)
    queries:                             4004186 (66211.59 per sec.)
    queries:                             3793630 (62678.27 per sec.)
    queries:                             4422601 (73701.76 per sec.)

vs

virtual_file_io_mode="buffered"
    queries:                             4317267 (71950.35 per sec.)
    queries:                             4474432 (74567.74 per sec.)
    queries:                             4438765 (73970.43 per sec.)
    queries:                             4575301 (76247.35 per sec.)
    queries:                             4487703 (74785.26 per sec.)

The script I use to reproducing this:
test-perf.sh.txt

Would it make sense to investigate this deeper, say, to find the exact simple workload, which will show the difference more prominently? Or maybe this behavior is more or less known and expected?
IMaybe it's well-known under which condtions the direct-rw setting should improve performance?

@problame , could you share your thoughts on this?

[problame]

Your script disables fsync (sync = false) and you're setting = buffered here.

Further your script doesn't configure PSes in the production configuration, e.g. with sufficient PS PageCache or large enough virtual file fd cache.

We don't care about these != production settings in most cases, and certainly not with test-images.

This ticket is about fixing test-images timeouts, let's constrain it to this. Is suggest to either decide that we don't care about test-images using =buffered, or we adjust the test timeout, or we adjust the workload size such that it fits into the 20min envelope.

For discussions on direct IO, I suggest to read the backlog of #proj-pageserver-direct-io , lots of discussions on perf wrt direct IO enablement happened there, don't want to rehash it.

[alexanderlaw]

@problame , thank you for your attention to this and for the explanation and the link! I disabled sync to speed-up performance testing on that machine (with sync=on for safekeeper, tests run ~4.5 times slower: https://www.notion.so/neondatabase/Benchmarking-neon-with-pg-mark-1cbf189e004780eba94dd068d39045e1?source=copy_link#1d0f189e004780139cbcfa75c3b363dd)

I'll try to enable fsync for pageserver (it was disabled by default when neon_local is used, in #9677, if I'm not mistaken) to re-evaluate the effect of direct-rw, but my point was that that change made a difference, if we compare January vs July versions (on the concrete machine). There are other differences in terms of performance, of course, I just started from this one.

[problame]

but my point was that that change made a difference, if we compare January vs July versions (on the concrete machine)

Yeah, not surprising perf is better if you

1. disable durability (sync = false) and
2. use more DRAM (=buffered)

But we

1. want durability and
2. be in control of our DRAM usage, and =direct-rw is the tool to do that.

Again, let's please constrain this ticket to making the test not time out anymore. If you think you have found a severe case (let's say > 10% on some relevant perf metric, tput or latency), then please open a separate issue for that, so Storage team can allocate time for an investigation.

[zenithdb-bot-dev]

This issue was moved to Jira: LKB-1605