tokio-epoll-uring: use it for on-demand downloads

[problame]

Problem

On-demand downloads are still using tokio::fs, which we know is inefficient.

Changes

• Add pagebench ondemand-download-churn to quantify on-demand download throughput
  • Requires dumping layer map, which required making history_buffer impl Deserialize
• Implement an equivalent of tokio::io::copy_buf for owned buffers => owned_buffers_io module and children.
• Make layer file download sensitive to io_engine::get(), using VirtualFile + above copy loop
  • For this, I had to move some code into the retry_download, e.g., sync_all() call.

Drive-by:

• fix missing escaping in scripts/ps_ec2_setup_instance_store
• if we failed in retry_download to create a file, we'd try to remove it, encounter NotFound, and abort() the process using on_fatal_io_error. This PR adds treats NotFound as a success.

Testing

Functional

• The copy loop is generic & unit tested.

Performance

• Used the ondemand-download-churn benchmark to manually test against real S3.
• Results (public Notion page): https://neondatabase.notion.site/Benchmarking-tokio-epoll-uring-on-demand-downloads-2024-04-15-newer-code-03c0fdc475c54492b44d9627b6e4e710?pvs=4
  • Performance is equivalent at low concurrency. Jumpier situation at high concurrency, but, still less CPU / throughput with tokio-epoll-uring.
  • It’s a win.

Future Work

Turn the manual performance testing described in the above results document into a performance regression test: #7146

[koivunej]

OwnedAsyncWriter looks good, nice tests around the buffering.

[problame]

Only got to do the performance testing now, but, I think this is a fairly safe change, so, merging it now, even if there's less soak time before the next release.