test_crafted_wal_end flakyness

[koivunej]

The test_crafted_wal_end has been flaky in a number of ways. This issue tracks all findings.

test_crafted_wal_end[release-pg15-last_wal_record_xlog_switch_ends_on_page_boundary]: release

[koivunej]

test_crafted_wal_end[debug-pg15-last_wal_record_crossing_segment]: debug

[koivunej]

test_crafted_wal_end[debug-pg15-wal_record_crossing_segment_followed_by_small_one]: debug

[jcsp]

https://github.com/neondatabase/neon/actions/runs/5752601432/job/15594169701?pr=4890

 FAILED test_runner/regress/test_crafted_wal_end.py::test_crafted_wal_end[release-pg15-last_wal_record_xlog_switch_ends_on_page_boundary] - RuntimeError:             Run ['/tmp/neon/bin/wal_craft', 'in-existing', 'last_wal_record_xlog_switch_ends_on_page_boundary', "host=localhost port=29015 user=cloud_admin dbname=postgres options='-cstatement_timeout=120s '"] failed:
              stdout: 
              stderr: [2023-08-03T15:27:26Z INFO  wal_craft] current_wal_insert_lsn=0/14F51F8, remaining_lsn=11784, base_wal_advance=8368, repeats=3426

[awestover]

could this be the same not flushing xlog problem as from #559 ?

[awestover]

John's Allure report had this in it:

2023-08-03T17:41:47.279783Z  INFO http request{otel.name=/extension_server/neon_test_utils http.method=POST}: serving /extension_server POST request, filename: "neon_test_utils" is_library: false
2023-08-03T17:41:47.279892Z ERROR http request{otel.name=/extension_server/neon_test_utils http.method=POST}: extension download failed: No remote extension storage

really hope it's unrelated to the test failure
but if it was, I think I wrote a patch to fix it

[jcsp]

@arssher can you look at recent failures of this test and see if it is the same issue as this ticket?

[arssher]

Some are
https://neondb.slack.com/archives/C033RQ5SPDH/p1714652474499819
and some are
#7588

[arssher]

#7592 is for the slack thread in the previous comment