Kafka: Add support for the delete-records API

[graphcareful]

This PR adds the delete-records feature to redpanda. This feature allows a kafka client to truncate data from a topic/partition at a given offset. To support this in redpanda two non-trivial tasks were implemented:

1. Addition of new stm (and corresponding special batch type) to process prefix truncation events.
2. Addition of new concept (visible_start_offset) which is a delta into the first segment of the log representing the first offset a fetch request could read from.

Note, as of this branch delete-records requests for cloud enabled topics will be rejected. Work on supporting this is slated to be completed in a follow up PR, which depends on a currently still in review PR #9994

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.1.x
• v22.3.x
• v22.2.x

Release Notes

Features

• Adds support for the delete-records API

[graphcareful]

Added more ducktape tests

[emaxerrno]

wondering if franz-go has some delete records test, or if the librdkafka test suite should be enabled for this or some other external-API project that exercises this path for edge case detection.

[jcsp]

Looks like lots of tests are failing, perhaps related to the timeout change in the PR.

[graphcareful]

Changes in force-push

• Modify prefix_truncate name schema to delete_records
• Fixed bug where truncating with -1 (high watermark) may cause a crash
• Fixed bug that made offset translation not work correctly due to not excluding the new special batch type in translation accounting
• Fixed bug that made delete_records request return early due to off-by-one error in firing new min offset event notification. Returned low_watermark would appear lower then expected.
• Fixed bug that would made subsequent request to prefix truncate within the same segment fail due to not updating conditionals in consensus::write_snapshot
• Fixed bug that would cause a crash when writing a snapshot at the first segment. consensus::write_snapshot expects get_term to always work, which was previously true since write_snapshot would never be called at a last_included_index value of 0. Now this is not the case since write_snapshot can be called just to increment the new log start delta.
• Fixed bug where log_eviction_stm would start processing events from start_offset() instead of visible_start_offset(). No issues arose because events would be ignored, after correctly determining that the older events aren't applicable to a log with a start_offset + delta ahead of the requested.
• Fixed bug where delete-records at the high watermark - 1 offset (truncate all data) failed due to off-by-one error
• Fixed bug where log_eviction_stm was not truncating the underlying log at segment end offsets but rather segment start offsets. (only for events arisen from delete-records requests)
• Fixed bug that caused some CloudStorage tests in CI to fail, random default timeout was incorrectly modified
• Modify KCL::delete_records for better error condition handling
• Much more ducktape tests

[graphcareful]

Leaving a comment here about the proposed testing plan. All testing will be done in Ducktape, desired scenarios to test are:

• Test moving delta of current segment
• Test moving delta of current segment multiple times in a row
• Test deletion of data at segment boundary
• Test deletion of data within a segment, between boundaries
• Test truncation at offsets <= 0 and >= high watermark all result in out of bounds errors
• Test correct errors returned on topics/partitions that don't exist
• Test truncation at high watermark (-1) deletes all data
• Test an edge case situations where begin, end are very close like [38, 40)
• Test all of the above parameterized with cloud storage enabled topics
• Test integration with transactions
• Test scenarios where node failures are introduced (list scenarios - in-progress)

[graphcareful]

Test failures look like:

FAIL: <NodeCrash docker-rp-17: ERROR 2023-04-18 01:52:13,290 [shard 1] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ffc8dc9474419720-1/redpanda/redpanda/src/v/raft/consensus.cc:2273) 'config.has_value()' Configuration for offset 7919 must be available in configuration manager: {configurations: { offset: {9049}, idx: {1},cfg: {current: {voters: {{id: {3}, revision: {25}}, {id: {1}, revision: {25}}, {id: {2}, revision: {25}}}, learners: {}}, old:{nullopt}, revision: 25, update: {nullopt}, version: 5}} } 
>

Note this must indicate a bug in the changes I've made to log_eviction_stm, when handling GC events from storage.

[graphcareful]

Changes in force-push

• Rebase against dev to resolve conflicts in feature_table (another change added a new feature)
• Modified log_eviction housekeeping loop to periodically sleep when there is no work to do, or in the case the work item was re-enqueued due to the offset being greater then the max collectible.
• Fixed a bug that was resulting in test failures incorrect variable was being referred to in log_eviction_stm::do_process_log_eviction
• Fixed a newly written flaky ducktape test by repeating if NOT_LEADER is observed
• Made some changes around previous off-by-one fixes to remove confusing logic around subtracting an extra 1 here and there.

[VladLazar]

There's still some things I don't understand, so I'll need to come back to it.

[graphcareful]

Changes in force-push

• Fix for CI issues which were causing some retention based tests to fail. Issue was writing a snapshot with offset::min as the value of log_start_delta
• Fixed issue where truncation at bounds wasn't entirely respected, new ducktape test written to test for this case. (log of one record, truncate at 0 & 1)
• Expanded a ducktape test to truncates at the high watermark, only truncation at 1 before the high watermark was tested.

[graphcareful]

Changes in force-push

• Reduce eviction housekeeping interval from 1s to 200ms. Fixed a failing raft unit test that had a 2s timeout on checking a segments eviction. 1s also seemed a bit too long, open to discussion on this.
• Removed a useless tracelog from log_eviction_stm

[graphcareful]

CI failures

• CI Failure (fatal error: in "test_counts_rebalancing") in partition_balancer_simulator_test_rpunit #11593
• CI Failure (unknown location(0): fatal error: in "test_adding_multiple_nodes": seastar::timed_out_error: timedout) in cluster_replicas_rebalancing_tests_rpfixture #11455
• [v22.2.x] Update start timeout for leadership_transfer_test on debug builds #11480

Investigating a leak seen in RackAwarePlacementTest.test_replica_placement

[graphcareful]

/ci-repeat 1
debug
skip-units
dt-repeat=20
tests/rptest/tests/rack_aware_replica_placement_test.py::RackAwarePlacementTest.test_replica_placement

[graphcareful]

/ci-repeat 1
debug
skip-units
dt-repeat=20
tests/rptest/tests/memory_stress_test.py::MemoryStressTest.test_fetch_with_many_partitions

[graphcareful]

/ci-repeat 1
debug
skip-units
dt-repeat=200
tests/rptest/tests/memory_stress_test.py::MemoryStressTest.test_fetch_with_many_partitions