refactor(storage): remove reading range tombstone in read path and compaction

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Resolve #14824

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Prior to version 1.6, for streaming jobs using dynamic filter (such as materialized views with temporal filter t > NOW() - INTERVAL '1 hour'), and sinks with decouple enabled (enabled by default for append only kafka, pulsar sinks, or explicitly enabled by setting session variable set SINK_DECOUPLE=true), we write range delete tombstones to our storage engine to do state cleaning. In release 1.6, we changed to use a much lighter way to do state cleaning, while maintaining compatibility with range delete tombstones. In the current release, we are going to deprecate the compatibility to range delete tombstones. Deprecating range delete tombstones is still compatible for materialized view with dynamic filter anyway. However, it may cause compatibility for sinks with decouple enabled if users upgrade to the current release version from a version before version 1.6.

Therefore, it is not recommended to upgrade from a version before 1.6 directly to the current version. If you are doing so, please make sure that no sink is created with decouple enabled. In version 1.7, we supported an internal table rw_sink_decouple to see the decouple status of all sinks. It is not required, but recommended, to first upgrade to version 1.7, and then run select * from rw_sink_decouple where is_decouple and watermark_vnode_count < 256 to see whether there is some incompatible sinks. If the query returns empty result, it means it's safe to upgrade to the current release version. In most cases that upgrade from version since 1.6, it should be compatible.

[MrCroxx]

Generally LGTM, I'll review it later after the assertions in the tests are restored.

BTW, don't forget to remove unused comments. 🥰

[wenym1]

Generally LGTM, I'll review it later after the assertions in the tests are restored.

BTW, don't forget to remove unused comments. 🥰

Most comments are on ignored unit test. The tests are remained for reference. Some codes are commented out only to pass the compile.

[Li0k]

Rest LGTM

[hzxa21]

Deprecating range delete tombstones is still compatible for materialized view with dynamic filter anyway.

The reason why it is compatible is because the read key range of dynamic filter doesn't overlap with the range tombstone, right?

[wenym1]

Deprecating range delete tombstones is still compatible for materialized view with dynamic filter anyway.

The reason why it is compatible is because the read key range of dynamic filter doesn't overlap with the range tombstone, right?

Yes. It seems that dynamic filter executor has a right table to store the watermark range, and when querying the left table, it will build a range that does not overlap with the range tombstone.