[Feature](compaction) add CompactionTaskTracker with system table and HTTP API

[Yukang-Lian]

Summary

Introduce CompactionTaskTracker to provide full lifecycle observability for base/cumulative/full compaction tasks. This PR adds two query channels:

1. information_schema.be_compaction_tasks system table (38 columns) — query via SQL across all BEs
2. GET /api/compaction/profile HTTP API — query via JSON on a single BE, with filtering support

What problem does this PR solve?

Currently, compaction execution metrics (input/output data sizes, row counts, merge latency, etc.) are lost once the compaction object is destructed. Operators have no way to inspect historical compaction performance — only the current running status is available. Users also cannot see pending or running compaction task details through SQL.

This PR solves both problems with a unified CompactionTaskTracker mechanism that tracks compaction tasks across their full lifecycle: PENDING → RUNNING → FINISHED/FAILED.

Architecture

                         ┌── Push ──┐
  Compaction entry points ──→ CompactionTaskTracker (in-memory) ←── Push ── Compaction base class
   (register_task)            │  _active_tasks (PENDING/RUNNING)          (complete/fail)
                              │  _completed_tasks (FINISHED/FAILED)
                              │
               ┌──── Pull ────┴──── Pull ────┐
               ↓                              ↓
   SchemaCompactionTasksScanner     CompactionProfileAction
    get_all_tasks() snapshot         get_completed_tasks() filtered
               ↓                              ↓
   FE fan-out to all BEs → SQL      JSON HTTP Response

• Push-based collection: Compaction entry points and execution layer push task info at lifecycle boundaries
• Pull-based querying: System table scanner and HTTP action pull snapshots with read locks, never blocking compaction
• Pure in-memory storage: Two containers, no persistence. Data clears on BE restart.

---

System Table: information_schema.be_compaction_tasks

Schema (38 columns)

Column	Type	Description
BACKEND_ID	BIGINT	BE node ID
COMPACTION_ID	BIGINT	Unique task ID
TABLE_ID	BIGINT	Table ID
PARTITION_ID	BIGINT	Partition ID
TABLET_ID	BIGINT	Tablet ID
COMPACTION_TYPE	VARCHAR	base / cumulative / full
STATUS	VARCHAR	PENDING / RUNNING / FINISHED / FAILED
TRIGGER_METHOD	VARCHAR	AUTO / MANUAL / LOAD_TRIGGERED
COMPACTION_SCORE	BIGINT	Tablet compaction score at registration time
SCHEDULED_TIME	DATETIME	Task registration time
START_TIME	DATETIME	Execution start time (NULL when PENDING)
END_TIME	DATETIME	Execution end time (NULL when not completed)
ELAPSED_TIME_MS	BIGINT	RUNNING: real-time now - start_time; FINISHED/FAILED: end - start
INPUT_ROWSETS_COUNT	BIGINT	Number of input rowsets
INPUT_ROW_NUM	BIGINT	Number of input rows
INPUT_DATA_SIZE	BIGINT	Input data size (bytes)
INPUT_INDEX_SIZE	BIGINT	Input index size (bytes)
INPUT_TOTAL_SIZE	BIGINT	Input total size (data + index)
INPUT_SEGMENTS_NUM	BIGINT	Number of input segments
INPUT_VERSION_RANGE	VARCHAR	Input version range, e.g., [0-5]
MERGED_ROWS	BIGINT	Merged row count
FILTERED_ROWS	BIGINT	Filtered row count
OUTPUT_ROWS	BIGINT	Merger output rows (0 for ordered compaction)
OUTPUT_ROW_NUM	BIGINT	Output rowset row count
OUTPUT_DATA_SIZE	BIGINT	Output data size (bytes)
OUTPUT_INDEX_SIZE	BIGINT	Output index size (bytes)
OUTPUT_TOTAL_SIZE	BIGINT	Output total size (data + index)
OUTPUT_SEGMENTS_NUM	BIGINT	Number of output segments
OUTPUT_VERSION	VARCHAR	Output version range
MERGE_LATENCY_MS	BIGINT	Merge phase latency in ms (0 for ordered compaction)
BYTES_READ_FROM_LOCAL	BIGINT	Bytes read from local storage
BYTES_READ_FROM_REMOTE	BIGINT	Bytes read from remote storage
PEAK_MEMORY_BYTES	BIGINT	Peak memory usage (bytes)
IS_VERTICAL	BOOLEAN	Whether vertical compaction was used
PERMITS	BIGINT	Compaction permits consumed
VERTICAL_TOTAL_GROUPS	BIGINT	Vertical compaction total column groups (0 if horizontal)
VERTICAL_COMPLETED_GROUPS	BIGINT	Completed column groups (real-time during RUNNING)
STATUS_MSG	VARCHAR	Error message (empty on success)

SQL Query Examples

-- View all running compaction tasks across all BEs
SELECT BACKEND_ID, TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'RUNNING'
ORDER BY ELAPSED_TIME_MS DESC;

-- Track vertical compaction progress for large compactions
SELECT BACKEND_ID, TABLET_ID, COMPACTION_TYPE,
       VERTICAL_COMPLETED_GROUPS, VERTICAL_TOTAL_GROUPS,
       CONCAT(ROUND(VERTICAL_COMPLETED_GROUPS * 100.0 / NULLIF(VERTICAL_TOTAL_GROUPS, 0), 1), '%') AS progress,
       ELAPSED_TIME_MS
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'RUNNING' AND IS_VERTICAL = true;

-- Find the slowest finished compactions
SELECT TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE, OUTPUT_DATA_SIZE,
       PEAK_MEMORY_BYTES, BYTES_READ_FROM_REMOTE
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED'
ORDER BY ELAPSED_TIME_MS DESC LIMIT 10;

-- Compaction statistics by trigger method
SELECT TRIGGER_METHOD, COUNT(*) AS cnt
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED' GROUP BY TRIGGER_METHOD;

-- Check remote IO ratio (disaggregated storage troubleshooting)
SELECT TABLET_ID, COMPACTION_TYPE, BYTES_READ_FROM_LOCAL, BYTES_READ_FROM_REMOTE,
       ROUND(BYTES_READ_FROM_REMOTE * 100.0 / NULLIF(BYTES_READ_FROM_LOCAL + BYTES_READ_FROM_REMOTE, 0), 2) AS remote_pct
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED' AND BYTES_READ_FROM_REMOTE > 0
ORDER BY remote_pct DESC;

-- View compaction history for a specific table
SELECT * FROM information_schema.be_compaction_tasks
WHERE TABLE_ID = 12345 ORDER BY SCHEDULED_TIME DESC LIMIT 20;

---

HTTP API: GET /api/compaction/profile

Queries completed compaction profiles on a single BE with optional filtering. All parameters can be combined (AND logic).

Request Parameters

Parameter	Type	Required	Description
tablet_id	int64	No	Filter by tablet ID
top_n	int64	No	Return most recent N records
compact_type	string	No	Filter by type: base / cumulative / full
success	string	No	Filter by result: true / false

Response Example

{
  "status": "Success",
  "compaction_profiles": [
    {
      "compaction_id": 487,
      "compaction_type": "cumulative",
      "tablet_id": 12345,
      "table_id": 67890,
      "partition_id": 11111,
      "trigger_method": "AUTO",
      "compaction_score": 10,
      "scheduled_time": "2025-07-15 14:02:30",
      "start_time": "2025-07-15 14:02:31",
      "end_time": "2025-07-15 14:02:31",
      "cost_time_ms": 236,
      "success": true,
      "input_rowsets_count": 5,
      "input_row_num": 52000,
      "input_data_size": 10706329,
      "input_index_size": 204800,
      "input_total_size": 10911129,
      "input_segments_num": 5,
      "input_version_range": "[12-16]",
      "merged_rows": 1200,
      "filtered_rows": 50,
      "output_rows": 50750,
      "output_row_num": 50750,
      "output_data_size": 5033164,
      "output_index_size": 102400,
      "output_total_size": 5135564,
      "output_segments_num": 1,
      "output_version": "[12-16]",
      "merge_latency_ms": 180,
      "bytes_read_from_local": 10706329,
      "bytes_read_from_remote": 0,
      "peak_memory_bytes": 33554432,
      "is_vertical": true,
      "permits": 10706329,
      "vertical_total_groups": 4,
      "vertical_completed_groups": 4
    }
  ]
}

Failed compactions additionally include "status_msg": "error message".

curl Examples

# Recent 10 compaction profiles
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?top_n=10"

# A specific tablet's recent 5 compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=5"

# Only failed base compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?compact_type=base&success=false"

# Combined: tablet's recent 3 failed cumulative compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=3&compact_type=cumulative&success=false"

---

Configuration

Config	Type	Default	Description
enable_compaction_task_tracker	mutable bool	true	Master switch. When disabled, all push operations are no-op, queries return empty.
compaction_task_tracker_max_records	mutable int32	10000	Max completed task records kept in memory (~500 bytes/record, default ~5MB).

---

Scope

• Tracked: base / cumulative / full compaction (local + cloud mode)
• Not tracked: single_replica, cold_data, index_change compaction (low frequency, limited diagnostic value)
• Trigger methods: AUTO (background scheduling), MANUAL (HTTP API), LOAD_TRIGGERED (load-time cumulative)

---

Test plan

• 15 unit tests (CompactionTaskTrackerTest.*) covering full lifecycle, failure paths, TRY_LOCK_FAILED cleanup, vertical progress, concurrent safety, config switch, filters
• Regression test: test_be_compaction_tasks (system table)
• Regression test: test_compaction_profile_action (HTTP API)

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?