[Bug] Streaming job stuck in PENDING with infinite MetaService retries in cloud mode

[maks3201]

Search before asking

• I searched in the issues and found no similar issues.

Version

master (4.2-SNAPSHOT), also affects 4.1.x

What's Wrong

In compute-storage-decoupled (cloud) mode, when a streaming insert job is created for the first time (or after MetaService loses its progress data), replayOnCloudMode() queries MetaService for persisted job progress. MetaService responds with STREAMING_JOB_PROGRESS_NOT_FOUND, which is the expected answer when no transaction has been committed yet.

However, replayOnCloudMode() returns void, so callers cannot distinguish "no progress exists" from "progress loaded successfully." As a result:

1. In handlePendingState() (StreamingJobSchedulerTask): the scheduler calls replayOnCloudMode() on every tick, receives the same NOT_FOUND response, and never transitions the job out of PENDING.
2. In journal replay: every UPDATE_JOB edit-log entry triggers another doomed RPC, flooding the FE log with WARN messages during startup.

Observed symptoms:

• Streaming job remains permanently stuck in PENDING state
• FE log fills with repeated lines: not found streaming job progress, response: ...
• In severe cases (many streaming jobs), the log volume during journal replay delays FE startup

How to Reproduce

1. Deploy Doris in compute-storage-decoupled mode
2. Create a streaming insert job (e.g., CDC from MySQL)
3. Observe the job status — it stays PENDING indefinitely
4. Check fe.log for repeated warnings about not found streaming job progress

The issue is 100% reproducible for any newly created streaming job that has not yet committed its first transaction. It also occurs for existing jobs if MetaService's FDB backing store loses the progress key.

Root Cause

replayOnCloudMode() has return type void. When it encounters STREAMING_JOB_PROGRESS_NOT_FOUND, it logs a warning and returns without any signal to the caller. The caller has no way to know the attempt was a permanent "not found" rather than a successful recovery, so it retries every scheduler tick.

Expected Behavior

• replayOnCloudMode() should return a boolean indicating whether progress was recovered
• On NOT_FOUND, the job should proceed using its configured offset property rather than waiting for cloud-persisted progress
• Repeated calls for the same job should be suppressed until the job actually commits a transaction (at which point progress will exist in MetaService)

Anything Else

This affects any deployment using the compute-storage-decoupled architecture. The fix is straightforward: change the return type to boolean and add a transient flag to prevent repeated RPCs.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_2bed6094f41f

Maintainer triage: the reported PENDING root cause is contradicted by the current control flow. The journal-replay RPC/log amplification is real in the code, but the issue does not yet demonstrate that STREAMING_JOB_PROGRESS_NOT_FOUND keeps a job PENDING. I would not accept the linked PR #66561 as-is without a reproducer and tests.

Verified facts

• I checked upstream/master at 4e3c1b84dd5936b8d751e0a95719350e8cd8e980 and upstream/branch-4.1 at e0115cac0cd7fcef867801469a05ceb2b3ca1616; both have the same relevant control flow.
• MetaService returns STREAMING_JOB_PROGRESS_NOT_FOUND when the (instance_id, db_id, job_id) progress key is absent. The first streaming transaction commit creates that key, so NOT_FOUND is expected for a genuinely new, never-committed job.
• On NOT_FOUND, StreamingInsertJob.replayOnCloudMode() returns normally from that method. In StreamingJobSchedulerTask.handlePendingState(), execution then continues to replayOffsetProviderIfNeed(), advanceSplitsIfNeed(), task creation, and updateJobStatus(RUNNING). The only early returns in this path are for an RPC/error response (which pauses the job), an independently paused split-advance path, or a source that has reached its end. Therefore the void return type does not cause a scheduler tick to stop, and NOT_FOUND alone cannot explain a permanent PENDING state.
• The journal concern is mechanically valid: JobManager.replayUpdateJob() calls StreamingInsertJob.replayOnUpdated(), which unconditionally calls replayOnCloudMode() in cloud mode. Thus each replayed streaming UPDATE_JOB can issue a synchronous MetaService RPC. For NOT_FOUND, the current implementation also emits two WARN messages (the generic failure warning and the specific not-found warning). The claimed startup delay/volume is plausible, but no timing, entry count, or log sample was supplied to quantify it.
• The issue currently has no labels, assignee, or discussion. It is cross-referenced by PR [fix](streaming) avoid infinite retry when cloud mode job progress is not found #66561.

Assessment of PR #66561

• The new boolean result is used by handlePendingState() only for an INFO message; both true and false still follow the same task-creation/status-transition path. Consequently, that change cannot account for the PR's claimed before/after PENDING behavior.
• The transient cloudProgressMissing flag can suppress repeated journal-replay RPCs after the first NOT_FOUND, but the PR has no automated test for that behavior.
• A missing key has two materially different meanings: “this job has never committed” and “previously committed MetaService progress was lost.” Silently treating the latter as a fresh start can replay data or create duplicates, depending on what FE/provider state remains. The recovery policy must distinguish or explicitly guard these cases before promising fallback to the configured initial offset.
• The flag is cleared only by the active afterCommitted() callback. Tests should also cover journal replay, FE role change, and progress becoming available through another recovery path so an in-process negative cache cannot mask newly available progress.

Information needed to confirm the reported incident

1. The exact FE build/git SHA (not only master or 4.1.x) and the sanitized CREATE JOB statement, including source/job offset mode.
2. The affected job ID plus status/task output showing that it remains PENDING across multiple scheduler intervals.
3. A contiguous FE log excerpt from job creation through at least two scheduler intervals, including not found streaming job progress, create new streaming insert task, Streaming insert job ... update status, and any exceptions from offset-provider replay, split advancement, source-reader setup, task registration, or scheduling.
4. For the startup-impact claim: number of streaming jobs and replayed UPDATE_JOB entries, count of MetaService calls/WARN lines, and startup timing with timestamps.
5. For the “MetaService lost progress” case: evidence that the job had previously committed and what recoverable FE/provider offset state still existed. This case should not be tested by simply resuming from the original configured offset without validating duplicate-load safety.

Recommended next steps

1. Add an FE test with a mocked MetaService response proving the current baseline: NOT_FOUND must still create the task and transition PENDING to RUNNING in the same scheduler invocation. If that passes—as the control flow indicates—the reported stuck state has another cause and the supplied logs should identify the actual failing step.
2. Address journal amplification separately: perform cloud recovery once per job/recovery epoch (or after edit-log replay completes), treat first-time NOT_FOUND as an expected INFO-level outcome, and avoid the duplicate WARN. Do not use a lifetime-wide negative cache without defining invalidation for commit replay and FE role transitions.
3. Add tests for multiple UPDATE_JOB records with an absent key, successful recovery, non-NOT_FOUND/RPC errors, a later-appearing progress key, and a previously committed job whose MetaService key is missing.

No source changes were made during this triage.

[maks3201]

Closing. The stuck PENDING root cause in this issue is incorrect, see the detailed explanation in #66561. Only a minor journal replay deduplication remains and it does not match this issue description, so a fresh narrow issue is more appropriate if I pursue it.