[refactor](storage) Unify BE and Recycler object clients

[sollhui]

1. What does this PR do?

BE and Cloud Recycler previously maintained separate object-storage abstractions and separate S3/Azure implementations. Although both sides called the same cloud-provider SDKs, credential construction, error conversion, metrics, pagination, batch deletion, and compatibility behavior were duplicated and could evolve differently.

This PR consolidates the implementation under common/cpp/client and exposes one ObjStorageClient facade to upper layers:

• ObjStorageClient owns backend-independent orchestration and is the only complete client used by BE and Recycler call sites.
• ObjStorageRateLimitPolicy keeps BE- and Recycler-specific admission behavior injectable without coupling common code to either environment.
• ObjStorageBackend is the storage implementation boundary, implemented by S3ObjStorageBackend and AzureObjStorageBackend.
• Shared request/response types, page-based listing, upper-layer lazy iteration, recursive deletion, backend batch capabilities, credentials, metrics, and error conversion live in the common layer.
• Request admission remains attached to actual backend work: one GET admission per list page and one PUT admission per backend-sized delete batch.

Before the refactor, BE and Recycler reached the cloud SDKs through parallel stacks:

                                     BEFORE

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder              |     |  Recycler call sites -> S3Accessor            |
  |                       |                        |     |                       |                        |
  |                       v                        |     |                       v                        |
  |  BE limiter + separate S3 / Azure clients      |     |  Recycler limiter + separate S3 / Azure clients|
  |  credentials / listing / recursive deletion    |     |  credentials / listing / recursive deletion    |
  |  error conversion / metrics                    |     |  error conversion / metrics                    |
  +------------------------+-----------------------+     +------------------------+-----------------------+
                           |                                                        |
                           v                                                        v
                    AWS SDK / Azure SDK                                      AWS SDK / Azure SDK

After the refactor, BE and Recycler stay on the left and right while the shared facade and backend components are centered below them:

                                      AFTER

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder / factory    |     |  Recycler call sites -> S3Accessor / adapter  |
  +-----------------------------+------------------+     +------------------+-----------------------------+
                                |                                           |
                                +-------------------+-----------------------+
                                                    |
                                                    v
                          +--------------------------------------------------+
                          |            ObjStorageClient facade               |
                          |                                                  |
                          |  +----------------------+  +-------------------+  |
                          |  | RateLimitPolicy      |  | ObjStorageBackend |  |
                          |  | - BE policy          |  |        |          |  |
                          |  | - Recycler policy    |  |   +----+----+     |  |
                          |  +----------------------+  |   |         |     |  |
                          |                            |   v         v     |  |
                          |                            | S3 Backend Azure   |  |
                          |                            | Backend            |  |
                          |                            +---+---------+------+  |
                          +--------------------------------|---------|---------+
                                                           v         v
                                                        AWS SDK   Azure SDK

This boundary prevents a raw backend from being used as the complete client and accidentally bypassing runtime policy. Backend code only implements cloud mechanics; common orchestration and policy dispatch remain in the facade.

2. How are the different behaviors unified?

Behavior	BE before	Recycler before	Unified behavior
Client API	BE-specific doris::io::ObjStorageClient and eager list results	Recycler-specific client and iterator APIs	One doris::ObjStorageClient facade and one set of request/response types. doris::io aliases keep BE call sites source-compatible; Recycler adapters preserve its integer-facing API.
Backend implementation	Separate BE S3/Azure clients	Separate Recycler S3/Azure clients	S3ObjStorageBackend and AzureObjStorageBackend are shared by both callers.
Rate limiting	BE owned QPS/bytes limiters and bucket-selection rules	Recycler owned its limiter and fault injection	Each environment injects an ObjStorageRateLimitPolicy; the facade performs admission immediately before backend work. Each list page and each backend-sized delete batch is admitted independently.
Error model	BE status codes plus HTTP metadata on selected paths	Recycler-specific return codes and messages	ObjectStorageResponse consistently carries a Doris status code, HTTP code, and request ID, while adapters preserve caller-facing behavior such as the Recycler 0/1/negative exists contract.
Listing	BE eagerly collected all pages	Recycler exposed a lazy iterator	ObjStorageClient::list_objects returns one fixed-size ObjectStorageListPage. One Client call performs one GET admission, one Backend call, and one SDK request. The upper ObjectListIterator owns the continuation token and requests the next page only after its cached page is consumed.
End of listing	Eager completion was represented by a finished vector	Iterator completion used empty/false results	END_OF_FILE is an internal upper-iterator sentinel and next() converts it to a successful empty result. Backend NOT_FOUND remains a real error.
Missing S3 prefix	S3-compatible NoSuchKey handling existed in the BE path	The compatibility behavior was maintained separately	The shared S3 backend preserves NoSuchKey-as-empty behavior once for both callers.
Direct batch deletion	Backend limits were embedded in separate implementations	Recycler maintained its own batching	The facade splits by backend capability (1000 for S3, 256 for Azure), acquires one PUT admission per backend-sized batch, and keeps defensive bounds in each backend.
Recursive deletion	Separate implementations and grouping behavior	Recycler supported expiration filtering and parallel execution	The facade owns one shared listing/filtering/grouping/error-propagation flow. Every list page goes through the policy-bearing one-page API, and every delete task acquires PUT admission immediately before its backend request. Recycler injects its SyncExecutor; BE uses the synchronous fallback.
AWS credentials	BE built static/default/role providers in its factory	Recycler maintained another construction path	AwsCredentialFactory implements static credentials, default provider chains, role ARN, and external ID once while callers retain their prior empty-credential behavior.
Azure credentials	BE and Recycler built shared-key clients separately	Separate construction and credential retention	AzureAuthFactory creates the container client and shared-key credential for both; BE TLS diagnostic context remains attached to Azure errors.
Metrics and latency	Duplicated stopwatch and failure-recording paths	Separate helpers recorded equivalent data	Backends use the shared client_bvar::ScopedLatency timer and common failure metrics. The previous BE 5-second slow-request logging behavior is retained for S3 uploads.
Backend-specific APIs	Shared interfaces forced unrelated test stubs	Lifecycle, versioning, and multipart-abort were Recycler-oriented	The backend boundary supplies default not-supported responses and implements supported APIs without leaking backend details to callers.

3. Design boundaries and follow-ups

• Upper layers keep std::shared_ptr<ObjStorageClient>; they do not store ObjStorageBackend directly.
• BE and Recycler own their policy configuration, but both use the same facade dispatch and the same S3/Azure backends.
• The upper ObjectListIterator performs lazy iteration by repeatedly calling the one-page Client API. Therefore each requested page has exactly one facade admission and one SDK request; reading objects already cached in that page performs no network request.
• A public delete_objects call may split input according to backend capability, but each resulting backend batch acquires its own PUT admission before issuing one SDK request.
• Recursive deletion composes the same one-page list and backend-sized delete operations. Consequently every actual list page and delete batch is counted by the injected rate-limit policy; Recycler may execute delete tasks in parallel through SyncExecutor.
• Local compilation and tests were not run as requested. Changed C++ files were formatted and git diff --check passed.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[sollhui]

run buildall

[sollhui]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30693205073

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[sollhui]

run buildall

[sollhui]

/review

[sollhui]

run buildall

[sollhui]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30794530392

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[github-actions]

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30797931848

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

[sollhui]

/review

[sollhui]

run buildall

[github-actions]

Static-only review of the full authoritative diff found six issues that should be addressed before merge (five P1, one P2).

Review-cycle status: incomplete after the three-round cap. Both normal agents returned NO_NEW_VALUABLE_FINDINGS in Round 3, but the risk-focused agent found a final FE-side scope correction that was independently verified and merged into the token round-trip comment; the review contract does not permit a fourth round. All currently known candidates are nevertheless adjudicated and included below.

Critical checkpoint conclusions:

• Data correctness: failed. Session-token credentials are dropped/staled across FE DDL and meta-service paths, and Recycler exists status mapping can turn real provider failures into false not-found results.
• Concurrency and lifecycle: delete-task ownership, executor waiting, batch clamping, and error propagation are sound; request admission during recursive deletion is not.
• Configuration and dynamic behavior: Recycler rate limiting and PUT fault injection are bypassed for the actual recursive-delete SDK requests; AWS provider precedence, refresh-capable providers, and client cache identity otherwise remain compatible.
• Compatibility and rolling behavior: the optional protobuf field is wire-compatible, but the Recycler 0/1/negative adapter contract and GCS iterator migration are broken.
• Parallel paths: BE/Recycler and S3/Azure/GCS paths were traced; the GCS path has an unconditional compile failure and Recycler differs from the preserved BE admission behavior.
• Tests and validation: no builds or tests were run, as required by the review prompt. Existing S3 accessor tests still require 1 for not-found, and there is no end-to-end token persistence/redaction/rotation coverage; the GCS compile error is statically evident.
• Observability and security: the session token lacks SK-equivalent encryption/log/display handling, and successful S3 writes now log at INFO on the hot path. This is credential-secret handling within authenticated control paths; no unsupported cross-tenant vulnerability claim is made.
• Persistence and recovery: token-bearing vault/stage records can either lose the token or retain it plaintext, so persistence round trips are not safe.
• Performance: recursive deletion can evade Recycler request controls, while per-write INFO logging adds log I/O proportional to storage QPS; page and provider batch limits themselves are sound.

User focus: review_focus.txt supplied no additional focus, so the entire PR was reviewed without narrowing scope.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

[sollhui]

run buildall

[sollhui]

run buildall

[sollhui]

run buildall

[sollhui]

/review

[github-actions]

Request changes: two P2 test regressions leave important failure paths able to pass unverified.

Findings:

• The migrated S3/role/Azure iterator loops do not assert the terminal iterator response, so an expected-empty post-delete check can pass when the list request actually failed.
• The replacement recursive-delete tests exercise only the shared facade's sequential fallback, while Recycler production always uses the parallel SyncExecutor path and its cancellation/unfinished-task handling.

Checkpoint conclusions:

• Goal and scope: reviewed all 63 authoritative changed paths and the full object-client unification across CommonCPP, BE, Cloud Recycler, credentials, build wiring, adapters, and tests. No additional user-provided focus was supplied, so the full PR remained the focus.
• Functional correctness and error handling: request-level GET/PUT admission, read-byte settlement, S3/Azure page and delete contracts, continuation tokens, prefix/key conversion, NOT_FOUND handling, and recursive-delete error propagation are internally consistent on this head. Known production concerns already raised in live threads were not duplicated.
• Concurrency and lifecycle: recursive tasks retain shared backend/policy ownership, batch limits are bounded, and executor failures are surfaced. The missing production-executor regression coverage is called out inline.
• Configuration, credentials, and compatibility: V1/V2 AWS provider selection, BE session tokens, role/external-ID/CA propagation, Azure shared-key construction, Result migration, namespace aliases, CommonCPP linkage, and Azure-on/off gating were checked statically with no additional substantiated defect.
• Persistence and data mutation: no FE/BE transaction or metadata protocol is introduced here; object deletion remains provider-batched and reports list/delete failures rather than clean partial success.
• Performance and observability: one admission maps to each provider list page/delete batch, provider batch caps remain 1000 for S3 and 256 for Azure, and latency/failure instrumentation plus slow-request logging remain present.
• Tests and verification: the two inline comments identify the remaining coverage/oracle gaps. Per the authoritative review contract, this was a static-only review; no local build or test command was run.
• Completion: two review rounds converged on this exact two-comment set, all candidates were accepted, deduplicated, or dismissed with evidence, and the live head/base still match the authoritative bundle.

[sollhui]

run buildall

[sollhui]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30913544145

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[sollhui]

run buildall

[sollhui]

/review

[sollhui]

run buildall

[sollhui]

/review