feat: support backend upload & existence check for persistent tasks

[gaius-qi]

Description

This pull request introduces several improvements and refactorings across the codebase, focusing on configuration enhancements, storage engine API consistency, and ID generation capabilities. The most significant changes include adding a configurable backend upload timeout, renaming and standardizing the storage engine existence-check API, and implementing a new persistent task ID generation mechanism.

Configuration and Backend Upload Improvements

• Added a new put_timeout field to the Backend struct in dragonfly-client-config/src/dfdaemon.rs, allowing users to configure the maximum duration for uploading objects to backend storage. This is set to 15 minutes by default, and all relevant constructors, defaults, and tests were updated to support this new field. [1] [2] [3] [4] [5]
• Updated the dependency on dragonfly-api to version 2.2.1 in Cargo.toml.

Storage Engine API Consistency

• Renamed the storage engine trait method from is_exist to exists throughout the codebase for clarity and consistency. This includes updates to the trait definition, all implementations, and all usages in dragonfly-client-storage/src/storage_engine/mod.rs, dragonfly-client-storage/src/storage_engine/rocksdb.rs, and dragonfly-client-storage/src/metadata.rs. Associated tests were also renamed and updated to use the new method name. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

Persistent Task ID Generation Enhancements

• Introduced a new PersistentTaskIDParameter enum and a corresponding persistent_task_id method to the IDGenerator, enabling flexible persistent task ID creation based on content or file metadata. Comprehensive unit tests were added to verify correct ID generation for different parameter combinations. [1] [2] [3]

gRPC Server Refactoring

• Updated the DfdaemonDownloadServer and its handler to include and initialize a persistent_task manager, ensuring support for persistent task operations in the download server. [1] [2] [3] [4] [5] [6] [7]

Related Issue

Motivation and Context

Screenshots (if appropriate)

[codecov]

Codecov Report

❌ Patch coverage is 37.90614% with 172 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.95%. Comparing base (b0f9e57) to head (38d9c33).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dragonfly-client/src/grpc/dfdaemon_download.rs	0.00%	98 Missing ⚠️
dragonfly-client/src/resource/persistent_task.rs	0.00%	73 Missing ⚠️
dragonfly-client/src/bin/dfdaemon/main.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1551      +/-   ##
==========================================
- Coverage   46.08%   45.95%   -0.14%     
==========================================
  Files          71       71              
  Lines       19562    19829     +267     
==========================================
+ Hits         9016     9112      +96     
- Misses      10546    10717     +171     

Files with missing lines	Coverage Δ
dragonfly-client-config/src/dfdaemon.rs	92.61% <100.00%> (+0.04%)	⬆️
dragonfly-client-storage/src/metadata.rs	56.94% <ø> (ø)
dragonfly-client-storage/src/storage_engine/mod.rs	100.00% <ø> (ø)
...onfly-client-storage/src/storage_engine/rocksdb.rs	97.41% <100.00%> (ø)
dragonfly-client-util/src/id_generator/mod.rs	98.73% <100.00%> (+0.37%)	⬆️
dragonfly-client/src/bin/dfdaemon/main.rs	0.00% <0.00%> (ø)
dragonfly-client/src/resource/persistent_task.rs	0.00% <0.00%> (ø)
dragonfly-client/src/grpc/dfdaemon_download.rs	2.99% <0.00%> (-0.28%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chlins]

lgtm