Client Reliability: Prevent Ignoring Server Errors, Duplicate Writes, and Dirty Connections

[JoshuaChi]

Description:

The d-engine-client module must be improved to handle server errors safely and avoid unsafe behaviors such as:

• Ignoring server errors
• Retrying requests that could cause duplicate writes
• Using stale or dirty connections

---

Acceptance Criteria:

1. Server Error Handling

• Client must detect server errors (unavailable, overloaded, or explicit error)
• Client propagates the error back to the caller instead of ignoring it

2. Idempotent / Duplicate-Safe Writes

• Retry of mutable operations preserves at-most-once semantics
• No duplicate writes occur during retries

3. Failover Behavior

• Client automatically fails over to healthy nodes if one becomes unavailable
• Caller is clearly notified of failed nodes