Repository navigation
HA and zero-downtime deployments: roadmap for engine redundancy and worker replicas? #2312
Replies: 3 comments
That is excellent! We are more than happy to keep you updated on iii’s progress on the below as you start your roll out.
We have a fix planned for this, you can see the recent discussion in #2084. This is a high priority fix for us.
You're correct that this is how
This is not currently implemented but is already high on our priority list because we are also starting our development of iii Cloud which will need this functionality as well. We do have the
Answering all questions below to reflect current behavior / best practices, not post-HA behavior/best practices. Also skipping most of what was addressed above to keep it succinct for you. Happy to dive into anything further.
Not yet.
Yes, partially via
Workers keep running during an engine downtime so they can receive requests but not send them to the engine during this period. This does mean you can have the worker's business logic retry on timeout. Can you give me more information about your use case / needs? I could then give you specific recommendations if you'd like. For example for HTTP requests it would be a combination of cache control headers and a reverse proxy that can redeliver failed messages. Happy to discuss this further privately as well.
Sync and fire-and-forget are lost, streams lose their connections but their data persists if using a persistent adapter (Redis, file-based). Streams are also being phased out in favor of a simpler and more reliable implementation.
Anything placed in a queue is automatically recovered and retried according to the policy of the queue that it's in. Any function call or trigger subscription (publisher side, subscriber side) can be placed in a queue.
Yes, high priority for us due to the aforementioned iii cloud plans.
These are great questions, it would be great to get your feedback as we implement this feature.
Job submission is directly handled by queues. Leases, checkpoints, and idempotency would be handled directly by applications themselves. Though some of this functionality (leases, checkpoints) could be generalized into their own reusable workers.
Roadmap
Use case specific, primarily application-side at the present moment
@guibeira is actively working on a fix to support this
Same answer
If the work was queued it's recoverable
|
|
Thank you so much for the comprehensive and insightful response! It's great to see that robust HA and rolling updates are high priorities on the roadmap, especially with the upcoming iii Cloud developments. |
|
You're welcome @jarvisaoieong please ping us if you have any further questions |
Uh oh!
There was an error while loading. Please reload this page.
Context
We are evaluating iii for a production system running long-lived AI agent workloads. We would like to understand the intended architecture and roadmap for high availability and zero-downtime upgrades, at both the worker and engine layers.
Based on the current documentation, we see two related challenges.
1. Worker replicas and rolling updates
Within a namespace, each function ID has a single owner. If another worker registers the same function, the engine rejects the new registration with
FUNCTION_NAMESPACE_CONFLICTand retains the existing owner.This seems to prevent a conventional rolling deployment where replacement workers become ready before existing workers stop.
Worker disconnection also removes its registrations and cancels in-flight invocations. The SDK’s
shutdown()appears to close the connection rather than drain active work first.We could implement blue-green namespaces and application-level routing, but this would require custom readiness checks, traffic switching, session ownership, and draining logic.
2. Engine availability and upgrades
The engine maintains the live worker/function registry and routes invocations. We have not found documentation describing a supported multi-engine HA topology or a zero-downtime engine upgrade procedure.
Our concern is that restarting or replacing a single engine would interrupt worker connections and potentially affect in-flight requests, even if the application workers remain healthy.
Simply putting multiple engines behind a load balancer does not appear sufficient unless function discovery and invocation routing work across engine instances.
Questions
Is there a supported or recommended production architecture for engine HA today?
What is the recommended procedure for upgrading the engine without service downtime?
Are worker replicas and graceful rolling deployments on the roadmap?
For long-running workloads, what guarantees should applications implement themselves?
Desired outcome
Ideally, we would be able to:
We recognize that uninterrupted existing connections, continuous request availability, and durable execution are different guarantees. It would be helpful to know which iii intends to support natively and which should remain application responsibilities.
Is this already possible through an architecture we have missed, or are there design discussions or roadmap items we could follow?
All reactions