Skip to content

NIFI-8843: ZooKeeper leader election for NiFi Registry HA#11046

Closed
prabhjyotsingh wants to merge 1 commit intoapache:mainfrom
prabhjyotsingh:NIFI-8843
Closed

NIFI-8843: ZooKeeper leader election for NiFi Registry HA#11046
prabhjyotsingh wants to merge 1 commit intoapache:mainfrom
prabhjyotsingh:NIFI-8843

Conversation

@prabhjyotsingh
Copy link
Copy Markdown

@prabhjyotsingh prabhjyotsingh commented Mar 26, 2026

NiFi Registry previously supported only single-node deployments. This PR introduces full HA support — leader election, write replication, cache coherency, durable event delivery, bootstrap DB sync, maintenance mode, and cluster health
observability.

Two coordination backends are selectable via nifi.registry.cluster.coordination:

  • zookeeper — Curator LeaderSelector, ZNode push-based cache invalidation, per-node local H2
  • database — DB TTL lease, CACHE_VERSION polling, shared PostgreSQL / MySQL

When nifi.registry.cluster.enabled=false (the default) all new code paths are inactive — single-node behaviour is unchanged.

Full design, component map, flow diagrams, and deployment guide: nifi-registry/NIFI-8843-HA-Design.md


Key Implementation Points

  • Follower writes return 307 Temporary Redirect to the leader (not a transparent proxy) so mTLS client identity is preserved end-to-end
  • Leader fan-out is asynchronous via X-Registry-Replication + X-Registry-Internal-Auth headers; constant-time token comparison (MessageDigest.isEqual) prevents timing attacks
  • Bootstrap sync uses H2's native SCRIPT/RUNSCRIPT — no external tooling required
  • CuratorFramework bean is @lazy — never instantiated in standalone/DB mode even with a blank connect string
  • New Flyway migration V9 adds CACHE_VERSION, CLUSTER_LEADER, and REGISTRY_EVENT tables (H2, MySQL, PostgreSQL)

ZooKeeper cache invalidation and event delivery
HTTP write replication for local-H2 HA
ZK TLS, bootstrap sync, and cluster health indicator
@pvillard31
Copy link
Copy Markdown
Contributor

@prabhjyotsingh - thanks for the contribution but, at the moment, NiFi Registry is marked as deprecated (as you can see at https://nifi.apache.org/projects/registry/) and we are not going to accept any contribution, especially of this magnitude. As it was discussed on the mailing lists, we may revert the deprecation if core issues are addressed first (https://lists.apache.org/thread/3gs4nl28yngsdmxd6xglzmbh5jqbscl3).

@pvillard31 pvillard31 closed this Mar 26, 2026
@prabhjyotsingh
Copy link
Copy Markdown
Author

@pvillard31 Thanks for the quick response and for keeping me in the loop!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants