-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ feat: enable separate producer and consumer kafka clusters #483
Conversation
…d add environment variables documentation to readmes
…e to include the --project-directory argument otherwise compose default to using the directory of the first compose file as the default work directory and cannot find files that may be attach with volume mappings from infra folders
Wow, that's amazing. Thanks for your contribution ❤️ |
Yeah, we usually clear the local storage in this case. The reason is that we store organization and project information in the local storage, which is then used to load the UI. In this situation, the information becomes outdated (from the previous run), so the UI fails to render. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR enables Featbit to function with Highly Available, Cross Data Center/Cross Region Kafka Clusters by allowing consumers to consume from different clusters than producers.
The scenario in which this is useful is a deployment that will use a Primary Cluster and a Secondary Aggregate Cluster, with Mirror Maker 2 synchronizing data from the local Primary to the local Secondary. All Producers will write to the local Primary and all Consumers will read from the local Aggregate. Additionally, Mirror Maker would perform one way synchronization between the local Primary Cluster and the Remote Aggregate Cluster. This keeps the Kafka Aggregate (or "consumer") clusters synchronized across data centers in near real-time, and keeps Clickhouse synchronized between data centers without the need for additional cross data center replication.
Testing Backward Compatibility:
--build
flag so the containers are using the new code.api
,ch_group
, andevaluation-server
.Testing New Functionality:
featbit-kafka
clusterfeatbit-kafka-consumer
clusterapi
,ch_group
, andevaluation-server
featbit-kafka
cluster, and thefeatbit-kafka-consumer
cluster.Note: It may be necessary to close your browser entirely and reopen it between testing docker-compose-pro-dev.yml and docker-compose-pro-dev-HA-kafka.yml otherwise the UI may fail to render. The cause was not investigated, but the assumption is that the browser is holding an invalid authentication cookie from the previous deployment.