-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plugin server ingestion base #2732
Conversation
Looks good to me! Just in case get buy-in from @fuziontech that nothing else has to be done to emit to a new kafka topic. |
There's also one failing test |
The nice thing about Kafka is that this is not true. Messages are persistent until they TTL out. You should be able to consume from WAL no problem |
Changes
This is some base changes to the Django server, that'll allow us to get started with moving ingestion to the Node plugin server.
All EE events get logged to a new Kafka topicevents_handoff
, from which the plugin server consumes them (we can't useevents_write_ahead_log
as consuming messages deletes them, and in the case of a log we want to keep them just in case).No new topic, as turns out we can reuse the WAL topic. So all EE events will get logged to the existing topic. Removed
LOG_TO_WAL
setting.