Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin server ingestion base #2732

Merged
merged 8 commits into from
Dec 14, 2020
Merged

Plugin server ingestion base #2732

merged 8 commits into from
Dec 14, 2020

Conversation

Twixes
Copy link
Collaborator

@Twixes Twixes commented Dec 10, 2020

Changes

This is some base changes to the Django server, that'll allow us to get started with moving ingestion to the Node plugin server.

All EE events get logged to a new Kafka topic events_handoff, from which the plugin server consumes them (we can't use events_write_ahead_log as consuming messages deletes them, and in the case of a log we want to keep them just in case).
No new topic, as turns out we can reuse the WAL topic. So all EE events will get logged to the existing topic. Removed LOG_TO_WAL setting.

@timgl timgl temporarily deployed to posthog-temporary-plugi-jvxnlq December 10, 2020 15:42 Inactive
@mariusandra
Copy link
Collaborator

Looks good to me! Just in case get buy-in from @fuziontech that nothing else has to be done to emit to a new kafka topic.

@mariusandra
Copy link
Collaborator

There's also one failing test

@Twixes Twixes temporarily deployed to posthog-temporary-plugi-jvxnlq December 10, 2020 16:02 Inactive
@fuziontech
Copy link
Member

fuziontech commented Dec 10, 2020

(we can't use events_write_ahead_log as consuming messages deletes them, and in the case of a log we want to keep them just in case)

The nice thing about Kafka is that this is not true. Messages are persistent until they TTL out. You should be able to consume from WAL no problem

@Twixes Twixes temporarily deployed to posthog-temporary-plugi-jvxnlq December 10, 2020 22:31 Inactive
@Twixes Twixes temporarily deployed to posthog-temporary-plugi-jvxnlq December 11, 2020 11:01 Inactive
@Twixes Twixes temporarily deployed to posthog-temporary-plugi-jvxnlq December 11, 2020 13:53 Inactive
@Twixes Twixes merged commit 7ba9f7d into master Dec 14, 2020
@Twixes Twixes deleted the temporary-plugin-processing branch December 14, 2020 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants