Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

protobuf messages in the length-delimited format #65

Closed
bad3bs opened this issue Mar 17, 2020 · 2 comments
Closed

protobuf messages in the length-delimited format #65

bad3bs opened this issue Mar 17, 2020 · 2 comments
Assignees
@lspgn

Comments

@bad3bs
Copy link

bad3bs commented Mar 17, 2020

It's will be awesome to add option to enable protobuf messages in the length-delimited format. That for native support clickhouse protobuf input (https://clickhouse.tech/docs/en/interfaces/formats/#protobuf).
@lspgn lspgn self-assigned this Mar 22, 2020
lspgn added a commit that referenced this issue Mar 22, 2020
@lspgn
Length-delimited protobuf #65
edcb2b6
@lspgn
Copy link
Contributor

lspgn commented Mar 22, 2020

This is a good idea.
Attached a PR.
I'm not sure the protobuf can be mapped 1:1 within Clickhouse (example: storing src/dst IP addresses as bytes instead of two columns being a UInt32 and a FixedString(16)).
But I tested locally (fed from a Kafka), it works.

lspgn added a commit that referenced this issue Mar 22, 2020
@lspgn
Merge pull request #67 from cloudflare/feature/fixed-length-proto
8b0a2cc 
Length-delimited protobuf #65
@lspgn
Copy link
Contributor

lspgn commented Mar 22, 2020

Added the functionality. I also updated flow-pipeline to provide a functioning example with Clickhouse.

@lspgn lspgn closed this as completed Mar 22, 2020
tgragnato pushed a commit to tgragnato/goflow that referenced this issue Aug 14, 2024
@lspgn
Merge pull request cloudflare#65 from netsampler/bugfix/offset
20e8e56 
Bugfix: index out of range in some cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lspgn lspgn

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lspgn @bad3bs