New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Lumberjack protocol v2 #279

Closed
trixpan opened this Issue Nov 7, 2015 · 9 comments

Comments

Projects
None yet
6 participants
@trixpan

trixpan commented Nov 7, 2015

https://github.com/elastic/libbeat/blob/master/outputs/logstash/client.go#L48 seems to suggest libbeat implements lumberjack protocol v2, however all documentation around (e.g. https://github.com/logstash-plugins/logstash-input-beats/blame/master/PROTOCOL.md#L12 seems to document only version 1)

Given most in the community expect the lumberjack input plugin to decay with time (as the community and customer base move to beats) It would be great to document the new version of the protocol.

@trixpan

This comment has been minimized.

Show comment
Hide comment
@trixpan

trixpan Nov 7, 2015

e.g. seems like the json frame type has been added in v2 (as it is absent from logstash-forwarder protocol documentation )

trixpan commented Nov 7, 2015

e.g. seems like the json frame type has been added in v2 (as it is absent from logstash-forwarder protocol documentation )

@ondrap

This comment has been minimized.

Show comment
Hide comment
@ondrap

ondrap Feb 8, 2016

There seem to be some rather undocumented behaviour

  • according to some source (I think logstash input plugin?) lumberjack v1 does not support partial acks
  • it seems to me that the 'window' is actually not a window; the behaviour is rather "I will send you X messages, please ack immediately when you get them". Which prompts me to ask:
    • Shouldn't the description rather be "the sender will block until the Xth message is acked?
    • Does the sender really allow for partial acks? I.e. does it make sense to ack after x/2 or is it a wasto of bandwidth?

ondrap commented Feb 8, 2016

There seem to be some rather undocumented behaviour

  • according to some source (I think logstash input plugin?) lumberjack v1 does not support partial acks
  • it seems to me that the 'window' is actually not a window; the behaviour is rather "I will send you X messages, please ack immediately when you get them". Which prompts me to ask:
    • Shouldn't the description rather be "the sender will block until the Xth message is acked?
    • Does the sender really allow for partial acks? I.e. does it make sense to ack after x/2 or is it a wasto of bandwidth?
@trixpan

This comment has been minimized.

Show comment
Hide comment
@trixpan

trixpan Feb 12, 2016

any updates on this?

trixpan commented Feb 12, 2016

any updates on this?

@asafm

This comment has been minimized.

Show comment
Hide comment
@asafm

asafm Mar 7, 2016

Need this as well.

asafm commented Mar 7, 2016

Need this as well.

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Mar 7, 2016

lumberjack v1 does not support partial acks

The protocol itself supports acks up to a given sequence number; so partial acks are possible.

it seems to me that the 'window' is actually not a window

The original design mimics TCP, which is where the window term comes from. It is a window, but the server can choose to ack only once a full window is received. We are still experimenting with the best throughput options here (partial acks vs full acks, but both should be supported).


If you're asking about protocol details, we can document them as time permits, but note that the protocol is evolving and we don't guarantee backwards compatibility or upgrade paths that we don't maintain (beats + logstash are the only supported client+servers).

In terms of what changed in "v2". We can update the protocol docs, but the short version is there is now a 2J frame that ships JSON as a payload.

jordansissel commented Mar 7, 2016

lumberjack v1 does not support partial acks

The protocol itself supports acks up to a given sequence number; so partial acks are possible.

it seems to me that the 'window' is actually not a window

The original design mimics TCP, which is where the window term comes from. It is a window, but the server can choose to ack only once a full window is received. We are still experimenting with the best throughput options here (partial acks vs full acks, but both should be supported).


If you're asking about protocol details, we can document them as time permits, but note that the protocol is evolving and we don't guarantee backwards compatibility or upgrade paths that we don't maintain (beats + logstash are the only supported client+servers).

In terms of what changed in "v2". We can update the protocol docs, but the short version is there is now a 2J frame that ships JSON as a payload.

@ondrap

This comment has been minimized.

Show comment
Hide comment
@ondrap

ondrap Mar 7, 2016

The original design mimics TCP

Actually, it seems to me that it does not. It seems to me that it works as follows:

  1. Send window with size N
  2. Send N items of data, sequence numbers start from 1
  3. Wait for ACK
  4. Repeat

I'd expect the sequence numbers increment indefinitely, not to reset after every window change.

ondrap commented Mar 7, 2016

The original design mimics TCP

Actually, it seems to me that it does not. It seems to me that it works as follows:

  1. Send window with size N
  2. Send N items of data, sequence numbers start from 1
  3. Wait for ACK
  4. Repeat

I'd expect the sequence numbers increment indefinitely, not to reset after every window change.

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Mar 7, 2016

As a short term solution, until we refresh the documentation, I could delete the PROTOCOL.md as many of you find it inaccurate. Let me know.

jordansissel commented Mar 7, 2016

As a short term solution, until we refresh the documentation, I could delete the PROTOCOL.md as many of you find it inaccurate. Let me know.

@dedemorton

This comment has been minimized.

Show comment
Hide comment
@dedemorton

dedemorton Feb 14, 2018

Contributor

@jordansissel Can we close this?

Contributor

dedemorton commented Feb 14, 2018

@jordansissel Can we close this?

@dedemorton dedemorton removed their assignment Feb 14, 2018

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Feb 14, 2018

I think so. The lumberjack protocol is not intended at this time for implementation by any other systems, so documenting it is out of scope.

jordansissel commented Feb 14, 2018

I think so. The lumberjack protocol is not intended at this time for implementation by any other systems, so documenting it is out of scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment