Fluent source ack wrong format

[ChezBunch]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

As mentioned in #7533 (comment), the ACK format returned by the fluent source is not understood by fluent-bit, and generates an error:
[error] [output:forward:forward.0] ACK response not MAP (type:2)

Indeed, when the ack was implemented, #10176 mentioned that the ack format was not very clear, and was implemented as plain JSON, while in fact fluent-bit expects a msgpack format.

The issue as initially described in #7533 (comment):

Hello. Yes, I have some problems.
I using Vector as an aggregator and fluent-bit as a log collector, but in fluent-bit logs, I see errors like this:

[error] [engine] chunk '96-1683047707.895527211.flb' cannot be retried: task_id=0, input=tail.0 > output=forward.0
[error] [output:forward:forward.0] ACK response not MAP (type:2)

Vector deployed via Helm chart version: 0.20.1 with these options:

customconfig:   
  acknowledgements:
      enabled: true
image:
  tag: "0.29.1-distroless-libc"

Vector configuration is:

vector:
  address: 0.0.0.0:9000
  type: fluent
  receive_buffer_bytes: 1048576
  connection_limit: 2000
  keepalive:
     time_secs: 30

fluent-operator deployed via Helm chart 2.2.0, also tried 2.1.0

fluent-bit:
  image:      
    repository: "kubesphere/fluent-bit"
    tag: "v2.0.11"
  forward:    
    host: vector.fluent.svc    
    port: 9000    
    requireAckResponse: true

If you need more details, I can provide it.
Thank you

Configuration

No response

Version

0.30.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

#10176
#7533

[neuronull]

Just noting that I also confirmed this behavior with fluentd

2023-05-18 17:32:45 +0000 [trace]: #0 dequeueing a chunk instance=2220
2023-05-18 17:32:45 +0000 [trace]: #0 chunk dequeued instance=2220 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="hi", variables=nil, seq=0>
2023-05-18 17:32:45 +0000 [trace]: #0 trying flush for a chunk chunk="5fbfb2d8e62247804568afd73ee62022"
2023-05-18 17:32:45 +0000 [trace]: #0 executing delayed write and commit chunk="5fbfb2d8e62247804568afd73ee62022"
2023-05-18 17:32:45 +0000 [trace]: #0 writing a chunk to destination chunk_id="5fbfb2d8e62247804568afd73ee62022"
2023-05-18 17:32:45 +0000 [debug]: #0 connect new socket
2023-05-18 17:32:45 +0000 [trace]: #0 getting response from destination host="0.0.0.0" port=24224 chunk_id="5fbfb2d8e62247804568afd73ee62022" response=123
2023-05-18 17:32:45 +0000 [error]: #0 unexpected error while receiving ack message error_class=TypeError error="no implicit conversion of String into Integer"
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward/ack_handler.rb:141:in `[]'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward/ack_handler.rb:141:in `read_ack_from_sock'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward/ack_handler.rb:71:in `block in collect_response'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward/ack_handler.rb:70:in `each'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward/ack_handler.rb:70:in `collect_response'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward.rb:517:in `ack_check'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_forward.rb:512:in `ack_reader'
  2023-05-18 17:32:45 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[jszwedko]

Closed by #17407