Es plugin as output for Humio stopped working after updating fluent bit version 1.8.4

[surajthakur]

Hi Team,

We updated fluent-bit version from 1.7.4 to 1.8.4( latest coming from apt repo). After the update the agent start giving error

[2021/08/23 17:05:28] [ warn] [engine] failed to flush chunk '9909-1629702310.123277989.flb', retry in 9 seconds: task_id=0, input=storage_backlog.2 > output=es.0 (out_id=0)
[2021/08/23 17:05:37] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response:
Humio can only handle index actions for the elastic search api.

We are running OS - Debian 9

The fluentbit config: output plugin config is as below

[OUTPUT]
        Name        es
        Match       *
        Host        humio-ip
        Port        9200
        Index       <humio_repo_name>
        HTTP_User   <ingest-key>

Any ideas what could be wrong ?
We have rolled back to 1.7.4 and now it's working fine.
It was hard to find old version of fluentbit in apt repo neither the documentation gives any link of direct download.
From some github issue i was able to get https://packages.fluentbit.io/debian/stretch/pool/main/t/td-agent-bit/td-agent-bit_1.7.4_amd64.deb
Is there any docs for correct direct download link and format of the url

Thanks

[joenzx]

I have encountered this error, it is a problem with es, check whether your es cluster is normal @surajthakur

[surajthakur]

I have encountered this error, it is a problem with es, check whether your es cluster is normal @surajthakur

@joenzx , but how is it working fine by downgrading the version ?

The cluster is fine, but i see that we have reached 85% disk, but that should not be an issue there

[Crevil]

We see the same. Upgrading from 1.7.9 to 1.8.0 breaks ingestion in Humio. (Same with versions up until 1.8.4)

The only change I can see that might be related is #3788 which touches the es output.

[nokute78]

Is it related to this patch #2026 7f0db9e ?
The patch was merged from v1.8.0.

Error is Humio can only handle index actions for the elastic search api.
The above patch is to use create instead of index.
https://github.com/fluent/fluent-bit/pull/2026/files#diff-3fc4adb0ddceaceddc7686c08d19a9ea90c5cf3a79fb0e1c14a3d14bc64a6257

Note: #2664

[Crevil]

That indeed looks to be the reason.

The reasoning for making that change makes great sense I think, so I guess we should direct this issue to Humio and their compatible elasticsearch API instead. I'll open an issue with them to get their input.

[Crevil]

I have spoken to Humio support and they have provided the following answer. Slightly redacted.

We've gone ahead and created a feature request to let our engineering team know about this issue with ingesting data from the elasticsearch output plugin on fluent-bit 1.8+. I've also notified them that this is a priority for you as it is preventing fluent-bit upgrades. If you're able to keep an eye on our Release Notes page, any updates to this issue will be shown there:
https://library.humio.com/notices/releases/release-notes/#

One thing I did want to mention is that it might be possible to downgrade the elasticsearch plugin itself after upgrading fluent-bit. I'm not positive how that would work but it may be worth looking into as a workaround in the meantime. I'll go ahead and close this ticket out for now but feel free to reach out if you have any further questions!

I'm unsure about their proposal to downgrade the elasticsearch plugin. Does any one know if that is doable at all?

[surajthakur]

Thanks for raising a case to the humio support. I will keep watching their release notes. I should have provided humio related details before but maybe some it's helpful.
We are running humio v1.26.1

"version" : {
    "number" : "6.2.0",
    "build_hash" : "Humio-1.26.1--build-***",
    "build_date" : "2000-01-01T00:00:00.000000Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  }

You can close the ticket, if you like. I will keep watching humio updates.

[surajthakur]

@Crevil, From some github issue i was able to get https://packages.fluentbit.io/debian/stretch/pool/main/t/td-agent-bit/td-agent-bit_1.7.4_amd64.deb
Is there any docs for correct direct download link and general format of the url ?

Thanks

[Crevil]

Just got another update from Humio

Good news! I heard back on the feature request and it looks like they've already come up with a potential solution for this.
I can't give an exact timeline on when that would be released since they'll likely want to do some testing/verification but I wouldn't expect it to be too far out. (...)

We are running the same version of Humio currently.

I cannot help with the download URLs as we are using the Docker images of fluent-bit.

[surajthakur]

Thanks for the update @Crevil , appreciate that.
We were also running using the docker images, but we also wanted to use fluenbit to collect the host based logs. So we had to install this on host as td-agent.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[Crevil]

This should be fixed in Humio 1.31.0. We have yet to test it out though.

[surajthakur]

Thanks for the update @Crevil. Looking forward to know your results.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.