external Go service returns HTTP Status 400

[jsarkar]

Bug Report

Describe the bug
When I am trying to test my output HTTP plugin in failure scenario (moderate load) I saw following error fluent-bit log:

[2020/04/24 18:25:09] [ warn] [engine] failed to flush chunk '1-1587752693.632712160.flb', retry in 64 seconds: task_id=726, input=tail.0 > output=http.0
[2020/04/24 18:25:11] [ warn] [engine] failed to flush chunk '1-1587752691.241037672.flb', retry in 34 seconds: task_id=695, input=tail.0 > output=http.0
[2020/04/24 18:25:11] [error] [out_http] hostname.domain.com:443, HTTP status=400

I am using fluent-bit 1.3.11 container in kubernetes
To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

Sample 1:
log="2020-04-24T18:34:21.732404578Z stdout F 57.169.194.71 - - [24/Apr/2020:18:34:21 +0000] "GET /amplicative HTTP/1.0" 204 2216"
Sample 2:
log="2020-04-24T18:35:07.862616565Z stdout F 2020-04-24T18:35:07+0000 WARN A warning that should be ignored is usually at this level and should be actionable."
Sample 3:
log="2020-04-24T18:35:28.969774421Z stdout F {"@timestamp": "2020-04-24T18:35:28+0000", "level": "INFO", "message": "takes the value and converts it to string."}"

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used:1.3.11
• Configuration: Kubernetes
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes v1.14.10
• Server type and version:
• Operating System and version: Ubuntu 18.04.4 LTS
• Filters and plugins: Kubernetes in and http out plugin with record_modifier, nest/list, modify

Additional context

@edsiper Could please elaborate the failure mechanism of fluent-bit, it will be really helpful for me.

[edsiper]

It's important to understand the nature of the error messages. The HTTP error status 400 means Bad Request, so the payload being sent by Fluent Bit is not being accepted by the remote service.

Would you please share your config map here? (formatted way pls)

[jsarkar]

@edsiper Here is the configmap for fluentbit

apiVersion: v1
kind: ConfigMap
metadata:
    name: fluent-bit-config
    namespace: logging
    labels:
        k8s-app: fluent-bit
data:
    # Configuration files: server, input, filters and output
    # ======================================================
    fluent-bit.conf: |
        [SERVICE]
            Flush         1
            Log_Level     info
            Daemon        off
            Parsers_File  parsers.conf
            HTTP_Server   On
            HTTP_Listen   0.0.0.0
            HTTP_Port     2020

        @INCLUDE input-kubernetes.conf
        @INCLUDE filter-kubernetes.conf
        @INCLUDE output-http.conf

    input-kubernetes.conf: |
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              /var/log/containers/*.log
            Parser            docker
            DB                /var/log/flb_kube.db
            Mem_Buf_Limit     20MB
            Buffer_Chunk_Size 5MB
            Buffer_Max_Size   5MB
            Skip_Long_Lines   On
            Refresh_Interval  10

    filter-kubernetes.conf: |
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc:443
            Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
            Kube_Tag_Prefix     kube.var.log.containers.
            Merge_Log           On
            Merge_Log_Key       log_processed
            Merge_Log_Trim      On
            K8S-Logging.Parser  On
            K8S-Logging.Exclude On
            Annotations         Off
        [FILTER]
            Name                record_modifier
            Match               *
            Record              fluent-bit-host ${HOSTNAME}
        [FILTER]
            Name                nest
            Match               *
            Wildcard            pod_name
            Operation           lift
            Nested_under        kubernetes
            Add_prefix          kubernetes_
        [FILTER]
            Name                modify
            Match               *
            Copy                kubernetes_pod_name host
        [FILTER]
            Name                nest
            Match               *
            Wildcard            customer_index_name
            Operation           lift
            Nested_under        kubernetes_labels
        [FILTER]
            Name                modify
            Match               *
            Copy                kubernetes_customer_index_name customer_index_name

    output-http.conf: |
        [OUTPUT]
            Name            http
            Match           *
            Host            hostname.domain.com
            Port            443
            tls             On
            tls.verify      off
            json_date_key   timestamp
            json_date_format iso8601
            URI             /shardRouter/putRecord
            Format          json_stream

    parsers.conf: |
        [PARSER]
            Name   apache
            Format regex
            Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z

        [PARSER]
            Name   apache2
            Format regex
            Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z

        [PARSER]
            Name   apache_error
            Format regex
            Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

        [PARSER]
            Name   nginx
            Format regex
            Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z

        [PARSER]
            Name   json
            Format json
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z

        [PARSER]
            Name        docker
            Format      json
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L
            # Time_Keep   On
            # Testing removing the time_keep to have a record injected at the top level
            # Command      |  Decoder | Field | Optional Action
            # =============|==================|=================
            # Decode_Field_As   escaped    log

        [PARSER]
            Name        syslog
            Format      regex
            Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
            Time_Key    time
            Time_Format %b %d %H:%M:%S

[edsiper]

what kind of service is running on hostname.domain.com ?

[jsarkar]

It's go based service which does some validation logic (e.g. looking for a correct index name ) before it forward to Kinesis stream

[edsiper]

so the error status is because your Go service is likely not accepting our payload format.

[jsarkar]

You are correct. The api is looking for customer_index_name and rejecting the messages with having the same key