Newest patch version is not a patch version

[pvanderlinden]

We used to use tag: v0.12-alpine-elasticsearch
As an emergency fix I had to rollback to the last known good version which was fluent/fluentd-kubernetes-daemonset@sha256:fe67be752f17dd4a66b0c88a46b1d937bff1fa9bd653c5be18880a6b744744cb

When the image was updated because of a node upgrade, everything stopped working. After looking it up this "patch" version changed running as root to non-root, causing all kind of errors. The new version with the work around doesn't parse JSON anymore though and leaves the original message as is.
The issues:

• json is not parsed anymore
• Access denied to /var/log unless FLUENT_UID is set to 0 #173
• Lot of "error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied" messages with new docker image version of fluentd-kubernetes-deamonset:elasticsearch #172

[pkeuter]

Same here. This broke our cluster all of a sudden. I've managed to get the logging to work again by setting FLUENT_UID, but json-parsing is still not working.

I've tried appending this to the config but this will break all parsing:

<filter kubernetes.var.log.containers.**>
      @type parser
      format json
      replace_invalid_sequence true
      suppress_parse_error_log false
      emit_invalid_record_to_error false
      key_name log
      reserve_data true
    </filter>

I've also tried using the 1.2 version of the Docker image, but this pod gets into a crash loop because it is unable to access journalctl logs. I'm not really sure what is going on there.

If there is someone who can help out with the parsing configuration so that the JSON will be parsed again, that would be amazing!

[pvanderlinden]

@pkeuter I haven't managed to have time to look for a permanent solution (like upgrading, and figuring out the new configuration). I have resorted to rollback to the last known good version, unfortunately without a tag: the image we currently use is fluent/fluentd-kubernetes-daemonset@sha256:fe67be752f17dd4a66b0c88a46b1d937bff1fa9bd653c5be18880a6b744744cb

[pkeuter]

Hey @pvanderlinden, thanks for the message! I saw that in your previous post indeed, but it's good to know that you haven't been able to find a definitive solution yet. I was hoping that maybe @repeatedly (or another maintainer) would be able to point us in the right direction.

Using an untagged version is a good workaround, but I'd rather try to "fix" it.

[mbolek]

Most of the issues you mentioned are due to the fact that fluentd is no longer running as root which means it cannot access /var/log nor create pos_file there.
As for the JSON I have (this is for the 1.2-debian container):

    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

• (I'm currently using multiparse plugin)

    <filter kubernetes.var.log.containers.**.log>                                                                                                      
      @type parser                                                                                                                                     
      key_name log                                                                                                                                     
      reserve_data true                                                                                                                                
      <parse>                                                                                                           
        @type multi_format                                                                                              
        <pattern>                                                                                                       
          format json                                                                                                   
          time_format %Y-%m-%dT%H:%M:%SZ                                                                                
          time_key time                                                                                                 
        </pattern>                                                                                                      
        <pattern>                                                                                                       
          format none                                                                                                   
        </pattern>                                                                                                      
      </parse>                                                                                                          
    </filter>   

and it is parsed properly.

[whereisaaron]

I agree @pvanderlinden this was in no way a patch level change. It caused us a lot of grief, and real loss of trust in the project 😞

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days