fluent-bit installed via helm chart on AWS EKS, annotations seem not working

[jdetroy]

Hello,

I'm pretty new to the fluent-bit community, I've only recently started using it.
I'm running Fluent-bit 1.9.9 on AWS EKS and I'm experimenting with setting the parser via annotations on the Pods.
I'm deploying fluent-bit via the eks blueprint Terraform module

My fluent-bit configuration is below

[SERVICE]
Parsers_File /fluent-bit/parsers/parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/.log
DB /var/log/flb_kube.db
Parser cri
Docker_Mode On
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
Merge_Log_Key data
Keep_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Buffer_Size 32k
[OUTPUT]
Name opensearch
Match *
Host

In my pods I have e.g.

apiVersion: v1
kind: Pod
metadata:
annotations:
fluentbit.io/parser: docker

The logs get initially parsed with the cri parser, the fields: stream, log_tag and message field gets populated correctly.

The value message field is e.g.
"{"t":{"$date":"2023-06-06T16:04:33.266+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1686067473:266418][1:0x7f50d6599700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 11698, snapshot max: 11698 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 7"}}",

Would using the annotation above that not get split up in different fields? Or does the parser set via the annotation not look at the message field, but at the full 'logline' instead?

In the parser there is:
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On

I've tried both json and docker in the annotation

Should I be able to see where it's going wrong in debug or trace logs of fluent bit? Or does someone immediately see what I'm doing wrong?
I could really use some help from experienced fluent-bit users.

Best Regards,
Jo