Opensearch rejects records because of datatype mismatch

[sdwerwed]

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

Create 2 logs with diffrent format ex: ts=1.6530758923877115E9 and ts=2022-05-20T20:38:47.298361917Z

Expected Behavior or What you need to ask

I would expect to not lose logs because of luck.
In my case this happened in AKS, one managed service after upgrade created timestamp in diffrent format and then some logs were missing, this had serious impact because logs got lost and we did not know about this issue. I would like to not lose logs if this happen again. If I know the problematic key/value I can handle it with transformation or drop it, but what if this happens frequently with other key/values?

Using Fluentd and OpenSearch plugin versions

Dynamic datatype has conflicts, looks like different apps use different formats for the key ts and get datatype that is not supported by all microservices.

OpenSearch rejects records from app1 in the dev environment:
error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [ts] of type [date] in document with id 'mHYA44ABjc1sEmR6h9ze'. Preview of field's value: '1.6530758923877115E9''"

OpenSearch rejects record from app2 in qa environment:
error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [ts] of type [float] in document with id 'Y2Mx44ABx8DDlswJ0RGZ'. Preview of field's value: '2022-05-20T20:38:47.298361917Z''"

It is running in AKS
Version and plugins:

I have no name!@fluentd-0:/opt/bitnami/fluentd$ fluentd --version
fluentd 1.14.4
I have no name!@fluentd-0:/opt/bitnami/fluentd$ fluent-gem list

activesupport (7.0.1)
addressable (2.8.0)
aws-eventstream (1.2.0)
aws-partitions (1.547.0)
aws-sdk-core (3.125.2)
aws-sdk-kms (1.53.0)
aws-sdk-s3 (1.111.1)
aws-sdk-sqs (1.49.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (2.3.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.9)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
domain_name (0.5.20190701)
elasticsearch (7.16.1)
elasticsearch-api (7.16.1)
elasticsearch-transport (7.16.1)
elasticsearch-xpack (7.16.1)
etc (default: 1.1.0)
excon (0.89.0)
faraday (1.9.3)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.5)
ffi-compiler (1.0.1)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-dedot_filter (1.0.0)
fluent-plugin-detect-exceptions (0.0.14)
fluent-plugin-elasticsearch (5.1.4)
fluent-plugin-grafana-loki (1.2.16)
fluent-plugin-kafka (0.17.3)
fluent-plugin-kubernetes_metadata_filter (2.9.4)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.0)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.6.1)
fluent-plugin-systemd (1.0.5)
fluentd (1.14.4)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.4)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
i18n (1.8.11)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.5.0)
json (default: 2.3.0, 2.1.0)
jsonpath (1.1.0)
kubeclient (4.9.2)
logger (default: 1.4.2)
lru_redux (1.1.0)
ltsv (0.1.2)
matrix (default: 0.2.0)
mime-types (3.4.1)
mime-types-data (3.2022.0105)
minitest (5.15.0, 5.13.0)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
netrc (0.11.0)
observer (default: 0.1.0)
oj (3.3.10)
open3 (default: 0.1.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (2.1.0)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
racc (default: 1.4.16)
rake (13.0.6, 13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
recursive-open-struct (1.1.3)
reline (default: 0.1.5)
rest-client (2.1.0)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.4.0)
ruby2_keywords (0.0.5)
rubygems-update (3.3.4)
sdbm (default: 1.0.0)
serverengine (2.2.4)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
test-unit (3.3.4)
timeout (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2021.5)
unf (0.1.4)
unf_ext (0.0.8)
uri (default: 0.10.0)
webrick (1.7.0, default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.1)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

[github-actions]

@sdwerwed this issue was automatically closed because it did not follow the issue template.

[sdwerwed]

cosmo0920 The bot closed the issue directly, I have updated the description, can we open it? If there is no solution for this we can suggest a workaround and update the FAQ documentation