Deserialization Failure

[clearscreen]

NEST/Elasticsearch.Net version: 7.8.1

Elasticsearch version: 7.8.0

Description of the problem including expected versus actual behavior: We're ingesting suricata eve logs with timestamps in the default suricata format: 2020-07-23T14:43:46.129022-0400. We are getting an exception (see below) when we try to deserialize in NEST.

Steps to reproduce:

1. Ingest a suricata log into elasticsearch
2. Attempt to deserialize the log. It will throw an exception complaining about the datetime format of the timestamp field.

Exception:

System.InvalidOperationException: invalid datetime format. value:2020-07-23T14:43:46.129022-0400
   at Elasticsearch.Net.Utf8Json.Formatters.ISO8601DateTimeFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)

Test Document:

{
	"_index": "suricata-2020.07.23",
	"_type": "_doc",
	"_id": "DSpHfXMBvOItDiFCmKJQ",
	"_version": 1,
	"_seq_no": 3756084,
	"_primary_term": 1,
	"found": true,
	"_source": {
		"flow": {
			"bytes_toserver": 60,
			"bytes_toclient": 0,
			"pkts_toclient": 0,
			"start": "2020-07-23T16:04:33.400720-0400",
			"pkts_toserver": 1
		},
		"@timestamp": "2020-07-23T20:04:34.363Z",
		"input": {
			"type": "log"
		},
		"tags": [
			"suricata",
			"beats_input_codec_plain_applied"
		],
		"stream": 0,
		"@version": "1",
		"timestamp": "2020-07-23T16:04:33.400720-0400",
		"payload": "",
		"ecs": {
			"version": "1.5.0"
		},
		"event_type": "alert"
	}
}

[russcam]

Hi @clearscreen,

The client expects a date format of yyyy-MM-ddThh:mm:ss.fffffff(Z|[+-]hh:mm),to be able to deserialize to a DateTime or DateTimeOffset - note the : between the hours and minutes offset.

Looking at ISO8601 however,

4.2.5.1 Difference between local time and UTC of day
When it is required to indicate the difference between local time and UTC of day, the representation of
the difference can be expressed in hours and minutes, or hours only. It shall be expressed as positive
(i.e. with the leading plus sign [+]) if the local time is ahead of or equal to UTC of day and as negative
(i.e. with the leading minus sign [-]) if it is behind UTC of day. The minutes time element of the
difference may only be omitted if the difference between the time scales is exactly an integral number
of hours.
Basic format:
±hhmm Example: +0100
±hh +01
Extended format:
±hh:mm Example: +01:00
Expressions of the difference between local time and UTC of day are a component in the representations
defined in 4.2.5.2; they shall not be used as self-standing expressions.

It looks like the ISO8601DateTimeFormatter implementation inherited from utf8json only supports the extended format, but I think it is reasonable that it also supports the basic formats, which is what the timestamp field in your example adheres to.

[russcam]

opened #4910 to address