[Infoblox BloxOne DDI] Failure to Ingest DHCP Lease Logs

[MakoWish]

We just spun up the Infoblox BloxOne DDI integration the other day, and we have not yet received a single event. I tested the API with Postman, and I do see plenty of events to be ingesting. I checked out the Elastic Agent logs, and I am seeing thousands of messages for the events failing to ingest. Here is a sample error message:

{
  "@timestamp": "2023-09-21T20:27:04.014Z",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "httpjson-default",
    "type": "httpjson"
  },
  "ecs.version": "1.6.0",
  "log": {
    "source": "httpjson-default"
  },
  "log.level": "warn",
  "log.logger": "elasticsearch",
  "log.origin": {
    "file.line": 446,
    "file.name": "elasticsearch/client.go"
  },
  "message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.September, 21, 13, 27, 2, 972979155, time.Local), Meta:{\"input_id\":\"httpjson-infoblox_bloxone_ddi-68793c46-1a78-43a1-9dfe-cc5006ce15ca\",\"raw_index\":\"logs-infoblox_bloxone_ddi.dhcp_lease-default\",\"stream_id\":\"httpjson-infoblox_bloxone_ddi.dhcp_lease-68793c46-1a78-43a1-9dfe-cc5006ce15ca\"}, Fields:{\"agent\":{\"ephemeral_id\":\"01234567-89ab-cdef-fedc-ba9876543210\",\"id\":\"01234567-89ab-cdef-fedc-ba9876543210\",\"name\":\"Host1\",\"type\":\"filebeat\",\"version\":\"8.9.2\"},\"data_stream\":{\"dataset\":\"infoblox_bloxone_ddi.dhcp_lease\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"01234567-89ab-cdef-fedc-ba9876543210\",\"snapshot\":false,\"version\":\"8.9.2\"},\"event\":{\"created\":\"2023-09-21T20:27:02.972Z\",\"dataset\":\"infoblox_bloxone_ddi.dhcp_lease\"},\"input\":{\"type\":\"httpjson\"},\"message\":\"{\\\"address\\\":\\\"10.10.10.10\\\",\\\"client_id\\\":\\\"\\\",\\\"ends\\\":\\\"2023-09-22T00:27:00Z\\\",\\\"fingerprint\\\":\\\"System Name\\\",\\\"fingerprint_processed\\\":\\\"processed\\\",\\\"ha_group\\\":\\\"dhcp/ha_group/01234567-89ab-cdef-fedc-ba9876543210\\\",\\\"hardware\\\":\\\"00:11:22:33:44:55\\\",\\\"host\\\":\\\"dhcp/host/123456\\\",\\\"hostname\\\":\\\"system_name.contoso.com\\\",\\\"iaid\\\":0,\\\"last_updated\\\":\\\"2023-09-21T20:27:00.774Z\\\",\\\"options\\\":\\\"{\\\\\\\"Options\\\\\\\":[{\\\\\\\"Code\\\\\\\":\\\\\\\"51\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"AAA4QA==\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"53\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"Aw==\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"55\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"AQIDBAYPKjYHoEIr\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"6\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"CmQGYwozAGM=\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"60\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"UG9seWNvbS1TU0lQNzAwMA==\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"1\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"///+AA==\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"125\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"LoremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaUtenimadminimveniamquisnostrudexercitationullamcolaborisnisiut\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"12\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"Loremipsumdolorsitametconse=\\\\\\\"},{\\\\\\\"Code\\\\\\\":\\\\\\\"3\\\\\\\",\\\\\\\"Value\\\\\\\":\\\\\\\"CnVgAQ==\\\\\\\"}]}\\\",\\\"preferred_lifetime\\\":\\\"2023-09-21T20:27:00Z\\\",\\\"protocol\\\":\\\"\\\",\\\"space\\\":\\\"ipam/ip_space/01234567-89ab-cdef-fedc-ba9876543210\\\",\\\"starts\\\":\\\"2023-09-21T20:27:00Z\\\",\\\"state\\\":\\\"used\\\",\\\"type\\\":\\\"DHCPv4\\\"}\",\"tags\":[\"preserve_original_event\",\"forwarded\",\"infoblox_bloxone_ddi-dhcp_lease\"]}, Private:(*cursor.updateOp)(0xc001e6a080), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:727] failed to parse field [infoblox_bloxone_ddi.dhcp_lease.options] of type [flattened] in document with id '5ajFreVU0GdBz0vi/y4KX0hJKOA='. Preview of field's value: '{\\\"Options\\\":[{\\\"Code\\\":\\\"51\\\",\\\"Value\\\":\\\"AAA4QA==\\\"},{\\\"Code\\\":\\\"53\\\",\\\"Value\\\":\\\"Aw==\\\"},{\\\"Code\\\":\\\"55\\\",\\\"Value\\\":\\\"AQIDBAYPKjYHoEIr\\\"},{\\\"Code\\\":\\\"6\\\",\\\"Value\\\":\\\"CmQGYwozAGM=\\\"},{\\\"Code\\\":\\\"60\\\",\\\"Value\\\":\\\"UG9seWNvbS1TU0lQNzAwMA==\\\"},{\\\"Code\\\":\\\"1\\\",\\\"Value\\\":\\\"///+AA==\\\"},{\\\"Code\\\":\\\"125\\\",\\\"Value\\\":\\\"LoremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaUtenimadminimveniamquisnostrudexercitationullamcolaborisnisiut\\\"},{\\\"Code\\\":\\\"12\\\",\\\"Value\\\":\\\"Loremipsumdolorsitametconse=\\\"},{\\\"Code\\\":\\\"3\\\",\\\"Value\\\":\\\"CnVgAQ==\\\"}]}'\",\"caused_by\":{\"type\":\"parsing_exception\",\"reason\":\"Failed to parse object: expecting token of type [START_OBJECT] but found [VALUE_STRING]\",\"line\":1,\"col\":727}}, dropping event!",
  "service.name": "filebeat"
}

It seems the pipeline for the DHCP logs is attempting to rename the options field to a flattened infoblox_bloxone_ddi.dhcp_lease.options field, but the value is always a string, so they ingest fails. This field needs to first be parsed to JSON, and it can then be ingested properly.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@MakoWish Have you also take a look at the dns_data datastream? It also has an options field, it would be good to know if we need to fix that too.

[MakoWish]

No fix required on that one that I can see. We are ingesting those events without any modifications to the pipeline.