When using the threatintel module, and misp data the fields do not accept cidr notation #29949

toddferg · 2022-01-21T16:40:36Z

Operating System: Centos 8.

Version: Filebeat 7.16.3. and Elasticsearch 7.16.3
Module: Threatintel
Sub Config: Misp.

Depending on the data from the Misp source, some of the ip's could be in cidr notation specifically when using the ip address field, it doesn't accept the cidr notation as a valid ip.

example field:

threatintel.indicator.ip

Quick test to check with elasticsearch:


PUT test
{
  "mappings": {
    "properties": {
      "ip": {
        "type": "ip"
      }
    }
  }
}



POST test/_doc
{
  "ip": "192.168.50.0/24"
}


POST test/_doc
{
  "ip": "192.168.50.2"
}


GET test/_search
{
  "query": {
    "term": {
      "ip": {
        "value": "192.168.50.0/24"
      }
    }
  }
}

DELETE test

In general the misp data might include cidrs notation and the beat will have to work with that information.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2022-01-21T16:40:37Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2022-01-23T20:56:03Z

elastic/ecs#86 is related.

@toddferg, what behaviour would you like to see here? At the moment I think all we can to is broaden grok pattern tolerance/add disects to allow 192.168.50.0 to be kept, but lose the CIDR prefix size. This would allow this class of search to succeed, but loses the IP range.

Also, do you have example log lines that have this type of input?

toddferg · 2022-01-27T18:30:07Z

@efd6 Sorry for the delay.

So I'm not sure what would be best. Because siem rules might do best to use the ip range searches which would work?

In regards to log lines, what do you mean? From the misp input, or the output from the beat to elastic?

efd6 · 2022-01-27T21:42:40Z

So I'm not sure what would be best. Because siem rules might do best to use the ip range searches which would work?

I guess it could be conditionally put into network.cidr though that has not been added yet. Knowing the context of the information coming in would be helpful.

In regards to log lines, what do you mean? From the misp input, or the output from the beat to elastic?

Both? at least whichever you have available.

brsolomon-deloitte · 2023-01-05T14:35:16Z

We are seeing this as well; for example,

{"type":"mapper_parsing_exception","reason":"failed to parse field [threat.indicator.ip] of type [ip] in document with id 'PaaH0fjVM3nSacGwE2UWYchSYbk='. Preview of field's value: '160.202.208.0/22'"

The comments here are missing the crux of the issue. MISP is passing a CIDR range, but the Elasticsearch ip type does not accept a /suffix. You can prove this easily in Dev Tools:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "ip_addr": {
        "type": "ip"
      }
    }
  }
}

PUT my-index-000001/_doc/1
{
  "ip_addr": "103.220.232.0/22"
}

Causes error:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [ip_addr] of type [ip] in document with id '1'. Preview of field's value: '103.220.232.0/22'"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [ip_addr] of type [ip] in document with id '1'. Preview of field's value: '103.220.232.0/22'",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "'103.220.232.0/22' is not an IP string literal."
    }
  },
  "status": 400
}

And to be fair to Elasticseach: this is not an IP address; it is an IP network. It will be correctly rejected by Python's ipaddress module too.

>>> import ipaddress
>>> ipaddress.IPv4Address("103.220.232.0/22")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ipaddress.py", line 1314, in __init__
    raise AddressValueError(f"Unexpected '/' in {address!r}")
ipaddress.AddressValueError: Unexpected '/' in '103.220.232.0/22'
>>> ipaddress.IPv4Network("103.220.232.0/22")
IPv4Network('103.220.232.0/22')

I suggest either:

setting ignore_malformed in the mapping or
doing something like https://stackoverflow.com/a/42178830

brsolomon-deloitte · 2023-01-05T14:37:08Z

This should be classified as a bug in https://github.com/elastic/beats/tree/main/x-pack/filebeat/module/threatintel/misp and the pipeline at https://github.com/elastic/beats/blob/main/x-pack/filebeat/module/threatintel/misp/ingest/pipeline.yml should be fixed to get rid of this error.

brsolomon-deloitte · 2023-01-05T14:38:24Z

@efd6 It's a major bug because it causes MISP data to be rejected outright and not even ingested.

brsolomon-deloitte · 2023-01-05T14:41:24Z

I guess it could be conditionally put into network.cidr though that has not been added yet

This is still not part of ECS as of now either.

brsolomon-deloitte · 2023-01-05T14:56:14Z

Also, do you have example log lines that have this type of input?

Full exception:

{"log.level":"warn","@timestamp":"2023-01-05T14:23:11.585Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 5, 14, 23, 7, 468581449, time.Local), Meta:{\"pipeline\":\"filebeat-8.5.3-threatintel-misp-pipeline\"}, Fields:{\"agent\":{\"ephemeral_id\":\"b195b798-43ff-43f5-a429-204f82cd19f5\",\"id\":\"05906d4e-05e5-406f-9687-552efda1a6d0\",\"name\":\"threatintel8-eck-beats-beat-filebeat-8568dc8d66-4l77z\",\"type\":\"filebeat\",\"version\":\"8.5.3\"},\"ecs\":{\"version\":\"1.12.0\"},\"event\":{\"created\":\"2023-01-05T14:23:07.468Z\",\"dataset\":\"threatintel.misp\",\"module\":\"threatintel\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"misp\"},\"input\":{\"type\":\"httpjson\"},\"message\":\"{\\\"Event\\\":{\\\"Attribute\\\":{\\\"Galaxy\\\":[],\\\"ShadowAttribute\\\":[],\\\"category\\\":\\\"Network activity\\\",\\\"comment\\\":\\\"\\\",\\\"deleted\\\":false,\\\"disable_correlation\\\":false,\\\"distribution\\\":\\\"5\\\",\\\"event_id\\\":\\\"1528\\\",\\\"first_seen\\\":null,\\\"id\\\":\\\"4081012\\\",\\\"last_seen\\\":null,\\\"object_id\\\":\\\"0\\\",\\\"object_relation\\\":null,\\\"sharing_group_id\\\":\\\"0\\\",\\\"timestamp\\\":\\\"1670293864\\\",\\\"to_ids\\\":true,\\\"type\\\":\\\"ip-dst\\\",\\\"uuid\\\":\\\"4fa3610f-5412-41dd-9034-c8294fc3c7c2\\\",\\\"value\\\":\\\"146.88.240.0/24\\\"},\\\"CryptographicKey\\\":[],\\\"EventReport\\\":[],\\\"Galaxy\\\":[],\\\"Object\\\":[],\\\"Org\\\":{\\\"id\\\":\\\"1\\\",\\\"local\\\":true,\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"Orgc\\\":{\\\"id\\\":\\\"1\\\",\\\"local\\\":true,\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"RelatedEvent\\\":[{\\\"Event\\\":{\\\"Org\\\":{\\\"id\\\":\\\"1\\\",\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"Orgc\\\":{\\\"id\\\":\\\"3\\\",\\\"name\\\":\\\"CIRCL\\\",\\\"uuid\\\":\\\"55f6ea5e-2c60-40e5-964f-47a8950d210f\\\"},\\\"analysis\\\":\\\"2\\\",\\\"date\\\":\\\"2018-03-26\\\",\\\"distribution\\\":\\\"3\\\",\\\"id\\\":\\\"896\\\",\\\"info\\\":\\\"OSINT - Forgot About Default Accounts? No Worries, GoScanSSH Didnâ€™t\\\",\\\"org_id\\\":\\\"1\\\",\\\"orgc_id\\\":\\\"3\\\",\\\"published\\\":true,\\\"threat_level_id\\\":\\\"3\\\",\\\"timestamp\\\":\\\"1523865236\\\",\\\"uuid\\\":\\\"5acdb4d0-b534-4713-9612-4a1d950d210f\\\"}}],\\\"ShadowAttribute\\\":[],\\\"Tag\\\":[{\\\"colour\\\":\\\"#004577\\\",\\\"exportable\\\":true,\\\"hide_tag\\\":false,\\\"id\\\":\\\"1\\\",\\\"is_custom_galaxy\\\":false,\\\"is_galaxy\\\":false,\\\"local\\\":0,\\\"local_only\\\":false,\\\"name\\\":\\\"osint:source-type=\\\\\\\"block-or-filter-list\\\\\\\"\\\",\\\"numerical_value\\\":null,\\\"user_id\\\":\\\"0\\\"}],\\\"analysis\\\":\\\"2\\\",\\\"attribute_count\\\":\\\"2053\\\",\\\"date\\\":\\\"2022-07-26\\\",\\\"disable_correlation\\\":false,\\\"distribution\\\":\\\"0\\\",\\\"event_creator_email\\\":\\\"admin@admin.test\\\",\\\"extends_uuid\\\":\\\"\\\",\\\"id\\\":\\\"1528\\\",\\\"info\\\":\\\"firehol_level1 feed\\\",\\\"locked\\\":false,\\\"org_id\\\":\\\"1\\\",\\\"orgc_id\\\":\\\"1\\\",\\\"proposal_email_lock\\\":false,\\\"protected\\\":null,\\\"publish_timestamp\\\":\\\"0\\\",\\\"published\\\":false,\\\"sharing_group_id\\\":\\\"0\\\",\\\"threat_level_id\\\":\\\"4\\\",\\\"timestamp\\\":\\\"1672885819\\\",\\\"uuid\\\":\\\"2ed041a1-33ea-4e54-bf26-bf1c7ce191b3\\\"}}\",\"service\":{\"type\":\"threatintel\"},\"tags\":[\"threatintel-misp\",\"forwarded\"]}, Private:(*cursor.updateOp)(0xc0242b8cc0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [threat.indicator.ip] of type [ip] in document with id '2ZLj0k/Z4pZz7KHKJz47cOXESnA='. Preview of field's value: '146.88.240.0/24'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"'146.88.240.0/24' is not an IP string literal.\"}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

brsolomon-deloitte · 2023-01-05T14:59:42Z

Here's the record prettified with jq

{
  "Event": {
    "Attribute": {
      "Galaxy": [],
      "ShadowAttribute": [],
      "category": "Network activity",
      "comment": "",
      "deleted": false,
      "disable_correlation": false,
      "distribution": "5",
      "event_id": "1528",
      "first_seen": null,
      "id": "4081012",
      "last_seen": null,
      "object_id": "0",
      "object_relation": null,
      "sharing_group_id": "0",
      "timestamp": "1670293864",
      "to_ids": true,
      "type": "ip-dst",
      "uuid": "4fa3610f-5412-41dd-9034-c8294fc3c7c2",
      "value": "146.88.240.0/24"
    },
    "CryptographicKey": [],
    "EventReport": [],
    "Galaxy": [],
    "Object": [],
    "Org": {
      "id": "1",
      "local": true,
      "name": "ORGNAME",
      "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
    },
    "Orgc": {
      "id": "1",
      "local": true,
      "name": "ORGNAME",
      "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
    },
    "RelatedEvent": [
      {
        "Event": {
          "Org": {
            "id": "1",
            "name": "ORGNAME",
            "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
          },
          "Orgc": {
            "id": "3",
            "name": "CIRCL",
            "uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
          },
          "analysis": "2",
          "date": "2018-03-26",
          "distribution": "3",
          "id": "896",
          "info": "OSINT - Forgot About Default Accounts? No Worries, GoScanSSH Didnâ€™t",
          "org_id": "1",
          "orgc_id": "3",
          "published": true,
          "threat_level_id": "3",
          "timestamp": "1523865236",
          "uuid": "5acdb4d0-b534-4713-9612-4a1d950d210f"
        }
      }
    ],
    "ShadowAttribute": [],
    "Tag": [
      {
        "colour": "#004577",
        "exportable": true,
        "hide_tag": false,
        "id": "1",
        "is_custom_galaxy": false,
        "is_galaxy": false,
        "local": 0,
        "local_only": false,
        "name": "osint:source-type=\"block-or-filter-list\"",
        "numerical_value": null,
        "user_id": "0"
      }
    ],
    "analysis": "2",
    "attribute_count": "2053",
    "date": "2022-07-26",
    "disable_correlation": false,
    "distribution": "0",
    "event_creator_email": "admin@admin.test",
    "extends_uuid": "",
    "id": "1528",
    "info": "firehol_level1 feed",
    "locked": false,
    "org_id": "1",
    "orgc_id": "1",
    "proposal_email_lock": false,
    "protected": null,
    "publish_timestamp": "0",
    "published": false,
    "sharing_group_id": "0",
    "threat_level_id": "4",
    "timestamp": "1672885819",
    "uuid": "2ed041a1-33ea-4e54-bf26-bf1c7ce191b3"
  }
}

toddferg added the Team:Security-External Integrations label Jan 21, 2022

mdavis332 mentioned this issue Aug 2, 2022

[ti_cif3] Add New TI integration for the Collective Intelligence Framework v3 elastic/integrations#3839

Merged

4 tasks

brsolomon-deloitte mentioned this issue Jan 5, 2023

[filebeat][threatintel] Ignore bad indicator IPs for MISP fileset #34195

Merged

6 tasks

efd6 closed this as completed in #34195 Jan 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When using the threatintel module, and misp data the fields do not accept cidr notation #29949

When using the threatintel module, and misp data the fields do not accept cidr notation #29949

toddferg commented Jan 21, 2022

elasticmachine commented Jan 21, 2022

efd6 commented Jan 23, 2022 •

edited

toddferg commented Jan 27, 2022

efd6 commented Jan 27, 2022

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023 •

edited

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

When using the threatintel module, and misp data the fields do not accept cidr notation #29949

When using the threatintel module, and misp data the fields do not accept cidr notation #29949

Comments

toddferg commented Jan 21, 2022

elasticmachine commented Jan 21, 2022

efd6 commented Jan 23, 2022 • edited

toddferg commented Jan 27, 2022

efd6 commented Jan 27, 2022

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023 • edited

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

brsolomon-deloitte commented Jan 5, 2023

efd6 commented Jan 23, 2022 •

edited

brsolomon-deloitte commented Jan 5, 2023 •

edited