support /attributes and /events enpoints from MISP feed for Zeek intel generation

[mmguero]

We've got some MISP capabilities. The code that handles grabbing MISP indicators is here and says in its comments:

# download the URL and parse as JSON to figure out what it is. it could be:
# - a manifest JSON (https://www.circl.lu/doc/misp/feed-osint/manifest.json)
# - a directory listing containing a manifest.json (https://www.circl.lu/doc/misp/feed-osint/)
# - a directory listing of misc. JSON files without a manifest.json

Some colleagues at USAF have been poking at it having discussions with us about expanding its compatibility of what it can handle. They've suggested we look at running MISP with docker compose and pulling from it directly.

I'm going to quote some of that discussion here:

---

Hey @mmguero I figured out what you should be querying from MISP to integrate with Malcolm. They're called "attributes".

• Get a list of attributes from sending HTTP GET to the /attributes resource to get a list, as follows:

RESOURCE="/attributes"
MISP_URL=https://misp_url.local (yours will be different)
MISP_API_KEY=2uKLbs44NsGSx0dsJdM4V4MYj9BrH7tOlcRHLedj (yours will be different)

echo "requesting $RESOURCE"
curl \
        --header "Authorization:$MISP_API_KEY" \
        --header "Accept: application/json" \
        --header "Content-Type: application/json" \
        $MISP_URL$RESOURCE

• The result will be a JSON, which is a massive list of dictionaries (see the id field below, at 626143!). Each of which resembles the following:

    {
        "id": "626143",
        "event_id": "166",
        "object_id": "0",
        "object_relation": null,
        "category": "Network activity",
        "type": "ip-dst",
        "value1": "80.51.7.66",
        "value2": "",
        "to_ids": true,
        "uuid": "37bb0fca-9043-4e09-a758-0efb3eae9937",
        "timestamp": "1704950232",
        "distribution": "5",
        "sharing_group_id": "0",
        "comment": "",
        "deleted": false,
        "disable_correlation": false,
        "first_seen": null,
        "last_seen": null,
        "value": "80.51.7.66"
    },
   { ...

As you might guess, the meat of this is the "value" member of the JSON, as that IP address is the dirty IP address that MISP is trying to say is malicious. I assume the value1 and value2 fields should be parsed as well for similar reasons.

What I'd recommend is standing up a MISP instance, loading the full list of default feeds, enabling them, and fetching from them (a guide is here).

And then waiting a good few hours for it to fetch down this massive list of attributes.

After you feel you have a good number of attributes (use the bash script at the beginning of this post and note the id, as that that is the current size of the list), iterate over each of them and enumerate what members are possible. Then decide which members are relevant to Malcolm and how they should be appropriately parsed. Perhaps that is just the IP addresses ("value" for relevant attributes), but you would know better than me in this regard!

---

I also found a bug! In MISP though: where when it pulls the attributes list, it doesn't give you the complete list. It would only give me 60, despite there actually being 4,000,000. Thus, what I described as just GET-ing /attributes will not give you the full list.

The solution is to find out how long the list is by pulling the attribute list and checking each for the highest id number (it appears that the first element is the highest ID, but I do not conclusively know this. Hence, check each ID from the list given), and then individually pulling all IDs from 1 up to that number.

Code snippet in the comment below to avoid blowing up the channel. Also, note that when we access attributes individually, we now have an outer single-element dictionary of just "Attribute."

---

#!/usr/bin/env python3

import requests
import time

MISP_URL="your misp url"
MISP_API_KEY="your misp api key"

headers = {
        "Authorization": MISP_API_KEY,
        "Accept": "application/json",
        "Content-Type": "application/json"
    }

# show the length of the JSON retrieved, which was 60
r = requests.get(f"{MISP_URL}/attributes", headers=headers)
print("length of GET '/attributes' json (which shows a small number): " + str(len(r.json())))

print("===================================")

# get the largest ID number
largest_id = 0
for attribute in r.json():
    if "id" in attribute and int(attribute["id"]) > largest_id:
        print(f"new largest id {attribute['id']}!")
        largest_id = int(attribute["id"])
    else:
        print(f"not a new largest ID: {attribute['id']}!")
print("largest id (size of actual list of attributes): " + str(largest_id))

print("===================================")

# iterate over each id individually and request the JSON corresponding to it
for attribute_id in range(int(largest_id)):
    r = requests.get(f"{MISP_URL}/attributes/view/{attribute_id}", headers=headers)
    item = r.json()
    if 'Attribute' in item:
        print(f"id = {item['Attribute']['id']}, value = {item['Attribute']['value']}")
    else:
        print(f"NOT AN ATTRIBUTE: id = {attribute_id}, json={item}") # this happens on id=1

---

Sample output:

length of GET '/attributes' json (which shows a small number): 60
===================================
new largest id 4000346!
not a new largest ID: 4000345
not a new largest ID: 4000342
not a new largest ID: 4000336
not a new largest ID: 4000333
not a new largest ID: 4000332
not a new largest ID: 4000330
not a new largest ID: 4000328
not a new largest ID: 4000324
not a new largest ID: 4000321
not a new largest ID: 4000320
not a new largest ID: 4000319
not a new largest ID: 4000318
not a new largest ID: 4000314
not a new largest ID: 4000312
not a new largest ID: 4000309
not a new largest ID: 4000306
not a new largest ID: 4000303
not a new largest ID: 4000301
not a new largest ID: 4000299
not a new largest ID: 4000298
not a new largest ID: 4000294
not a new largest ID: 4000291
not a new largest ID: 4000289
not a new largest ID: 4000288
not a new largest ID: 4000286
not a new largest ID: 4000283
not a new largest ID: 4000281
not a new largest ID: 4000279
not a new largest ID: 4000277
not a new largest ID: 4000276
not a new largest ID: 4000275
not a new largest ID: 4000274
not a new largest ID: 4000273
not a new largest ID: 4000271
not a new largest ID: 4000268
not a new largest ID: 4000265
not a new largest ID: 4000263
not a new largest ID: 4000260
not a new largest ID: 4000256
not a new largest ID: 4000251
not a new largest ID: 4000249
not a new largest ID: 4000247
not a new largest ID: 4000243
not a new largest ID: 4000238
not a new largest ID: 4000234
not a new largest ID: 4000231
not a new largest ID: 4000227
not a new largest ID: 4000225
not a new largest ID: 4000223
not a new largest ID: 4000222
not a new largest ID: 4000219
not a new largest ID: 4000216
not a new largest ID: 4000214
not a new largest ID: 4000212
not a new largest ID: 4000211
not a new largest ID: 4000210
not a new largest ID: 4000209
not a new largest ID: 4000204
not a new largest ID: 4000202
largest id (size of actual list of attributes): 4000346
===================================
NOT AN ATTRIBUTE: id = 0, json={'name': 'Invalid attribute', 'message': 'Invalid attribute', 'url': '/attributes/view/0'}
id = 1, value = 101.32.254.178
id = 2, value = 103.123.62.146
id = 3, value = 103.151.125.131
id = 4, value = 103.193.179.52
id = 5, value = 104.131.72.118

and so on

[mmguero]

#!/usr/bin/env bash

RESOURCE="${1:-/attributes}"
MISP_URL=https://localhost:31443/
MISP_API_KEY=xxxxxxxxxxx

echo "requesting $RESOURCE" >&2
curl -fsSLk \
        --header "Authorization:$MISP_API_KEY" \
        --header "Accept: application/json" \
        --header "Content-Type: application/json" \
        $MISP_URL$RESOURCE

[mmguero]

• MISP OpenAPI spec Attributes

I believe we can use the page, limit, type, from|to|last, to page over these attributes rather than going through from 1 to the highest number and then pulling them one at a time.

[mmguero]

The updated documentation:

---

MISP

In addition to loading Zeek intelligence files on startup, Malcolm will automatically generate a Zeek intelligence file for all Malware Information Sharing Platform (MISP) JSON files found under ./zeek/intel/MISP.

Additionally, if a special text file named .misp_input.txt is found in ./zeek/intel/MISP, that file will be read and processed as a list of MISP feed URLs, one per line, according to the following format:

misp|misp_url|auth_key (optional)

For example:

misp|https://example.com/data/feed-osint/manifest.json|df97338db644c64fbfd90f3e03ba8870
misp|https://example.com/doc/misp/|
misp|https://example.com/attributes|a943f5ff506ee6198e996333e0b672b1
misp|https://example.com/events|a943f5ff506ee6198e996333e0b672b1
…

Malcolm will attempt to connect to the MISP feed(s) and retrieve Attribute objects of MISP events and convert them to the Zeek intelligence format as described above. There are publicly available MISP feeds and communities, or users may run their own MISP instance.

Upon Malcolm connects to the URLs for the MISP feeds in .misp_input.txt, it will attempt to determine the format of the data served and process it accordingly. This could be presented as:

• a manifest JSON file
• a directory listing containing a file named manifest.json
• a directory listing of JSON files without a manifest.json file
• a list of Events returned for a request via the MISP Automation API made to a MISP platform's /events endpoint
• a list of Attributes returned for a request via the MISP Automation API made to a MISP platform's /attributes endpoint

Note that only a subset of MISP attribute types can be expressed with the Zeek intelligence indicator types. MISP attributes with other types will be silently ignored.