Event collector API - fields and mapping

[regulartim]

This discussion serves the purpose of designing the Event collector API specification details. It will be implemented by @drona-gyawali during GSoC 2026. Here is my current draft of the API's endpoints and fields:

Sensor initialization endpoint

A sensor is defined as a single instance of a honeypot software or a single backend that collects data from honeytokens. Every Sensor must be created by an API call (except T-Pots). Every Sensor has a owner (except T-Pots): the person owning the API key that was used creating it.

Required fields

• address: IP address of the sensor

Optional fields

• honeypot_type - free text, description of what kind of honeypot e.g. ssh
• honeypot_software - free text, e.g. Cowrie
• honeypot_description - free text, human readable description of the honeypot
• sensor_label - free text, identifying name of the sensor
• group_label - free text, name of a group of sensors that this sensor belongs to
• country - 2 characters, ISO code of the country where the sensor is located, e.g. FR
• asn - integer, AS number of the AS where the sensor is located, e.g. 12345

Response

• id - the id of the created or already existing sensor
• an information whether the sensor alredy existed or was created

Event collection endpoint

An event is defined as a single interaction with a sensor. Events that belong to the same logical unit, share a common session.

Required fields

• src_ip - attacker IP address
• event_type - free text, e.g. login
• timestamp - event detection time (ISO 8601)
• sensor_id - reference to the sensor

Optional fields

• session_id - reference to a session (UUID?)
• token_id - identifier of the honeytoken that triggered the event
• src_port - source port (integer 1–65535)
• dest_port - targeted port (integer 1–65535)
• protocol - free text, e.g. http
• service_name - free text, e.g. nginx
• username - free text
• password - free text
• related_url - free text, payload or download URL
• payload_hash - hex string
• command - free text, shell command captured by interactive honeypots (e.g. Cowrie)
• cve_id - free text, CVE identifier of the vulnerability being targeted (e.g. CVE-2024-4577)

Fallback fields

• data - json, fallback for additional information

Process flow

1. user creates sensor via Sensor initialization endpoint → gets sensor_id
2. user sends event to Event collection endpoint which must contain the sensor_id

[regulartim]

Field mapping

Note: some of these models / model fields don't exist yet and would need to be created. Those are written in italic.

Sensor data

• address -> Sensor.address
• honeypot_type -> Sensor.honeypot_type
• honeypot_software -> Sensor.honeypot_software
• honeypot_description -> Sensor.honeypot_description
• sensor_label -> Sensor.label
• group_label -> Sensor.group_label
• country -> Sensor.country
• asn -> Sensor.autonomous_system (ForeignKey)

Event data

Every event that is sent to the API is first saved into a RawEvent model. The raw data is dumped into this model immediately, then an async django-q task does post-processing later. The fields of the RawEvent model could (and maybe should) be named identically to those in the API. The mapping from RawEvent to our existing models could be like this:

• src_ip -> IOC.name
• event_type -> None
• timestamp -> IOC.last_seen / IOC.first_seen / None
• sensor_id -> IOC.sensors
• session_id -> None (or we could generalise the Cowrie Session model)
• token_id -> None
• src_port -> None
• dest_port -> IOC.destination_ports
• protocol -> None
• service_name -> None
• username -> Credential.username
• password -> Credential.password
• related_url -> IOC.related_urls
• payload_hash -> None
• command -> CommandSequence
• cve_id -> IOC.cves or None
• data -> None

Fields that are mapped to "None" can be excluded from post-processing right now. So they only get written into RawEvent and nothing else happens to them. I think this is fine for now. We can make use of them later, when we come up with further ideas. Also, we should create a admin page for the RawEvent, such that we can explore them in die admin interface. These fields should be visible there.

[drona-gyawali]

Hi @regulartim :), I hope you're doing well. Could you please take a look at this mapping when you have a moment?

I have been looking into the remaining fields that are currently mapped to None and wanted to discuss a few mappings that seem fairly straightforward and less debatable before I start implementing them.

• src_port → IOC.source_ports (new field, similar to destination_ports)

  • Since we already aggregate destination_ports on IOC, I was wondering if aggregating source ports would also make sense for attacker behaviour and analytics.
  • One thing I'm unsure about is whether src_port is available in our T-Pot/Elasticsearch logs as well. Could you please help me to confirm?

• protocol → IOC.protocols (new field)

  • Protocols ( ssh, etc.) seem like attacker characteristics and could be useful for filtering and statistics.

• service_name → IOC.services (new field)

  • Service names (nginx , etc.) also seem useful as aggregated IOC metadata.

• cve_id and event_type → Tag

  • These feel more like metadata than first-class entities, so reusing the existing Tag model seems appropriate.

For the remaining fields:

• session_id: I initially thought about mapping it to CowrieSession, but there seems to be a type mismatch since the Event Collector allows arbitrary session identifiers while CowrieSession.session_id is a BigIntegerField.

• token_id: I haven't found a natural place for it in the current models yet and I'm not sure whether it would eventually require a dedicated HoneyToken model or something else.

• I intially thought mapping payload_hash to HoneypotPayload.sha256, but I'm not sure whether HoneypotPayload is intended to represent generic observed payload hashes or only files that have actually been downloaded and quarantined . Since the Event Collector API only receives a hash and not the file itself, I'm hesitant to reuse this model without your guidance.

These are the mappings I was planning to tackle if they make sense. I'd appreciate your thoughts and guidance before I proceed further. Thank you :)

[drona-gyawali]

Awesome, thank you :) See you tomorrow.

[regulartim]

src_port → IOC.source_ports

The source port is in fact available for most events that T-Pot records (field name: src_port) . So it would be quite easy to make a new array field and fill it with data. However I am not 100% sure if it is a good idea to do so. The source port is often just a random number, so that array field might typically fill up quickly with many of entries without providing much value. So I think the status quo is fine: src_port is stored on RawEvent and visible in the admin, but not processed into IoC data.

protocol → IOC.protocols (new field)

Yes, let's do that. For the Cowrie honeypot, there is also a protocol field in the T-Pot data. Could be also available on other honeypots.

service_name → IOC.services (new field)

Here I don't have any idea how to fill it with T-Pot data, though. The service name is more relevant for high interaction honeypots. Also, it is sensor metadata and we already carry a relation to the sensor in the IOC model. So it feels kind of redundant. Even worse, you can most of the time infer the service when you look at protocol and dest_pot. I would just keep it in RawEvent and not process it.

cve_id → Tag

I would prefer a separate field here, like IOC.cves. It's easier to query and the Tag model mainly contains enrichtment data from third parties. For this field we could also look at the T-Pot data (I can provide you a dump if you want). I guess some honeypot provide a corresponding field.

event_type → Tag

This definitely is metadata, but of the event. My thought behind the field is, that at some point we can exclude events from post processing that have a certain type, like "info". I would like to leave it out of the IOC model.

session_id

You are correct here. Let's just leave it in RawEvent, it can be used to cluster single events to a session there. Maybe we will process the data in the future, when we have a concrete use case.

token_id

Yes, I think a HoneyToken model could make sense here. However it could also be treated as sensor metadata. A honeytoken (to my knowledge) can not emit events on its own. It needs a backend (= a sensor) that does that. Again, I would wait until we have a concrete use case here.

payload_hash

We can use that model as sha256 is the only required field. But than we would basically have two types of rows and consumers need to handle that. What's your opinion @opbot-xd ?

[opbot-xd]

Hi @regulartim @drona-gyawali, hope you both are doing well :D
Reusing the HoneypotPayload model seems fine to me here...

Why:

1. The SHA-256 hash is a unique identifier. If we observe a hash via the Event Collector API (without a physical file) and the same file is later downloaded/quarantined via the PayloadExtractionJob cron job.
2. HoneypotPayload already has the necessary Many-to-Many relationships with IOC and CowrieSession. Having a separate model would duplicate these relations
3. The schema already defines payload_file as optional (blank=True, null=True)

to handle the two types of rows (with/without files):

• Add a helper property (e.g. is_quarantined) to check if payload_file is populated
• API Serializers / Views: Check is_quarantined when handling download requests

I think it should work fine in my opinion....

[regulartim]

Thanks @opbot-xd ! :)

Add a helper property (e.g. is_quarantined) to check if payload_file is populated

Checking if is_quarantined is False would be equivalent to check if payload_file is NULL, right? If so, we could just do the latter.

[opbot-xd]

Thanks @opbot-xd ! :)

Add a helper property (e.g. is_quarantined) to check if payload_file is populated

Checking if is_quarantined is False would be equivalent to check if payload_file is NULL, right? If so, we could just do the latter.

Yes, for right now it would be fine to make a if check and later if we would need to have such checks again then we can use the property wrapper