Coverting packets object to pcap file

[DDB-en]

Please let me know i can write packets as pcap file?
I decode unified files and i have packets list now,

[jasonish]

This does not include a tool for converting unified2 to pcap. There is a tool to convert Suricata eve.json to pcap though.

[DDB-en]

Yes i know but i think there is some issues in 'data' (i mean for example packets[0]['data'])
Because i build a pcap file (base on idstools-eve2pcap method) and i found some errors in packets (or maybe my fault)

This is my code (summary):

class Pcap:
    #same as idstools-eve2pcap 
def eve2pcap(event):
    if not "data" in event:
        return None, None
    packet = event["data"]
    hdr = pcap_pkthdr()
    hdr.ts_sec, hdr.ts_usec = event["packet-second"], event["packet-microsecond"]
    hdr.pktlen = len(packet)
    hdr.caplen = len(packet)
    return (hdr, packet)

linktype=event[0]['linktype']
pcap = Pcap.open_dead(linktype, 65535)
dumper = pcap.dump_open('/tmp/1.pcap')
for p in event:
    hdr, packet = eve2pcap(p)
    if hdr and packet:
        dumper.dump(hdr, ctypes.c_char_p(packet))
dumper.close()

Unified decoded record:

{'extra-data': [], 'mpls-label': None, 'destination-ip': '104.66.89.155', 'signature-revision': 2, 'vlan-id': None, 'signature-id': 2027390, 'protocol': 6, 'classification-id': 2, 'sport-itype': 50990, 'priority': 3, 'appid': None, 'dport-icode': 80, 'blocked': 0, 'impact': 0, 'generator-id': 1, 'source-ip': '192.168.70.25', 'impact-flag': 0, 'source-ip.raw': b'\xc0\xa8F\x19', 'destination-ip.raw': b'hBY\x9b', 'sensor-id': 0, 'event-id': 280, 'packets': [{'sensor-id': 0, 'length': 400, 'data': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x01\x82\x00\x00\x00\x00\x00\x06\xf0\xd7\xc0\xa8F\x19hBY\x9b\xc7.\x00P\x00\x00\x00\x00\x00\x00\x00\x00P\x00\x00\x00\xb2q\x00\x00POST /fwlink/?LinkID=252669&clcid=0x409 HTTP/1.1\r\nConnection: Keep-Alive\r\nContent-Type: text/xml; charset="UTF-16LE"\r\nUser-Agent: MICROSOFT_DEVICE_METADATA_RETRIEVAL_CLIENT\r\nSOAPAction: "http://schemas.microsoft.com/windowsmetadata/services/2007/09/18/dms/DeviceMetadataService/GetDeviceMetadata"\r\nContent-Length: 1420\r\nHost: go.microsoft.com\r\n\r\n', 'packet-microsecond': 742421, 'linktype': 1, 'event-id': 280, 'event-second': 1602229464, 'packet-second': 1602229464}, {'sensor-id': 0, 'length': 1474, 'data': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x05\xb4\x00\x00\x00\x00\x00\x06\xec\xa5\xc0\xa8F\x19hBY\x9b\xc7.\x00P\x00\x00\x00\x00\x00\x00\x00\x00P\x00\x00\x00\xc6K\x00\x00\xff\xfe<\x00?\x00x\x00m\x00l\x00 \x00v\x00e\x00r\x00s\x00i\x00o\x00n\x00=\x00"\x001\x00.\x000\x00"\x00 \x00e\x00n\x00c\x00o\x00d\x00i\x00n\x00g\x00=\x00"\x00U\x00T\x00F\x00-\x001\x006\x00"\x00?\x00>\x00<\x00s\x00:\x00E\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00 \x00x\x00m\x00l\x00n\x00s\x00:\x00s\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00x\x00m\x00l\x00s\x00o\x00a\x00p\x00.\x00o\x00r\x00g\x00/\x00s\x00o\x00a\x00p\x00/\x00e\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00/\x00"\x00>\x00<\x00s\x00:\x00H\x00e\x00a\x00d\x00e\x00r\x00>\x00<\x00h\x00:\x00c\x00d\x00 \x00x\x00m\x00l\x00n\x00s\x00:\x00h\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00m\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00.\x00c\x00o\x00m\x00/\x00w\x00i\x00n\x00d\x00o\x00w\x00s\x00m\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00/\x00s\x00e\x00r\x00v\x00i\x00c\x00e\x00s\x00/\x002\x000\x000\x007\x00/\x000\x009\x00/\x001\x008\x00/\x00d\x00m\x00s\x00"\x00>\x00<\x00h\x00:\x00c\x00v\x00>\x001\x000\x00.\x000\x00.\x001\x009\x000\x004\x001\x00<\x00/\x00h\x00:\x00c\x00v\x00>\x00<\x00h\x00:\x00c\x00c\x00>\x00U\x00S\x00A\x00<\x00/\x00h\x00:\x00c\x00c\x00>\x00<\x00/\x00h\x00:\x00c\x00d\x00>\x00<\x00/\x00s\x00:\x00H\x00e\x00a\x00d\x00e\x00r\x00>\x00<\x00s\x00:\x00B\x00o\x00d\x00y\x00>\x00<\x00D\x00e\x00v\x00i\x00c\x00e\x00M\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00B\x00a\x00t\x00c\x00h\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00 \x00x\x00m\x00l\x00n\x00s\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00m\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00.\x00c\x00o\x00m\x00/\x00w\x00i\x00n\x00d\x00o\x00w\x00s\x00m\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00/\x00s\x00e\x00r\x00v\x00i\x00c\x00e\x00s\x00/\x002\x000\x000\x007\x00/\x000\x009\x00/\x001\x008\x00/\x00d\x00m\x00s\x00"\x00>\x00<\x00L\x00o\x00c\x00L\x00i\x00s\x00t\x00>\x00<\x00l\x00o\x00c\x00>\x00M\x00u\x00l\x00t\x00i\x00L\x00o\x00c\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00l\x00o\x00c\x00>\x00e\x00n\x00-\x00U\x00S\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00l\x00o\x00c\x00>\x00e\x00n\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00/\x00L\x00o\x00c\x00L\x00i\x00s\x00t\x00>\x00<\x00M\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00g\x00d\x00m\x00d\x00m\x00i\x00d\x00>\x00<\x00r\x00i\x00d\x00>\x001\x00<\x00/\x00r\x00i\x00d\x00>\x00<\x00m\x00i\x00d\x00>\x003\x00E\x005\x00B\x005\x00E\x00A\x009\x00-\x005\x007\x005\x005\x00-\x005\x00B\x00C\x007\x00-\x00B\x001\x00B\x00C\x00-\x00F\x00A\x003\x006\x007\x001\x001\x00B\x006\x00C\x002\x008\x00<\x00/\x00m\x00i\x00d\x00>\x00<\x00/\x00g\x00d\x00m\x00d\x00m\x00i\x00d\x00>\x00<\x00/\x00M\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00H\x00W\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00g\x00d\x00m\x00d\x00h\x00w\x00i\x00d\x00>\x00<\x00r\x00i\x00d\x00>\x001\x00<\x00/\x00r\x00i\x00d\x00>\x00<\x00h\x00w\x00i\x00d\x00s\x00>\x00<\x00h\x00w\x00i\x00d\x00>\x00D\x00O\x00I\x00D\x00:\x00M\x00O\x00N\x00I\x00T\x00O\x00R\x00\\\x00D\x00e\x00f\x00a\x00u\x00l\x00t\x00_\x00M\x00o\x00n\x00i\x00t\x00o\x00r\x00<\x00/\x00h\x00w\x00i\x00d\x00>\x00<\x00/\x00h\x00w\x00i\x00d\x00s\x00>\x00<\x00/\x00g\x00d\x00m\x00d\x00h\x00w\x00i\x00d\x00>\x00<\x00/\x00H\x00W\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00/\x00D\x00e\x00v\x00i\x00c\x00e\x00M\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00B\x00a\x00t\x00c\x00h\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00>\x00<\x00/\x00s\x00:\x00B\x00o\x00d\x00y\x00>\x00<\x00/\x00s\x00:\x00E\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00>\x00', 'packet-microsecond': 742421, 'linktype': 1, 'event-id': 280, 'event-second': 1602229464, 'packet-second': 1602229464}], 'event-microsecond': 742421, 'event-second': 1602229464}

This is the output file:
265750498.zip

also this record is same as above record in eve.json:

{"timestamp":"2020-10-09T11:14:24.742421","flow_id":12984035864390,"in_iface":"eth0","event_type":"alert","src_ip":"192.168.70.25","src_port":50990,"dest_ip":"104.66.89.155","dest_port":80,"proto":"TCP","metadata":{"flowbits":["FB180732_0"]},"tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2027390,"rev":2,"signature":"ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent","category":"Unknown Traffic","severity":3,"metadata":{"updated_at":["2019_09_28"],"signature_severity":["Minor"],"performance_impact":["Low"],"former_category":["USER_AGENTS"],"deployment":["Perimeter"],"created_at":["2019_05_28"],"attack_target":["Client_Endpoint"],"affected_product":["Web_Browsers"]}},"http":{"hostname":"go.microsoft.com","url":"\/fwlink\/?LinkID=252669&clcid=0x409","http_user_agent":"MICROSOFT_DEVICE_METADATA_RETRIEVAL_CLIENT","http_method":"POST","protocol":"HTTP\/1.1","length":0},"app_proto":"http","flow":{"pkts_toserver":9,"pkts_toclient":8,"bytes_toserver":4060,"bytes_toclient":1072,"start":"2020-10-09T11:14:24.618310"},"payload":"UE9TVCAvZndsaW5rLz9MaW5rSUQ9MjUyNjY5JmNsY2lkPTB4NDA5IEhUVFAvMS4xDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQpDb250ZW50LVR5cGU6IHRleHQveG1sOyBjaGFyc2V0PSJVVEYtMTZMRSINClVzZXItQWdlbnQ6IE1JQ1JPU09GVF9ERVZJQ0VfTUVUQURBVEFfUkVUUklFVkFMX0NMSUVOVA0KU09BUEFjdGlvbjogImh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vd2luZG93c21ldGFkYXRhL3NlcnZpY2VzLzIwMDcvMDkvMTgvZG1zL0RldmljZU1ldGFkYXRhU2VydmljZS9HZXREZXZpY2VNZXRhZGF0YSINCkNvbnRlbnQtTGVuZ3RoOiAxNDIwDQpIb3N0OiBnby5taWNyb3NvZnQuY29tDQoNCv\/+PAA\/AHgAbQBsACAAdgBlAHIAcwBpAG8AbgA9ACIAMQAuADAAIgAgAGUAbgBjAG8AZABpAG4AZwA9ACIAVQBUAEYALQAxADYAIgA\/AD4APABzADoARQBuAHYAZQBsAG8AcABlACAAeABtAGwAbgBzADoAcwA9ACIAaAB0AHQAcAA6AC8ALwBzAGMAaABlAG0AYQBzAC4AeABtAGwAcwBvAGEAcAAuAG8AcgBnAC8AcwBvAGEAcAAvAGUAbgB2AGUAbABvAHAAZQAvACIAPgA8AHMAOgBIAGUAYQBkAGUAcgA+ADwAaAA6AGMAZAAgAHgAbQBsAG4AcwA6AGgAPQAiAGgAdAB0AHAAOgAvAC8AcwBjAGgAZQBtAGEAcwAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ALwB3AGkAbgBkAG8AdwBzAG0AZQB0AGEAZABhAHQAYQAvAHMAZQByAHYAaQBjAGUAcwAvADIAMAAwADcALwAwADkALwAxADgALwBkAG0AcwAiAD4APABoADoAYwB2AD4AMQAwAC4AMAAuADEAOQAwADQAMQA8AC8AaAA6AGMAdgA+ADwAaAA6AGMAYwA+AFUAUwBBADwALwBoADoAYwBjAD4APAAvAGgAOgBjAGQAPgA8AC8AcwA6AEgAZQBhAGQAZQByAD4APABzADoAQgBvAGQAeQA+ADwARABlAHYAaQBjAGUATQBlAHQAYQBkAGEAdABhAEIAYQB0AGMAaABSAGUAcQB1AGUAcwB0ACAAeABtAGwAbgBzAD0AIgBoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZABvAHcAcwBtAGUAdABhAGQAYQB0AGEALwBzAGUAcgB2AGkAYwBlAHMALwAyADAAMAA3AC8AMAA5AC8AMQA4AC8AZABtAHMAIgA+ADwATABvAGMATABpAHMAdAA+ADwAbABvAGMAPgBNAHUAbAB0AGkATABvAGMAPAAvAGwAbwBjAD4APABsAG8AYwA+AGUAbgAtAFUAUwA8AC8AbABvAGMAPgA8AGwAbwBjAD4AZQBuADwALwBsAG8AYwA+ADwALwBMAG8AYwBMAGkAcwB0AD4APABNAEkARABSAGUAcQB1AGUAcwB0AHMAPgA8AGcAZABtAGQAbQBpAGQAPgA8AHIAaQBkAD4AMQA8AC8AcgBpAGQAPgA8AG0AaQBkAD4AMwBFADUAQgA1AEUAQQA5AC0ANQA3ADUANQAtADUAQgBDADcALQBCADEAQgBDAC0ARgBBADMANgA3ADEAMQBCADYAQwAyADgAPAAvAG0AaQBkAD4APAAvAGcAZABtAGQAbQBpAGQAPgA8AC8ATQBJAEQAUgBlAHEAdQBlAHMAdABzAD4APABIAFcASQBEAFIAZQBxAHUAZQBzAHQAcwA+ADwAZwBkAG0AZABoAHcAaQBkAD4APAByAGkAZAA+ADEAPAAvAHIAaQBkAD4APABoAHcAaQBkAHMAPgA8AGgAdwBpAGQAPgBEAE8ASQBEADoATQBPAE4ASQBUAE8AUgBcAEQAZQBmAGEAdQBsAHQAXwBNAG8AbgBpAHQAbwByADwALwBoAHcAaQBkAD4APAAvAGgAdwBpAGQAcwA+ADwALwBnAGQAbQBkAGgAdwBpAGQAPgA8AC8ASABXAEkARABSAGUAcQB1AGUAcwB0AHMAPgA8AC8ARABlAHYAaQBjAGUATQBlAHQAYQBkAGEAdABhAEIAYQB0AGMAaABSAGUAcQB1AGUAcwB0AD4APAAvAHMAOgBCAG8AZAB5AD4APAAvAHMAOgBFAG4AdgBlAGwAbwBwAGUAPgA=","stream":1,"packet":"AFBWhcYcxBL1MguzCABFAAAouHVAAIAGebvAqEYZaEJZm8cuAFAHVxwRqyIFNlARAgBJ9QAAAAAAAAAA","packet_info":{"linktype":1}}

This pcap file is generated by idstools-eve2pcap:
1.zip

as you can see in first file there is some issues.
please help me to fixed them,
Thanks

[jasonish]

Can you describe what is wrong, and what you expect?

On brief look, in 265750498.zip I see an extra packet. I'm not sure where that is coming from. It looks like 1.zip is a pcap generated from eve using the --payload option? Is that correct?

One difference to note is that the payload in an eve record contains re-assembled data, perhaps from multiple TCP packets and contains no packet headers, so the packet headers are reconstructed by the eve2pcap tool.

Unified2 records have the alerting packet, so this is much likely to be smaller than the payload you'd see in the Eve record. The packet data in unified2 also contains the network header, so they're shouldn't be a need to reconstruct it, but there is also no verification that it is correct.

If you're goal is to convert unified2 to pcap, aren't there existing tools out there to do that?

[DDB-en]

I expect a pcap file same as eve2pcap output (plus network header),

Yes, it was generated from eve using the --payload option.
And about the wrong things, I have some additional warning and information in wireshark on my pcap rather than eve2pcap output. For instance:

1-

Is it normal?

2-
Some times network header is zero, Is there any wrong on my code?

Both issues occurs on non ssl TCP packets usually.
SSL (TCP) and UDP packets are fine

[jasonish]

Code looked fine. Remember that the pcap generation from the Eve payload (Suricata only) crafts a TCP/UDP/IP header which ensures its more or less correct - as the payload field in eve lacks headers.

When using the packet from unified2, the header is taken as-is.

You should look at u2boat and see what its pcap output looks like for comparison.

[jasonish]

One other item, I haven't verified if the Unified2 reader is correct anymore. Snort has been known to update it in incompatible ways, or change the output depending on compile time options.

[DDB-en]

Thanks for the reply @jasonish.