What is the output format of capture packet?

[rickyzhang82]

I tested tcpflow in FreeBSD and Fedora Linux to capture packets from my network interface.

But the output binary file can't be read by wireshark. I tried auto format detection and also tcpdump pcap. But wireshark report it doesn't understand the format.

What is the specification of tcpflow output format?

I'm using 1.5.0 tcpflow and 3.0.3 wireshark.

TIA

[simsong]

Tcpflow doesn’t output packets, it outputs tcp streams. Wire shark doesn’t read tcpstreams, it reads packets. You are welcome to integrate wireshark’s dissecting code into tcpflow. It would be a fun project.

---- Sent from my phone.

On Oct 19, 2019, at 10:58 PM, Ricky Zhang ***@***.***> wrote:

 I tested tcpflow in FreeBSD and Fedora Linux to capture packets from my network interface. But the output binary file can't be read by wireshark. I tried auto format detection and also tcpdump pcap. But wireshark report it doesn't understand the format. What is the specification of tcpflow output format? I'm using 1.5.0 tcpflow and 3.0.3 wireshark. TIA — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[rickyzhang82]

Is tcp stream consist of packets? What is the output format of each output file?

[simsong]

It's the user data without the packet headers and options.

[simsong]

Is tcp stream consist of packets? What is the output format of each output file?

Tcpflow doesn’t output packets, it outputs tcp streams. Wire shark doesn’t read tcpstreams, it reads packets. You are welcome to integrate wireshark’s dissecting code into tcpflow. It would be a fun project.

[rickyzhang82]

I see. I'm doing a research project that collects first 16 packets in each TCP connection. I will need packet header starting form IP protocol.

I'm also looking into different tools build on top of libpcap. Very few of them supports in FreeBSD. But tcpflow does. I will take a look the source code and see how much work is needed. Thanks

[simsong]

From your description, you should just use libpcap or tcpdump directly.

[AlbertVeli]

Is it possible to implement a cmdline option to output in pcap format? It would be useful to output each stream in a separate pcap-file and to have the packets re-ordered at the same time ackording to sequence numbers. Then other tools (like tshark) can be used to continue analyzing the individual pcap-files. Use case. A common CTF challenge involves to reorder tcp-packets and then extract some field (which is being used to hide data). I know this is possible with a sequence of tshark commands piped to sort etc but tcpflow already does most of the work here so the only step left is to add an option to output in pcap-format.

[simsong]

Yes, it is completely possible to implement such an option. All you need to do is to implement a second output file for every stream and then write the packet to the file if the option is set. If you create the option and send me a pull request, I'm happy to take it.

[AlbertVeli]

Yes, it is completely possible to implement such an option. All you need to do is to implement a second output file for every stream and then write the packet to the file if the option is set. If you create the option and send me a pull request, I'm happy to take it.

Cool. I'll start looking at it tomorrow and see if I can understand how libpcap works. It shouldn't be too hard.

[simsong]

Correct. It isn't hard. I'm currently adding a lot of unit tests to be13_api and to dfxml, so be careful NOT to update those sub modules, or else you won't be able to compile tcpflow!

[AlbertVeli]

Hmm, there is already a -K option that seems to do what I want. Introduced in commit b5297f1.

[simsong]

Looks like you are right! Sorry for not realizing that.

[rickyzhang82]

Hey guys,

I wrote an app based on forked version PcapPlusPlus. See my repo. I didn't send a PR to upstream PcapPlusPlus because the maintainer prefers to have none C++ 11.

It meets my own needs:

• Support both Linux and FreeBSD. I used it it pfSense router.
• Support online and offline TCP packet sorting.
• Sort packets by TCP connection by 5 tuples.
• Within TCP connection, sort TCP packets by sequence number.
• Output packets as pcap format.

I used the tool for my machine learning project. See my blog