write support for pcapng files would be helpful

[patrickpreuss]

hi

we have several appliances that produces output similar to "tcpdump -x" hexdumps.
they can contain several interfaces at once, when using text2pcap infos are lost,
like: direction / captured interface.

timeformat can be from start of capture or full time
sample:

0.367441 wan1 in 54.239.26.180.443 -> 192.168.13.31.58723: ack 1152503568
0x0000 0000 0000 0001 f4ec 38fc 24c2 0800 4500 ........8.$...E.
0x0010 0028 cc67 4000 d906 b5fd 36ef 1ab4 c0a8 .(.g@.....6.....
0x0020 0d1f 01bb e563 9336 03f3 44b1 cf10 5010 .....c.6..D...P.
0x0030 012c fd33 0000 .,.3..

0.367837 internal7 out 54.239.26.180.443 -> 10.255.86.140.58723: ack 1152503568
0x0000 0000 0000 0000 085b 0eaa 66e1 0800 4500 .......[..f...E.
0x0010 0028 cc67 4000 d806 233a 36ef 1ab4 0aff .(.g@...#:6.....
0x0020 568c 01bb e563 9336 03f3 44b1 cf10 5010 V....c.6..D...P.
0x0030 012c 6970 0000

[tannewt]

I'd like write support as well so that I can use Python to convert a proprietary csv format to pcapng. I'm happy to help implement it if you (@rshk) give more details on the approach.

[tannewt]

My repo has the beginnings of write support: https://github.com/tannewt/python-pcapng

[rshk]

@tannewt I don't have time to work on that unfortunately, but contributions are very welcome; from a quick look it seems like you're using a reasonable approach.

Just a quick one I noticed re: https://github.com/rshk/python-pcapng/compare/master...tannewt:master?expand=1#diff-87f13c0cc3fe1abcdf0a9d48aab13fabR50 (Block.encode method): IIRC you have to write the original size here, which is then used to strip padding from the read data. I.e. if you had to encode the letter 'A', that would mean you write 1 as the size, then b"A\x00\x00\x00" as the data; now a reading process knows that the last three NULL bytes are just padding and need to be stripped.

Anyways, feel free to open a PR once you want me to start having a look at your changes (even if it's not ready for merging yet, just mark it as WIP); make sure you add some tests too 😄 (how to test is probably the tricky bit, but for starters I guess some simple encode / read back trying to cover most edge cases should do).

Thanks!

[Flameeyes]

@luchotiz fyi.

[Boolean263]

Just a quick one I noticed re: https://github.com/rshk/python-pcapng/compare/master...tannewt:master?expand=1#diff-87f13c0cc3fe1abcdf0a9d48aab13fabR50 (Block.encode method): IIRC you have to write the original size here, which is then used to strip padding from the read data. I.e. if you had to encode the letter 'A', that would mean you write 1 as the size, then b"A\x00\x00\x00" as the data; now a reading process knows that the last three NULL bytes are just padding and need to be stripped.

For whatever it's worth, a strict reading of the current spec's section on block structure seems to agree with @tannewt 's approach. The stated purpose of the overall block size is to allow readers to easily skip over blocks they don't recognize, and making them check/calculate padding on reading each block would slow that down.

There is no mention of padding in the overall block description. The descriptions for options and for individual block types are where I find mentions of padding to a 32-bit boundary. Each includes its own internal length field(s) for how much of the data is meaningful to it. E.g. the EPB has a "Captured packet length" field, which is specified to not include the padding in the data (which in turn is specified to be padded to a 32-bit boundary).

Just 2¢ from another capture format enthusiast. ¯\(ツ)/¯

[Boolean263]

Hi again all,

I've decided to take @tannewt 's approach and run with it. Here's what I've come up with so far. I've changed the bottom section of the README to describe my progress so far. I'd be grateful to know what you think. Especially if you'd be happy to see it in the main code.

For whatever it's worth, my desire for this feature is to generate/modify pcapng files that demonstrate features of pcapng, to be used as test files for other software. (e.g. adding multiple comments to a single packet, as a test case for Wireshark bug #14329.)

Thanks for creating such a useful library, and making it Open Source!

[jaedolph]

@rshk I have been using the latest master branch in an application that uses the write support features. It is working well :)

Is a pypi release planned that includes this feature?

[bridgetrossiter]

+1 for the pypi release of write support. Would be awesome if we could get this.

[rshk]

This is now supported since version 2.0.0. Awsome work everyone!

(Tagging @jaedolph and @bridgetrossiter so you can upgrade your deps 🙂 )

[jaedolph]

This is now supported since version 2.0.0. Awsome work everyone!

(Tagging @jaedolph and @bridgetrossiter so you can upgrade your deps 🙂 )

Thanks for this!