Memory leak while reading packets from pcap file?

[codingtony]

Hi,

I'm using pcap4j for reading large pcap files.
I just dump the rawdata to Standard stdout.

At around 400 000 - 450 000 packets the program start to get slow.
Then later on I get an OOM (heap space).

Is there any option to discard the packets as I read them ? I have the impression that the packets might be accumulated in PcapHandle ?

Here's the code (tested with 1.3.1-SNAPSHOT and 1.3.0):

        PcapHandle h = Pcaps.openOffline(pcapFilename);
        Packet p = null;
        long nbPackets = 0;
        while(true) {
            p = h.getNextPacket();
            if (p == null) {
                break;
            }
            UdpPacket up = p.get(UdpPacket.class);
            if (up != null) {
                System.out.println(nbPackets++ + " " + p.toString());
            }
        }

[codingtony]

This code has the same problem :

        PcapHandle h = Pcaps.openOffline(pcapFilename);
        h.loop(0, new PacketListener() {
            long nbPackets=0;
            @Override
            public void gotPacket(Packet p) {
                UdpPacket up = p.get(UdpPacket.class);
                if (up != null) {
                    System.out.println(nbPackets++ + " " + p.toString());
                }
            }
        });

[kaitoy]

PcapHandle doesn't hold any packets. Anyway it seems someone (packet classes, jna, or stuff) leaks memory. I will investigate it.

[kaitoy]

I tried your code with my pcap file including 600000 UDP packets but I couldn't reproduce the OOM.
Can I try with your pcap file?

[codingtony]

I can't, the pcap file contain some sensitive information. I finally used the Perl lib Net::Pcap to extract the data from the file without problem. If I have a non-private data to share, I will.

[codingtony]

What version of JRE you use?

I use :

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

on Ubuntu 14.04 LTS

[kaitoy]

I tried it with
java version "1.7.0_21"
OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-1ubuntu1)
OpenJDK Server VM (build 23.7-b01, mixed mode)
on Ubuntu 13.04 but no error occured.
Can I see the OOM error message you saw?

[codingtony]

Hi Kaitoy,

It looks like the problem occurs only when I run within Eclipse (It crashes my workspace most of the time when the number of packets are high).

I use Eclipse 4.4 (Luna) with a lot of plugins installed.

Can you try to run in Eclipse with a significant amount of packets?

When I inspect with jvisualvm I see a lot of LazyValue and char[] that does not seem to be freed over time.

However, I did the same test on the command line (without Eclipse) and there was no memory problem.

I'm not sure what is the exact root cause of the OOM, but I think that you can close the issue.

Thanks for your time.

[kaitoy]

I ran it on Eclipse Kepler but no error. And when I monitored memory usage and objects with jvisualvm while reading a huge pcap file, it looked all objects including LazyValue and char[] are properly GCed.
So, let me close this ticket without changes.

[omidvd79]

happend to me as well on MAC.
try packet = h.getNextPacketEx(); see if it helps.

[crowlogic]

why was this closed? I still get this with pcap4j , lots of LazyValue objects not being freed. happens with both loop function and iterate with readPacketEx. If you want example files, see this for example. From https://iextrading.com/trading/market-data/#tops

https://www.googleapis.com/download/storage/v1/b/iex/o/data%2Ffeeds%2F20190808%2F20190808_IEXTP1_TOPS1.6.pcap.gz?generation=1565312136721342&alt=media