a few issues with pcap-over-ip

[mosajjal]

Describe the bug
I've been trying to use Arkime in a container to receive pcap-over-ip from polarproxy (amongst other sources) and I've observed a few things:

1. when the bpf= option is set, I could not receive any packets. in my case, the option was set to pbf=not port 9200
2. I believe only big-endian timestamped and ethernet-based pcaps can be sent through the socket to be ingested. this creates a bit of an issue with typical packet capture software (like tcpdump and tshark) since by default they create pcaps with little-endian timestamps. I had to write my own makeshift packet capture that supports big-endian.
3. there's a bit of a delay between when the packets are captured and when they show up on Arkime. My theory is because the underlying TCP connection between polarproxy and Arkime is kept alive and is persistent, Arkime falls back to tcpSaveTimeout and essentially saves the session after 400 seconds. I tested this by lowering that number and sure enough the sessions showed up quicker.
4. the last session(s) on Arkime do not have all their packets saved on disk. I believe this is closely related to the third point because after sending multiple sessions, the previous ones start to index and show up properly.

Elasticsearch version:
tested with opensearch 2.x and ES 8.x

Arkime/Moloch version:
tested with 4.6.0 and 5.0.0rc1

OS name and version:
Ubuntu 22.04 docker base image

How was Arkime/Moloch built/installed: (rpm, deb, easybutton, ...)
deb package on a container

Provide logs, stack traces and steps to reproduce:
the docker compose to set up the entire stack is here, with all the configuration items for ES, Polarproxy and Arkime:
https://github.com/mosajjal/aio-gw

[awick]

So on a M1 mac everything seems to work fine.

1. I'm running in server mode with where I add bpf=port 22 to the tests/config.tests.ini section test-pois and then run ../capture/capture -c config.test.ini -n test-pois. To send files I use nc localhost 57012 < ~/arkime/tests/pcap/ssh2.pcap for example which shows I get the ssh packets

2. both big and little endian are supported, ssh2.pcap and bigendian.pcap are different endianess. You are right that only ethernet is supported. I wonder if whatever you wrote is causing issues?

3. this is just using the normal Arkime packet processing, the same delays if it had been live capture will be there. since all we are getting are packets and no other meta data there is no way to do anything different. or maybe I don't understand your isssue.

4. this is just the typical Arkime packet write buffering that the FAQ talks about since its using the normal Arkime packet processing.

[mosajjal]

Thanks Andy. I believe since the source interface was tun, issue 2 might've been related to that and the code just converts the packets back to ethernet which would've solved the issue regadless of the endianness.

as for 3, my question is when does Arkime decides when to consider a session "finished" when receiving the pcap file over IP? is it when the underlying pcap-over-ip TCP session finishes, or the last byte of the pcap file that's transferred over TCP?

[awick]

2. ok yes, i'll need to add support for non eth
3. all the packet processing is the same, it has no clue its pcap over ip vs live vs from a file. It just uses the timestamps and the normal logic. the state of the pcap-over-ip socket isn't used.

[mosajjal]

cool got it. thanks mate :)

[awick]

So #2576 should hopefully fix the issue

[awick]

5.0.0-rc2 has this fix