-
-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leaking #146
Comments
The recovery happens when we restart the processes or when OOM killer kills one of the running DSCs. |
Wow^Hopsi, will look into it! |
Note. I just checked a server where we had old 201203250530-3 running and the old version did not had the leak. I now restarted one server without all the geoIP features - on Monday I will see if the leak is related to the new geoIP datasets. |
Great, thanks! I will run some large captures through |
Okay, I've gone through all the code and found one issue with the IPv4 fragment reassembly code, old fragments are not freed. TCP reassembly code is OK since it clears them after 60 seconds. GeoIP seems OK also, checked their latest code on GitHub, sure you might run an older version with memory leaks (haven't dug through their changelog). |
Can you test the latest develop and tell me how it goes? There may be a small performance impact if you have a lot of fragments that are not getting reassembled because it will need to iterate and clear the old. If you see drops in packets captured during interval we can try add an option to disable reassembly of ipv4 fragments (v6 are ignored) and only process the first segment. |
I disabled TCP capturing and all the geoip features and memory still leaks. Hence, the sgementation resambling may be indeed the problem. btw: does DSC handle segmented IPv6 packets? |
There is two different reassembles going on, one for the TCP segments and one for IP fragmentation. The IP fragmentation reassembly has been in the code for many years, don't know if it has always been enabled or not. I can't see why your filter would block IP fragments, maybe you need to The current code in DSC drops all IPv6 packets that has a fragmentation header. |
- add conf option `drop_ip_fragments` to control IP fragmentation reassembly within pcap_layers - Fix spacing in conf man-page
- Issue DNS-OARC#146: add conf option `drop_ip_fragments` to control IP fragmentation reassembly within pcap_layers - Fix spacing in conf man-page
Current develop has |
a) I did test with your mem-leak-fix, and it seems to work (only running for 24h now) b) My filter c) What does Will DSC analyze an incomplete answer (e.g. only the first fragment was seen)? |
a) Great! then I consider this issue resolved :) DSC needs the header and first question to process the query/response, otherwise it is marked as malformed. Making it process only the first fragment and skip reassembly would be another feature request for which I currently can't say when I will have time for. |
so, when using |
No, any fragmented packets are put on a list until they can be fully reassembled. Only after that are they processed. If they timeout, they are dropped. |
Hi!
dsc seems to leak memory:
We are seeing similiar behavior on all our nodes. The more traffic a node has, the faster the memory usage grows. We are using version 2.4.0, but we saw this behavior also with old versions (e.g pre2.0). I can't remember if we had this problem also with very old Debian version (201203250530).
I once had a problem with the old 201203250530 where I found out this special memory leak (not slowly over days but consuming all memory within a few minutes) was caused by some crappy TCP traffic which caused dsc to loop inside the TCP reassembly code.
So, now I do not have an idea where this is coming from. Maybe you add some memory debugging to DSC (eg reported also in the XML files) to find out where this leak is coming from.
The text was updated successfully, but these errors were encountered: