Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix memory leaks #241

Merged
merged 2 commits into from
Jan 16, 2024
Merged

fix memory leaks #241

merged 2 commits into from
Jan 16, 2024

Conversation

tarunKoyalwar
Copy link
Member

Context

  • Due to improper config or changes in config or someother scenario multiple fastdialer are created on demand
  • Every time new instance of fastdialer is created all entries are load from hosts file and stored in selected storage backend
  • in retryablehttp-go we were using fastdialer memory backend and multiple calls due to different scenarios caused hosts file to be loaded multiple lines (and if hosts file is large it caused a mem leak/over consumption of memory and crash nuclei or cause init to take too long (see: Nuclei version 3.x large memory allocation due to /etc/hosts entries. nuclei#4632 )
  • ref: reuse fastdialer + bump go retryablehttp-go#198

Proposed Changes

  • seperate host file and dns cache logic
  • make hostsfile hmap shared /global and is initialized on demand
  • finalizer cleans up any temporary files created by hmap in hybrid mode
  • add env vars to tweak behaviour
MAX_DNS_CACHE_SIZE=10MB # current default
HF_MAX_HOSTS=4096 # hosts file max hosts to load in memory
HF_LOAD_ALL=false # when true uses hybrid backend instead and loads all entries from hosts file
  • added max dns cache size on memory backend
  • now we use either memory or hybrid backend for dns cache / hostsFile data

Copy link
Member

@Mzack9999 Mzack9999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Good redesign against possible fastdialer misuses.

A note about the original implementation: the intention was to leave up to the caller the duty or declaring fastdialer as a singleton instance and pass it along with a global reference. Generally it's the canonical choice also for other similar libraries such as net/http where there isn't a shared client used by all instances. The hosts file permanent caching was left out from a global shared kv, as a sort of mimicking the behavior of the hosts file which is read everytime there is a network lookup, so that new declarations of fastdialer could somehow still benefit from potentially refreshed values (assuming that the caller creating a new instance would have desired the most up to date versions of the file)

@ehsandeep ehsandeep merged commit e5ed9c0 into main Jan 16, 2024
5 checks passed
@ehsandeep ehsandeep deleted the reuse-hosts-file branch January 16, 2024 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants