-
Notifications
You must be signed in to change notification settings - Fork 328
Performance Tuning
If you're not seeing acceptable scan performance from ZGrab, there's a number of dials we've found helpful.
While not everyone has a choice of which host they're scanning from, reducing the loss and latency of your connection is important for performance. We recommend scanning from a host with a wired, high bandwidth link to maximize performance.
For reference, a ZGrab scan for http against the 10,000 top domains from my MacBook over WiFi (300-400 Mb/s bandwidth) in an office environment achieved pretty poor scan performance with 10 domains scanned per second. When I recorded network traffic through Wireshark, there wereTCP retransmissions around 4%. This forces TCP to lower the bandwidth it's sending at and massively harms performance. Lost packets and subsequent retransmissions mean we're getting far worse performance than bandwidth alone would indicate we ought.
Using a server with a 1 Gb/s wired link saw 1,000 domains scanned per second with the same parameters.
TLDR: Use the most stable, high bandwidth links you can. It makes a huge difference in performance.
Having DNS servers that do not rate-limit and can support high volumes of lookups is critical for scan performance. It is even more imperative for modules like http or ipp that can have redirects. In some cases, a single target might be redirected to many other domain names, each of which needs to be looked up. For example, at the time of writing passwords.google.com was resolved after 7 HTML redirects. The outcome of this is that your DNS resolver will see many more DNS lookups than targets in some cases and it needs to be able to handle this load.
By default, ZGrab uses the configured default DNS servers on your machine.
These are found on Unix machines at /etc/resolv.conf.
These can point to a local DNS resolver on your machine, a DNS server on your local network, or a public resolver like Google's or Cloudflare's.
We've found that public resolvers can better handle the large volume of queries required better than local ones. You can specify DNS resolvers with:
echo "example.com" | zgrab2 http --dns-resolvers="1.1.1.1,8.8.8.8"We don't use these as the defaults since some users may have configured their OS to use a particular DNS resolver due to privacy or other concerns. ZGrab should behave in a user-predictable way, but this override can be helpful in garnering additional performance.
Senders and Timeouts are an area of trade-off where it's difficult to make recommendations that apply to all host + network settings. By understanding the trade-offs you can make a more informed configuration for your setting.
The more senders you have, the more requests will be in-flight at any given point. While initially this would seem to be a case of "more = better", there are tradeoffs. Some content delivery providers (CDNs) can rate-limit based on IP across the names they serve. By scanning faster you're more likely to trigger these limits. This can be mitigated somewhat by adding source IPs with --local-addr, but not eliminated. Additionally, if you're using a local DNS resolver to resolve names, it can become overloaded if too many lookups happen at once. In conclusion, if you notice your hit-rate is unexpectedly low lowering --senders can help.
A trade-off exists where you want to give enough time to resolve the slower or targets with more redirects to follow, but not so long that you waste time on targets that will never respond. We've found for most modules in our network setup that the default values of --connect-timeout=10s and --target-timeout=60s work well, but these can be adjusted to fit your workflow. The --connect-timeout is how long to wait for the initial connection establishment (think TCP 3-way handshake) and --target-timeout is the time a given module has to completely finish it's scan of an input target. For example, http involves following all HTTP redirects to resolve an input target to it's final response.