Skip to content

Conversation

@djw8605
Copy link
Member

@djw8605 djw8605 commented Jan 31, 2024

Add DNS cache after profiling identified it as slow. For example, before the addition, the profile appears:

         9107132 function calls (9088394 primitive calls) in 735.939 seconds

   Ordered by: cumulative time
   List reduced from 438 to 30 due to restriction <30>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10340    0.805    0.000  732.876    0.071 /home/jovyan/xrootd-monitoring-collector/Collectors/DetailedCollector.py:442(process)
    13073    0.102    0.000  707.242    0.054 /home/jovyan/xrootd-monitoring-collector/Collectors/DetailedCollector.py:50(_determineHostname)
    13073  707.013    0.054  707.125    0.054 {built-in method _socket.gethostbyaddr}
     3603    0.330    0.000  186.601    0.052 /home/jovyan/xrootd-monitoring-collector/Collectors/DetailedCollector.py:91(addRecord)
       14    0.020    0.001   75.320    5.380 /home/jovyan/xrootd-monitoring-collector/Collectors/DetailedCollector.py:425(process_tcp)

After adding the Cache:

         309876753 function calls (309131296 primitive calls) in 681.991 seconds
...
     1680  114.268    0.068  114.283    0.068 {built-in method _socket.gethostbyaddr}

After 735 seconds of profiling, 707 seconds was used for DNS resolution (96% of the execution). After adding the dns cache, 16% of the time was spent determining hostnames. Additionally, as the cache matures and there are repeated requests, we will get see even less time used by DNS resolution.

The execution now is limited by the speed of the DNS cache and servers.

@djw8605 djw8605 requested review from bbockelm and jthiltges January 31, 2024 18:53
Copy link
Collaborator

@jthiltges jthiltges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
A 1-day TTL is probably OK, particularly for reverse lookups.

@djw8605 djw8605 merged commit b75bfac into opensciencegrid:master Jan 31, 2024
@djw8605 djw8605 deleted the add-dns-cache branch January 31, 2024 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants