Option to shuffle addresses from DNS #1694

dreckard · 2017-07-21T11:43:56Z

This patch adds an option CURLOPT_DNS_SHUFFLE_ADDRESSES to explicitly request shuffling of addresses returned for a hostname when there is more than one. This can be useful in situations where the application knows that a round robin approach is appropriate and is willing to accept the consequences of potentially discarding some preference order returned by the system's implementation.

I came across this post from a few years ago which discusses the address ordering issue and mentions shuffling the addresses after they are returned but rejects it as too problematic. I agree that it is not the right approach for general use but I think enough use cases exist for it to be available as an option.

In my case I am working on retrofitting an existing application to use libcurl instead of its own old socket based networking implementation. Its only network task is to connect to a specific hostname which is known to be configured for use with a round robin approach.

I'd be happy to write the docs/tests if this seems merge worthy but I thought I should ask for opinions first.

mention-bot · 2017-07-21T11:44:02Z

@dreckard, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @captain-caveman2k to be potential reviewers.

bagder · 2017-07-21T11:58:34Z

Thanks for your contribution. I think it is an interesting addition that others might be interested in as well.

Have you thought about how this code should behave on re-use of the cached DNS data? My first cursory look on your code hints that you shuffle the order only once so that repeated uses of the cache will use the same address?

coveralls · 2017-07-21T12:16:27Z

Coverage decreased (-0.09%) to 75.189% when pulling 6ed9930 on dreckard:dns_shuffle_master into 42a4cd4 on curl:master.

dreckard · 2017-07-21T13:55:36Z

Right about the code not currently reshuffling on reuse of cached names. I did think about that and I didn't see a really decisive argument either way so I went for the simpler option of leaving it out.

The most compelling case I could think of where reshuffling would be preferred is when retrying requests after a failure but the existing fallback logic should usually allow at least a few other addresses to be attempted (timeoutms_per_addr) so I thought the benefit there would be marginal.

bagder · 2017-07-26T23:34:27Z

Well, if you ask for a page every second from a round-robin-DNS served site using libcurl with a cached DNS entry, you'll now use the same IP (first) for 60 consecutive requests, then do a new DNS resolve and the following 60 requests again to the (new) first IP. That could be a little surprising or just exactly what you'd expect.

It mostly just needs to be clearly documented that it only shuffles them in the actual resolve moment and not later, even if it gets used several times.

dreckard · 2017-07-29T22:19:15Z

Ok agreed, sounds good. I'll try to draft up the docs additions for it in the next few days.

Add new option CURLOPT_DNS_SHUFFLE_ADDRESSES to allow shuffling address order if multiple A records are returned for a name.

…e_master

unit1607: fixed failures due to hacky .c include

dreckard · 2017-08-04T13:59:52Z

Added man page changes and a basic unit test.

…e_master # Conflicts: # include/curl/curl.h # lib/url.c

bagder · 2017-09-21T18:35:20Z

I missed merging this PR before the merge window closed, but it will open again on the release date (October 4) and it would be nice if you could fix the merge conflict until then.

dreckard · 2017-09-25T20:09:57Z

Yup will do.

…e_master

dreckard · 2017-10-01T16:17:05Z

Fixed.

jay · 2018-02-21T07:58:57Z

I notice this is labeled next-feature-window and has been open for a while. I don't like the name CURLOPT_DNS_SHUFFLE_ADDRESSES because I think it sounds ambiguous. I think CURLOPT_SHUFFLE_ADDRESSES and --shuffle-addresses would be better. Otherwise if a user is reading a script I think it will sound to them like it's shuffling the dns- server addresses, which can also be specified.

edit:
or CURLOPT_SHUFFLE_HOST_ADDRESSES. longer but easier to figure out

…uffle_master # Conflicts: # include/curl/curl.h # lib/url.c # tests/data/Makefile.inc # tests/data/test1607 # tests/unit/unit1607.c

…into dns_shuffle_master # Conflicts: # tests/unit/unit1607.c

dreckard · 2018-03-06T08:34:16Z

Fixed the conflicts again. I'm not hugely attached to the name so I can go ahead and change it to CURLOPT_SHUFFLE_HOST_ADDRESSES in the near future.

Several of the resolution related options are currently using the CURLOPT_DNS_ prefix so I was hoping to remain consistent with those but it's a fair point that there's ambiguity introduced.

bagder · 2018-03-16T07:20:50Z

ok @dreckard and @jay, let's finally get this feature merged this window!

@dreckard, can you please rebase this, fix the conflicts (again) and squash the commits that should be squashed? I'm sorry for having dragged this out so that you've had to do this multiple times.

…uffle_master

bagder · 2018-03-17T19:13:12Z

docs/libcurl/opts/CURLOPT_DNS_SHUFFLE_ADDRESSES.3

+CURLcode curl_easy_setopt(CURL *handle, CURLOPT_DNS_SHUFFLE_ADDRESSES, long onoff);
+.fi
+.SH DESCRIPTION
+When a name is resolved and more than one A record is returned, shuffle all


This means "more than one IP address", right? Because A record implies IPv4 address, but surely this also shuffles IPv6 addresses fine?

Yes it does also shuffle IPv6 addresses so good point.

bagder · 2018-03-17T19:45:45Z

Thank you. I squashed and rebased your work, and you will also see that I edited it slightly. Amongst others so that torture tests rune fine on it.

dreckard · 2018-03-17T23:20:43Z

Ah great, thanks! I just committed the option name change suggested by @jay before seeing the notifications for these messages, if it's still desired. I think the tool command line option is still needed as well, sorry for missing that earlier.

dfandrich · 2018-03-18T21:35:16Z

Test 1608 now fails with a memory leak under Polar SSL: https://curl.haxx.se/dev/log.cgi?id=20180318182706-28132#prob75 Was this introduced with this commit or did it just tickle a pre-existing problem?

dreckard force-pushed the dns_shuffle_master branch 2 times, most recently from 25dcd34 to 8a8b593 Compare August 3, 2017 12:26

lib: add option CURLOPT_DNS_SHUFFLE_ADDRESSES

eb5761d

Add new option CURLOPT_DNS_SHUFFLE_ADDRESSES to allow shuffling address order if multiple A records are returned for a name.

dreckard force-pushed the dns_shuffle_master branch from 8a8b593 to eb5761d Compare August 3, 2017 14:38

dreckard added 3 commits August 3, 2017 21:02

Merge branch 'master' of https://github.com/curl/curl into dns_shuffl…

c52a632

…e_master

unit1607: fixed failing test build under gnu89 mode

85a3ad6

lib: made addr_shuffle function non-static

a617c21

unit1607: fixed failures due to hacky .c include

Merge branch 'master' of https://github.com/curl/curl into dns_shuffl…

9043bd6

…e_master # Conflicts: # include/curl/curl.h # lib/url.c

curl deleted a comment from coveralls Sep 21, 2017

bagder added the feature-window A merge of this requires an open feature window label Sep 21, 2017

dreckard added 4 commits September 30, 2017 19:18

Merge branch 'master' of https://github.com/curl/curl into dns_shuffl…

757d8ee

…e_master

lib: fixed incorrect CURLOPT numbering after merge

52f2185

lib: fixed whitespace issues

8a5ba3c

unit1607: fixed whitespace issues and clang warning

904dfa4

dreckard added 6 commits March 3, 2018 10:24

Merge branch 'master' of https://github.com/dreckard/curl into dns_sh…

537451e

…uffle_master # Conflicts: # include/curl/curl.h # lib/url.c # tests/data/Makefile.inc # tests/data/test1607 # tests/unit/unit1607.c

Merge branch 'dns_shuffle_master' of https://github.com/dreckard/curl …

4d038a5

…into dns_shuffle_master # Conflicts: # tests/unit/unit1607.c

CURLOPT_DNS_SHUFFLE_ADDRESSES: update docs year and version info

42c4a2c

unit1608: fix missing references after merge

5d7a733

fixed more missing DNS_SHUFFLE_ADDRESSES references

0e6ef78

docs: reversioned DNS_SHUFFLE_ADDRESSES for speculative 7.60.0

75e60f1

bagder removed the feature-window A merge of this requires an open feature window label Mar 16, 2018

Merge branch 'master' of https://github.com/dreckard/curl into dns_sh…

6eef2c2

…uffle_master

bagder reviewed Mar 17, 2018

View reviewed changes

bagder closed this in d95f3dc Mar 17, 2018

lock bot locked as resolved and limited conversation to collaborators Jun 16, 2018

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Option to shuffle addresses from DNS #1694

Option to shuffle addresses from DNS #1694

dreckard commented Jul 21, 2017

mention-bot commented Jul 21, 2017

bagder commented Jul 21, 2017

coveralls commented Jul 21, 2017

dreckard commented Jul 21, 2017 •

edited

Loading

bagder commented Jul 26, 2017 •

edited

Loading

dreckard commented Jul 29, 2017

dreckard commented Aug 4, 2017

bagder commented Sep 21, 2017

dreckard commented Sep 25, 2017

dreckard commented Oct 1, 2017

jay commented Feb 21, 2018 •

edited

Loading

dreckard commented Mar 6, 2018

bagder commented Mar 16, 2018

bagder Mar 17, 2018

dreckard Mar 17, 2018

bagder commented Mar 17, 2018

dreckard commented Mar 17, 2018

dfandrich commented Mar 18, 2018

Option to shuffle addresses from DNS #1694

Option to shuffle addresses from DNS #1694

Conversation

dreckard commented Jul 21, 2017

mention-bot commented Jul 21, 2017

bagder commented Jul 21, 2017

coveralls commented Jul 21, 2017

dreckard commented Jul 21, 2017 • edited Loading

bagder commented Jul 26, 2017 • edited Loading

dreckard commented Jul 29, 2017

dreckard commented Aug 4, 2017

bagder commented Sep 21, 2017

dreckard commented Sep 25, 2017

dreckard commented Oct 1, 2017

jay commented Feb 21, 2018 • edited Loading

dreckard commented Mar 6, 2018

bagder commented Mar 16, 2018

bagder Mar 17, 2018

Choose a reason for hiding this comment

dreckard Mar 17, 2018

Choose a reason for hiding this comment

bagder commented Mar 17, 2018

dreckard commented Mar 17, 2018

dfandrich commented Mar 18, 2018

dreckard commented Jul 21, 2017 •

edited

Loading

bagder commented Jul 26, 2017 •

edited

Loading

jay commented Feb 21, 2018 •

edited

Loading