CURLOPT_IPRESOLVE: preventing wrong IP version from being used

[lvella]

It is still missing documentation, but I hope it is pretty much this.

Added a new easy option CURLOPT_IP_VERSION, which can take either:

• CURL_IP_VERSION_WHATEVER
• CURL_IP_VERSION_V4
• CURL_IP_VERSION_V6

Where WHATEVER is default behavior, and Vx forces the use of that IP version for the connection, by either selecting an appropriate connection from the pool, or by establishing the connection only via an IP address of the appropriate kind.

This is useful for implementing BitTorrent's BEP-7, which requires the client to announce twice, once via IPv4 and once via IPv6.

[bagder]

Adding a new option for IP version selection makes it hard to figure out exactly how these two separate and independent options interact and why we allow them to be set differently etc. As far as I can tell, the new functionality you want to bring here is that the selection also prevents an existing connection with the "wrong" address version to be reused and in other ways it is expected to do the same?

An alternative take could then be to perhaps extend the existing option and add these two selectors:

CURL_IPRESOLVE_V4_STRICT and CURL_IPRESOLVE_V6_STRICT (the names can be tweaked further) that would work as the previous options but also be strict about only ever reusing connections with that specific IP version.

It seems easier to document and understand for users and less risk for internal confusions for us libcurl developers.

Thoughts?

[bagder]

The failed CI jobs are mostly test 1119 that fails due to missing symbols in docs/libcurl/symbols-in-versions

[lvella]

Adding a new option for IP version selection makes it hard to figure out exactly how these two separate and independent options interact and why we allow them to be set differently etc. As far as I can tell, the new functionality you want to bring here is that the selection also prevents an existing connection with the "wrong" address version to be reused and in other ways it is expected to do the same?

An alternative take could then be to perhaps extend the existing option and add these two selectors:

CURL_IPRESOLVE_V4_STRICT and CURL_IPRESOLVE_V6_STRICT (the names can be tweaked further) that would work as the previous options but also be strict about only ever reusing connections with that specific IP version.

It seems easier to document and understand for users and less risk for internal confusions for us libcurl developers.

Thoughts?

I do agree that it might be confusing, but I see two issues with extending CURLOPT_IPRESOLVE:

• I don't like the name, because the new settings do not affect name resolution.
• It is a valid setting to have both CURL_IPRESOLVE_WHATEVER and CURL_IP_VERSION_V4, because I might want to force an IPv4 connection, but still want the IPv6 addresses to be added to DNS cache (that is my use case).

[lvella]

If the two options can be kept separate, it would be better to change the internal variable names to better reflect what option has set them, so it would be obvious for a developer reading the code.

[bagder]

I do agree that it might be confusing, but I see two issues with extending CURLOPT_IPRESOLVE:
I don't like the name, because the new settings do not affect name resolution.

In some ways it does because it can trigger a new name resolve for cases where it previously without this option, wouldn't. So it is at least related.

It is a valid setting to have both CURL_IPRESOLVE_WHATEVER and CURL_IP_VERSION_V4, because I might want to force an IPv4 connection, but still want the IPv6 addresses to be added to DNS cache (that is my use case).

I think that's an internal decision that libcurl will not guarantee based on these options. Sure it might. But it also might not. We don't expose that behavior or knowlwedge to the outside.

To get that functionality we could instead make IPRESOLVE not affect get getaddrinfo() call but only limit what addresses to use from its result, as then it could still cache addresses for a family it won't use (right now).

I would rather argue that if you insist on having a new option for this, the new one should instead be a boolean and be something "insist-on-the-set-IP-version-for-reused-connection" so that it can't contradict the first option.

[lvella]

It is tricky to mimic the original behavior of CURL_IPRESOLVE_* if we actually allow both address types to be resolved. Imagine the case where the first request to a host uses CURL_IPRESOLVE_V4, then subsequent requests uses CURL_IPRESOLVE_V6. What I saw is that all connections will use IPv4 unless I also disable DNS caching. If I disable caching, a newly established connections will be of correct type, but reused connections will be random.

In my opinion, this original behavior is not really useful keeping, because it is random and unpredictable, thus unreliable.

If you also don't care to keep the original behavior, I think we should simply replace the behavior of CURL_IPRESOLVE_* with the strict behavior introduced in this PR. Maybe make the new name CURL_IP_VERSION_* an alias, to better reflect the new meaning.

If you want to keep the old behavior, I propose is to make the new options CURL_IPRESOLVE_Vx_STRICT not limit the name resolving, because the connection type will be restricted later, anyway.

What do you think, @bagder ?

[lvella]

In this new version pushed, the functionality was implemented by adding options CURL_IPRESOLVE_V4_STRICT and CURL_IPRESOLVE_V6_STRICT to CURLOPT_RESOLVE. The original options are the same, and restrict name resolution. The new option do not affect name resolution, but instead they restrict the connection creation and selection from the connection pool.

[lvella]

Now the new options are documented, too.

[bagder]

I've gone back and forth on this, both here in this PR and in my head.

I think I've now (finally) arrived at the place where I agree with your stance: let's change the meaning of this option slightly so that it actually means use this IP version - including when reusing connections. When we don't need any "STRICT" options, we can just use the ones we already have.

It is a minor change in behavior but I really think very very few applications risk getting hurt by this.

My apologies for my "sick-sacking" on this.

[lvella]

Done! CURLOPT_IPRESOLVE now means "use this IP version".

Since no new options or symbols are introduced, does the label "next-feature-window" still makes sense? Can it be considered a bug fix?

[lvella]

Without this change, this test fails in CI. I could not reproduce the failure locally. I assumed it failed because the interface whose IPv6 address is [::1] on the test machines is different from interface of 127.0.0.1, and since localhost resolved to [::1], the socket was bound to the wrong interface.

Maybe this is a good reason to keep CURLOPT_IPRESOLVE separated from the new option CURLOPT_IP_PROTO?

[bagder]

@lvella sorry for being slow on this PR. Isn't it a better take on the test 2100 to make it IPv4-only by adding -4 to the command line? Making it only run for non-ipv6 builds is basically gonna disable it and that would be unfortunate.

[lvella]

@bagder That is the original solution, but -4 no longer disables the IPv6 DNS request, it just prevents IPv6 addresses from being used. I am not sure how to fix this. Do you think I should reverse this behavior, and make DNS resolvers only request the IP version needed immediately? Or alternatively, is there a way to make the server on the test answer both DoH requests?

[bagder]

Ah right, didn't think that through! I now created #7083 as an attempt to make test 2100 work with IPv6 enabled.

[bagder]

Okay, now try to remove your test2100 commit and I think things should be fine!

[bagder]

I think you also need to rebase it on top of master (at least d9eb3e3)

[lvella]

Done! In repos with linear git history, I never know if I must push a merge commit in order to facilitate review, or force push a rebase directly and mess with GitHub "Files changed" tool.

Pushed as a rebase, since you mentioned.

[bagder]

Thank you so much for your hard work and patience with this pull request. Stellar work!

[Appla]

What if I only want to resolve IPv4?
We if we do not use IPv6 and dont want to waste another dns lookup?
this commit totally changed the original behavior which people may rely on
I think the problem described above should be solve via different path but change an option's behavior

[lvella]

What if I only want to resolve IPv4? We if we do not use IPv6 and dont want to waste another dns lookup? this commit totally changed the original behavior which people may rely on I think the problem described above should be solve via different path but change an option's behavior

To make everybody happy, maybe the option should indeed restrict the DNS lookup (like it was was before), and also restrict the connection type (like it is now). But to do this way, the DNS cache would have to change quite a bit (track what IP kinds were resolved or not, track the timeout independently, know if an IP version was not available or not requested, etc), and I was not inclined to do such invasive change back when I wrote this, being it my first patch and all (imagine being the guy who broke curl?)