IDN: Fix compile time detection of linidn2 TR46

[rockdaboot]

No description provided.

[mention-bot]

@rockdaboot, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @rousskov to be potential reviewers.

[bagder]

Yeah, I sort of noticed it when browsing but I didn't think it through all the way and didn't react either. Good catch.

[bagder]

Thanks!

[bagder]

Urgh! We can't use IDN2_TRANSITIONAL. It breaks the IDNA2008 feature for the .de TLD that is the primary reason we need IDNA2008 in the first place. It makes ß get converted to ss.

This is causes test 165 to fail.

[rockdaboot]

It depends what you want. If you want TRANSITIONAL, you have to adjust the test. If you want NON-TRANSITIONAL, you are basically at IDNA2008 plus input mapping (NFC, lowercasing, ...).
If you want both, you have to make it selectable with a new option. I considered this for wget/wget2 as well... overhead, but may satisfy users. With an option the default should IMO be NON-TRANSITIONAL.

Suggestion: --idna=<2008|tr46t|tr46>

If you go with that (or similar), I would implement this for wget/wget2 as well.

[bagder]

Well, I want curl to work with IDN on the .de TLD as they require it. Users won't know what option to use for that, so we need to pick one to use. Changing the test case doesn't make us work with that, only changing the code will. Clearly the transitional option is not good enough for that use case. I took out the transitional again in commit ee35766.

I rather wait for someone to actually need an option to modify the IDNA level before we introduce that.

[rockdaboot]

Wait... use NON-TRANSITIONAL. As IDNA2008 but much better...

[bagder]

Aha! Was that added in the same version as the IDN2_TRANSITIONAL one?

[rockdaboot]

For example... German Umlauts (same goes for accents and special northern europe letters) may be in decomposed form (NFD/NFKD) or composed form (NFC/NFKC). TR46 accepts them both due to internal NFC transforming.

Yes, IDN2_NONTRANSITIONAL has been added with the same version as IDN2_TRANSITIONAL.

[bagder]

Excellent, thank you! Landed in 7d6e3f8 just now.