url validator, exotic _wildcard_ issue

[jeroengui]

I use the validators library in a URL scanning project. But this URL that work in the browser doesn't get recognized as a URL (replaced the tt by xx because it's a malicious sample):

hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/

I suppose the _wildcard_ part of the domain has something to do with it...

[yozachar]

$ python -c "from validators import __version__; print(__version__)"
0.22.0

$ python -c "from validators import url; print(url('hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))" 
ValidationError(func=url, args={'value': 'hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'})

$ python -c "from validators import url; print(url('https://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))"
True

I do not understand, it works as expected. hxxp / hxxps is a convention, not a valid protocol.

[jeroengui]

I get that hxxp/hxxps is just a convention. I only wanted to raise an issue about the http version.

The Markdown syntax in this issue removed the underscores from the url with _wildcard_, sorry for that.

import validators
print(validators.url('https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'))

When I run this I get this output:

ValidationError(func=url, args={'value': 'https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'})

[yozachar]

Note to self:

Issues

• Underscores in URLs seem to break the validator #102
• validators.url Internationalisation validation fail #38
• Unicode chacter problem #14
• url(public = True) is broken #309

PRs

• Minor fix for hostname with underscores #207
• Updated domain validator coverage #179

Ref:

https://stackoverflow.com/questions/2180465/can-domain-name-subdomains-have-an-underscore-in-it#2183140

According to the specifications outlined in RFC 952 and RFC 1123 (which define the rules for host names), underscores (_) 
are not allowed in host names. Host names should only consist of alphanumeric characters (letters and numbers) and 
hyphens (-).

Domain names, on the other hand, can technically include underscores, but they are not commonly used. The Internet 
Engineering Task Force (IETF) recommends avoiding the use of underscores in domain names due to potential 
interoperability issues with some older systems and applications that may not handle underscores correctly.

In practice, it's best to stick with alphanumeric characters and hyphens for both host names and domain names to ensure 
compatibility and avoid any potential issues.

~ ChatGPT

Use domain name for public URLs and host name validation for non-public URLs.

[yozachar]

Hey @jeroengui the domain _wildcard_.jetsett.in seems to be invalid.

Because the following is valid according to rfc_2782 (TLDR: underscores are prepended not appended).

$ python -c "from validators import url; print(url('https://_wildcard._jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113', rfc_2782=True))"
True

Are you sure it's not _wildcard._jetsett.in instead?

[jeroengui]

I'm not sure about the standard and how more I read how more confused I get, but the thing is:

Most browsers are accepting urls with subdomains like _wildcard_.jetsett.in and malicious actors are using them in phishing campaigns etc.

At the moment it poses a real threat, because a lot of (antivirus) companies use validators for urls just like this one.
The urls never get in their blocklists, exposing their users to more risk because these malicious urls work fine in their browsers.

I don't know if this is something that should be fixed in the browsers, in the standard or in the validators, and the issue is definitely bigger than just this implementation.

So to be clear:

We are figuring out if http://_wildcard_.domain.com is a valid url.

We agree that http://_wildcard.jetsett.in and https://my_sarisari_store.typepad.com/ are valid urls.

So an underscore can be used in any place of the subdomain except the one right before the dot?

Stackoverflow has had some discussion on the topic also: https://stackoverflow.com/questions/2180465/can-domain-name-subdomains-have-an-underscore-in-it

EDIT: I added an example DNS record on my own domain that seems to work fine in most browsers : here (https://_test_.jeroengui.be/)