idna is dissatisfied with underscores

[bpicolo]

import idna
 idna.encode('ex_ample.example.org')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nail/home/benp/pg/utils/bravado-promised/virtualenv_run/local/lib/python2.7/site-packages/idna/core.py", line 354, in encode
    result.append(alabel(label))
  File "/nail/home/benp/pg/utils/bravado-promised/virtualenv_run/local/lib/python2.7/site-packages/idna/core.py", line 275, in alabel
    check_label(label)
  File "/nail/home/benp/pg/utils/bravado-promised/virtualenv_run/local/lib/python2.7/site-packages/idna/core.py", line 252, in check_label
    raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+005F at position 3 of u'ex_ample' not allowed

Seems like underscores in hostname should be perfectly allowed? Can't find any RFC that suggests otherwise (and browsers seem happy with URLs in hostname).

Perhaps this is more of a twisted bug where it shouldn't pass this sort of url to idna ever, not sure. This is just an issue I hit as an end user

[kjd]

Underscores are not legal in IDNs (or, for that matter, ASCII domain names that follow hostname rules which most domain registries apply). There are certain circumstances where they are legal but not as it pertains to IDNs.

Specifically, see http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml under "IDNA Derived Properties", U+003A-U+0060 are DISALLOWED. See also https://tools.ietf.org/html/rfc5890#section-2.3.1 for further discussion.

There may be an idea to add a "less strict" flag (i.e. non-STD 3 rules) to the idna library such that if it received a string that is pure ASCII, it doesn't throw an exception if it is not LDH.

[bpicolo]

@kjd Maybe you could help enlighten me a bit? It seems the point I'm dying at is ssl verification through twisted.
http://twistedmatrix.com/trac/browser/tags/releases/twisted-15.2.1/twisted/internet/_sslverify.py#L67

Should a hostname like foo_bar.baz.com never be a valid hostname? Or should it just never be passed to idna encoding since underscore is a valid ascii character?

That is, underscores appear to be valid in ascii hostnames, but that conflicts with IDNs in some way?

[jribbens]

Underscores are not valid in ASCII hostnames, and never were. That fact has nothing to do with IDNA. Only letters, digits and hyphen are valid. See RFC 952 for details.

[bpicolo]

@jribbens Awesome, I see now, thanks!

"While a hostname may not contain other characters, such as the underscore character (_), other DNS names may contain the underscore" looks like that's more or less where my confusion lies. I'll close this ticket. Thanks for the weigh in guys! This wasn't my area of expertise. : )