Be systematic about parsing and validating hostnames and addresses

[amenonsen]

This adds a parse_address(pattern) utility function that returns
(host,port), and uses it wherever where we accept IPv4 and IPv6
addresses and hostnames (or host patterns).

This is a proof of concept for how I think it should be done. Some changes are needed: for example, we may want to be a bit more restrictive about hostnames (e.g. "foo-" should be disallowed), and a few other details may need to be cleaned up. But this is a solid foundation, and it should be backwards-compatible as well.

See also discussion in #11747

I welcome feedback/reviews/testing.

cc: @abadger @srvg @ryanpetrello

[amenonsen]

See also the later discussion in #12108 regarding defining myUser@127.0.0.1:1234 in the inventory instead of setting ansible_ssh_user=myUser. This worked purely by accident earlier (the port was removed, and myUser@127.0.0.1 was passed to ssh(1) as-is.) Because ansible didn't know about the user, it would happily set -o User=otherUser if, for example, there were a conflicting group var setting.

@jimi-c said that we don't want to support this; the code in this PR already rejects the notation.

(Aside: if desired, however, "proper" support could easily be added on the basis of this PR, i.e. to detect the user@ prefix, remove it, and set ansible_ssh_user for the host.)

[jimi-c]

@amenonsen agreed, the support could be added if someone wanted to, in which case we would support it.

[amenonsen]

I did a bit more testing and implemented more precise validation of hostname labels (e.g. to enforce the DNS rule that a label can't end with a hyphen, i.e. foo- isn't valid).

At this point, my suggestion would be to merge this during the 2.0 alpha period so that we can see if there are any more obscure/undocumented/broken inventory definitions in the wild (such as user@127.0.0.1:1234, discovered by accident recently) and take a decision about whether to support them, or warn about them in the changelog, or whatever. I'm confident that it's easier to extend this code to support odd cases than it would have been to do with the earlier code, and the test cases will keep us more honest about inventory hostname parsing in general.

[srgvg]

FYI

I checked this PR on my large inventory by dumping everything into a json file. Dump from

• ansible 2.0.0 (devel 7ece767) last updated 2015/09/03 19:40:16 (GMT +200)
• and this PR
  yield an identical export

[amenonsen]

@chrrrles I believe you were looking at this PR. Did you come to any conclusions?

[chrrrles]

We should also test ranges of ipv6 addresses with a port, ie:

        '[abcd:ef98::7654:[1:9]]:2222': ['[abcd:ef98::7654:[1:9]]:2222', None]

[amenonsen]

(I've added this test in the latest version.)

[amenonsen]

I've squashed this PR down to three commits (but if it's merged, I request that it not be squashed further; I believe there's value in having the hostname parsing changes separated for better understanding and future debugging).

[amenonsen]

@chrrrles mentioned on IRC that this patch apparently caused an integration test failure. I had a quick look. The test in question is from test/integration/unicode.yml, which does the following:

    - add_host:
        name: '{{hostnames}}.{{item}}'
        groups: 'ĪīĬĭ'
        ansible_connection: local
        host_id: '{{item}}'
      with_sequence: start=1 end={{num_hosts}} format=%d

First, there's a trivial missing import in add_host.py; once that's fixed, we see that the '{{hostnames}}.{{item}}' syntax used for name: above doesn't actually work (and it appears to be the only place in the tests where this notation is used). It expands hostnames as an array, then adds .1, and passes the resulting string to add_host, which quite sensibly rejects it. So I think this is a clearly broken test.

If I change this to use, e.g. hostnames[2], the test passes. But note that the actual values in hostnames are a bit weird:

    hostnames:
        - 'host-#ϬϭϮϯϰ'
        - 'host-ͰͱͲͳʹ͵'
        - 'host-ΙΚΛΜΝΞ'
        - 'host-στυφχψ'
        - 'host-ϬϭϮϯϰϱ'

I can understand supporting Unicode word characters in inventory hostnames for convenience (though it's not very convenient in practice, because you have to set ansible_ssh_host for the names to be usable), but I see no reason to support Unicode punctuation characters in hostnames (e.g. hostnames[1] above, plus the # in hostnames[0]). The last three values all pass the test with a trivial two-line change to my parsing code in this PR (i.e. change a-z to \w and set the re.UNICODE flag).

[amenonsen]

@chrrrles @abadger I've rebased, fixed the import error, made the a-z → \w change (and added a couple of Unicode unit tests), squashed down to two clean commits, and updated the PR. I have also pushed an additional commit to fix the broken integration test.

(BTW, I can't help but feel a bit pleased that the sum total of changes required to make the new code support unicode inventory hostnames was two lines.)

AFAIK, I've addressed all the substantive comments on this PR, and it's ready to merge now.