SSL certificate verification fails in stats query

[jcharaoui]

Context

• XO origin: XO Applicance
• Versions:
  • Node: v8.9.4
  • xo-web: 5.17.0
  • xo-server: 5.17.0

Expected behavior

There should be no certificate validation errors when using a properly signed xapi certificate.

Current behavior

When managing servers for which unauthorized certificates is disabled, stuck tasks like the one below start to accumulate:

[XO] Xapi#getResource /rrd_updates (on <hostname>) 0%

In the logs there are error messages like this:

Hostname/IP doesn't match certificate's altnames: "IP: <host-ip> is not in the cert's list: "

I believe the origin of this problem is in xapi-stats.js, which seems to query hosts via IP address rather than FQDN.

It's possible to work around this problem by creating a custom pair of X509 certificates containing the IP address as a SubjectAltName, but you're out of luck if you get certificates through a recognized provider (eg. Let'sEncrypt) because they don't provide such certificates with IP addresses.

If xapi-stats.js was made to query hosts using the host's FQDN (like other parts of XO) it would solve this issue.

[olivierlambert]

IIRC, that's because XAPI returns an IP address and not a FQDN. From there, I don't know how we could deal with that.

[jcharaoui]

There may not be a straightforward solution but I think this doesn't reduce the importance of this issue. As I said, SSL certificates with IP addresses in Common Name or SAN are uncommon, so this problem is bound to pop up for other customers who need to use trusted certificates.

[olivierlambert]

As I said, we can't use anything else than XAPI is returning for URL to fetch. From there, I don't see any solution directly in XO...

[jcharaoui]

I don't really know much about XAPI, but unless I'm mistaken you get a list of hosts from masters using a host.get_all() call, right? If so, then that returns host objects containing address and hostname properties. You could use the latter, and infer the domain part from the master's FQDN.

So for example if you query hosts for pool master foo.example.net and it returns hostname bar, then you should query bar.example.net. For safety you could fallback to the host IP address if the FQDN doesn't respond.

[olivierlambert]

I'll take a look at the code, I think it's not "easy" as that. I think @badrAZ rewrote a bit the stats fetch code recently. He'll take a look after our next release.

[badrAZ]

I think the solution for this issue is to make the host's address editable. We will test this solution soon.

[jcharaoui]

Editable for what purpose? To add a domain suffix? There's already a XAPI parameters to get/set the host domain:
xe pif-param-get uuid=<management pif> param-name=other-config param-key=domain
As I understand it, this is valid per-host and not per-pool.

[julien-f]

@jcharaoui Nope, to be able to put the FQDN instead of the IP address in the address field of the host.

[jcharaoui]

Hopefully, by default it will be set to the hostname + domain suffix (as above).

[julien-f]

@jcharaoui We're not sure it's the correct solution, we are waiting for an answer from the XenAPI devs, in the meantime, do you have any documentation regarding the other_config.domain entry?

[olivierlambert]

Issue created here: https://bugs.xenserver.org/browse/XSO-853

[jcharaoui]

@julien-f Here: https://support.citrix.com/article/CTX118840

[olivierlambert]

It seems to allow hosts to resolve domains, but that's not what we need. We need to tell XAPI to give answer with the host FQDN and not the IP address. So we must find where we (if) could change that.

[jcharaoui]

I'm not really sure this can be solved at the XAPI level, because I don't think XenServer cares about it's own domain. It doesn't even really care about its own hostname, as evidenced by looking at /etc/hosts on a clean XS install:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Normally, a host's domain would appear here, instead of "localdomain". I don't know what other way XAPI would have of knowing its own domain.

Another idea would be to simply get the FQDN from the local DNS server using a dns.reverse(ip) call in nodejs. (This query could be made only when certificate verification is active.)

[olivierlambert]

I'll wait for Citrix answer before introducing something in XO :)

[jcharaoui]

I had a quick look inside XAPI and it appears like host.address is being set when XAPI starts up, from the dom0 management interface. Also from looking around elsewhere, it seems like other parts of XAPI expect this to be an actual IP address, for pool operations for example. So I'm not sure if this could really be changed from within the XAPI side, at least not wiithout significant effort.

[olivierlambert]

We changed it, but it didn't survived a reboot. So it's not the good way. Still hope for someone at Citrix answering…

[jcharaoui]

Yeah that's what I'm saying, when XAPI starts up it resets host.address in the database (as seen in the code).

[olivierlambert]

Yes, that's what I meant: you are right, it's not the good way :D

[jcharaoui]

What I'm also saying is that even if it survived a reboot, it might affect other XAPI operations, like leaving/joining a pool, there host.address is expected to be an IP.

[olivierlambert]

It's not possible that we can't use FQDNs at all in the end :/ Maybe I'm wrong or we missed something trivial.

[jcharaoui]

DNS to the rescue! 😆

[olivierlambert]

Answer from Citrix: won't fix. See: https://bugs.xenserver.org/browse/XSO-853

So there is little we could do 😞

[jcharaoui]

Answer from Citrix: won't fix. See: https://bugs.xenserver.org/browse/XSO-853

Yep, saw it. Can't say I'm surprised.

So there is little we could do disappointed

I disagree. There were several possible workarounds mentioned previously in this discussion, which are all relatively simple. I think it's worth exploring them because fixing the problem in XO will allow more users of XO to connect securely to their XS/XCP-ng hosts, which we can all agree is a good thing.

If the project decides not to fix it, then I think that at the very least the documentation should mention the fact that to work correctly, the SSL certificate must certify the host's FQDN and IP address. I could propose a documentation PR for that, if you like.

[olivierlambert]

That would be a good idea yes :) Your PR is very welcome! 👍