no valid transports or upstreams available

[zyrill]

Hi guys

I can't seem to get stubby to behave on my (linux) router. I'm running stubby successfully on my windows machine. But on the router, I always get errors when testing if the stubby works with dig @192.168.66.1 -p 453 www.heise.de.

Could you guys help me out pls? I'm sure I've just forgotten something, but I can't figure out what. I've also uploaded the error messages for better readability to https://pastebin.com/V696AZtQ

Googling for a few hours didn't provide me with a clue how to solve the problem, sadly - but I think it may have something to do with missing intermediate certificates? But where would I put those?

So here's the message:
stubby -l [16:36:54.381273] STUBBY: Read config from file /mod/root/.stubby.yml [16:36:54.383812] STUBBY: Starting DAEMON.... [16:36:57.602079] STUBBY: 145.100.185.15 : Conn opened: TLS - Strict Profile [16:36:57.695453] STUBBY: 145.100.185.15 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:57.700220] STUBBY: 145.100.185.15 : Conn closed : Transport=TLS - *Failure* [16:36:57.704930] STUBBY: 145.100.185.16 : Conn opened: TLS - Strict Profile [16:36:57.708250] STUBBY: 145.100.185.15 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:57.713019] STUBBY: 145.100.185.15 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:57.713692] STUBBY: 145.100.185.15 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:57.794389] STUBBY: 145.100.185.16 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:57.798158] STUBBY: 145.100.185.16 : Conn closed : Transport=TLS - *Failure* [16:36:57.801759] STUBBY: 185.49.141.37 : Conn opened: TLS - Strict Profile [16:36:57.803273] STUBBY: 145.100.185.16 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:57.803965] STUBBY: 145.100.185.16 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:57.804634] STUBBY: 145.100.185.16 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:57.857548] STUBBY: 185.49.141.37 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:57.859055] STUBBY: 185.49.141.37 : Conn closed : Transport=TLS - *Failure* [16:36:57.861521] STUBBY: 2001:610:1:40ba:145:100:185:15 : Conn opened: TLS - Strict Profile [16:36:57.862118] STUBBY: 185.49.141.37 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:57.864410] STUBBY: 185.49.141.37 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:57.870816] STUBBY: 185.49.141.37 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:57.937756] STUBBY: 2001:610:1:40ba:145:100:185:15 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:57.940717] STUBBY: 2001:610:1:40ba:145:100:185:15 : Conn closed : Transport=TLS - *Failure* [16:36:57.945270] STUBBY: 2001:610:1:40ba:145:100:185:16 : Conn opened: TLS - Strict Profile [16:36:57.947053] STUBBY: 2001:610:1:40ba:145:100:185:15 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:57.950262] STUBBY: 2001:610:1:40ba:145:100:185:15 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:57.953177] STUBBY: 2001:610:1:40ba:145:100:185:15 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:58.007956] STUBBY: 2001:610:1:40ba:145:100:185:16 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:58.009320] STUBBY: 2001:610:1:40ba:145:100:185:16 : Conn closed : Transport=TLS - *Failure* [16:36:58.011621] STUBBY: 2a04:b900:0:100::38 : Conn opened: TLS - Strict Profile [16:36:58.012227] STUBBY: 2001:610:1:40ba:145:100:185:16 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:58.013992] STUBBY: 2001:610:1:40ba:145:100:185:16 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:58.017795] STUBBY: 2001:610:1:40ba:145:100:185:16 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:58.064408] STUBBY: 2a04:b900:0:100::38 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:58.068870] STUBBY: 2a04:b900:0:100::38 : Conn closed : Transport=TLS - *Failure* [16:36:58.073570] STUBBY: 89.233.43.71 : Conn opened: TLS - Strict Profile [16:36:58.077700] STUBBY: 2a04:b900:0:100::38 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:58.082302] STUBBY: 2a04:b900:0:100::38 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:58.087956] STUBBY: 2a04:b900:0:100::38 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:58.164430] STUBBY: 89.233.43.71 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:58.169350] STUBBY: 89.233.43.71 : Conn closed : Transport=TLS - *Failure* [16:36:58.174020] STUBBY: 51.15.70.167 : Conn opened: TLS - Strict Profile [16:36:58.174591] STUBBY: 89.233.43.71 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:58.175285] STUBBY: 89.233.43.71 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:58.175885] STUBBY: 89.233.43.71 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0 [16:36:58.252260] STUBBY: 51.15.70.167 : Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate" [16:36:58.255366] STUBBY: 51.15.70.167 : Conn closed : Transport=TLS - *Failure* [16:36:58.258880] STUBBY: *FAILURE* no valid transports or upstreams available! [16:36:58.260495] STUBBY: 51.15.70.167 : Conn closed: TLS - Resps= 0, Timeouts = 0, Curr_auth = Failed, Keepalive(ms)= 0 [16:36:58.263636] STUBBY: 51.15.70.167 : Upstream : TLS - Resps= 0, Timeouts = 0, Best_auth = Failed [16:36:58.264774] STUBBY: 51.15.70.167 : Upstream : TLS - Conns= 0, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[wtoorop]

Yes, this is due to a missing certificate store.
What directory does openssl version -d return? Is there a cert directory underneath it?
Even though you might fix the issue by provisioning a store... I do think that it should not be needed when a sha256 pinset is given, so this is a bug that needs to be resolved. We have an getdns issue on this b.t.w. See: getdnsapi/getdns#293

[wtoorop]

Although it appears unrelated to getdnsapi/getdns#293 on the surface, it actually is not, and the solution mentioned in there should resolve this too. So that is why I labeled this duplicate.

[zyrill]

Thanks for the response, @wtoorop I don't have openssl installed, the only ssl related files on my (router) system are:
# find / | grep ssl
/bin/openssl_genrsa
/bin/openssl_req
/lib/libavmssl.so
/lib/libavmssl.so.2
/lib/libavmssl.so.2.0.0
/lib/libssl.so
/lib/libssl.so.1
/lib/libssl.so.1.0.0
/sys/module/kdsldmod/sections/.text.DHCPOPTIONS_classless_iproute_alloc
/sys/module/kdsldmod/sections/.text.DHCPOPTIONS_classless_iproute_free
/sys/module/kdsldmod/sections/.text.ipaccesslogging_string
/sys/module/kdsldmod/sections/.text.ipaccesslogging_parse
/usr/lib/freetz/libssl.so
/usr/lib/freetz/libssl.so.1.0.0
/var/flash/websrv_ssl_key.pem
/var/flash/websrv_ssl_cert.pem
/var/tmp/websrv_ssl_cert.pem

However, if you're looking for the path to where the (SSH?) keys are stored, I have this to offer:
# ll /var/mod/root/
0 drwxr-xr-x 3 root root 120 Nov 21 10:38 ./
0 drwxr-xr-x 11 root root 220 Jan 1 1970 ../
0 drwxr-xr-x 2 root root 40 Nov 21 10:38 .getdns/
0 lrwxrwxrwx 1 root root 23 Jan 1 1970 .profile -> /tmp/flash/mod/.profile
0 lrwxrwxrwx 1 root root 31 Jan 1 1970 .ssh -> /tmp/flash/authorized_keys_root/
0 lrwxrwxrwx 1 root root 21 Nov 15 10:25 .stubby.yml -> /tmp/flash/stubby.yml

Where does stubby look for the certificate store by default?

I might add that I have no idea how to provision a store that works with getdns. If that's a temporary fix, could somebody pls point me in the right direction?

[wtoorop]

Well... getdns uses the OpenSSL SSL_CTX_load_verify_locations function. It is hard to tell what the location would be without the openssl command line tool (maybe we could provide this information via the getdns_context_get_api_information function (i.e. -i option with stubby) in the next getdns release), but it is quite often /etc/ssl/certs.
Although it is a bug that you need these even if you have SPKI pinsets, it might still be a good idea to have the default locations configurable in getdns anyway (for upstreams which need to be authenticated by only name).
Oh... b.t.w. I forgot to mention, but you could bypass the error by disabling authentication too.
Of course you'd have only opportunistic DNS-over-TLS, but maybe better than nothing for the time being.

[Kdmeizk]

I had no choice but to reinstall Windows 7. Since this, Stubby gives a same message as in this bug:

Failure - (20) "unable to get local issuer certificate"

So, I assume it is related to this bug.

[saradickinson]

@Kdmeizk Hmm, a clean install of Stubby shouldn't give this error unless there is something wrong with the certificate. What version of Stubby are you using, how did you install and are you using the default Stubby config?

[Kdmeizk]

@saradickinson Ahaha, I tested until the latest optional update from Windows Update related to certificates and TLS, and it is this very last update which solved the problem. I will test more precisely later to find out what optional updates are concerned, or maybe you know them already?

It seems KB3172605 solved the problem, but I got it by installing other optional updates I believe. I will check.

PS: I have the latest version of Stubby for Windows (0.2.0), default/clean config

[saradickinson]

@Kdmeizk thanks for the info! I've added a link to this page as a known issue on the wiki for the Windows installer:
https://dnsprivacy.org/wiki/display/DP/Windows+installer+for+Stubby

[Kdmeizk]

@saradickinson Well, I thought wrong… I tried on a clean install but it did not work, even having done absolutely all the updates. I probably got lucky because I simply reset Windows 7 with the upgrade mode, so the necessary certificate was probably still here. I think the problem is the same as for curl. curl must have a certificate with him to work on Windows, so there is one provided with curl. It is probably the same for Stubby, which you are fixing in this bug if I understood?

[saradickinson]

The Stubby for Windows installer comes with its own certificate store which is why I don't think this is the same issue as the curl problem (where the system has no certificate store by default).
Does it give this message for all the upstreams or do you have a single upstream configured?
Also, I'm planning to update the Windows installer in the next couple of weeks with the 1.4 release of getdns which includes the option to use just a SPKI pin for an upstream (not requiring a cert) which would at least provide a workaround for you.

[Kdmeizk]

Does it give this message for all the upstreams

Yes it gives the same message for any upstream.

The Stubby for Windows installer comes with its own certificate store

How it works? Is this a file included with the others or Windows installer puts it in the Registry for example? Maybe I can check if this certificate exists after the installation of Stubby.

[jankal]

This is also related if one wants to setup cloudflares 1.1.1.1 DNS-over-TLS servers via Stubby on Windows.
Following configuration should work:

upstream_recursive_servers:
  - address_data: 1.1.1.1
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=

Unfortunatly it fails with a Verify failed : Transport=TLS - *Failure* - (20) "unable to get local issuer certificate"

This should really be fixed! Why doesn't Stubby just use the Windows certificate store to validate the TLS certificates? - Even just a cacerts file would do it, so the users can edit it and add the required certificates if they want to set up their own DNS-over-TLS server within Stubby.

@saradickinson @wtoorop

This is important!