Add Trusted Hosts Configuration

[JSanford42]

Several months ago, pub started throwing certificate errors. I have searched every way I can think of to get this issue resolved and have had no luck. I have discovered that it is an issue with the way the proxy is configured.

I would like the ability to specify trusted package sources in the pubspec.yaml file. The trusted hosts would not get stopped by certificate errors.

trusted_hosts:
  - pub.dartlang.org
  - pub.dev

The reason behind this is that I am on a network where the proxy behaves in an unusual manner. The proxy has its own certificate store. When it encounters a site for which it does not have the certificates, it uses an internal certificate chain for the connection between itself and my workstation. When it does have the certificates, the host certificates are "passed through" for the proxy to workstation connection.

This behavior has made it impossible for me to install or update packages. I have tried adding the certificates using the DART_VM_OPTIONS route with no success. I have verified that the certificates that I am adding are the same ones that the proxy has.

[jonasfj]

More context in dart-lang/sdk#45939

When it does have the certificates, the host certificates are "passed through" for the proxy to workstation connection.

Okay, it sounds as if the proxy allows TLS traffic through for some allow-listed domains, and that for the rest you the proxy terminates TLS and forwards traffic using a self-signed certificate.

In this case, the solution seems to be:

• Install self-signed certificate used the proxy on the system certificate store,
• Make Dart / pub load certificates from the system certificate store.

Am I understanding this wrong?

---

@JSanford42 Is it possible that you already have the self-signed certificate installed on the machine, but not a certificate store loaded by Dart SDK, see: dart-lang/sdk#45909

I'm guessing there will be a dev release available for testing next week -> https://dart.dev/tools/sdk/archive (2.14.0-85.0.dev as of writing doesn't look to have the possible fix)

[jonasfj]

To clarify: I'm not convinced we should add an option to ignore invalid HTTPS certificates.
If you really want that, I would suggest running a http <-> https proxy on localhost and using PUB_HOSTED_URL to direct dart pub to talk to the proxy.

Disabling HTTPS is dangerous, providing an easy options for doing so is likely undesirable.
Adding an option to remember and trust unknown certificate, in a trust-on-first-use kind of way, might be more viable :)

But if we could make installing into system certificate store work, that seems like a much better option.

[JSanford42]

Okay, it sounds as if the proxy allows TLS traffic through for some allow-listed domains, and that for the rest you the proxy terminates TLS and forwards traffic using a self-signed certificate.

That appears to be exactly what is happening. I am not privy to the network configuration but that is the behavior that I have seen with my testing.

In this case, the solution seems to be:

• Install self-signed certificate used the proxy on the system certificate store,
• Make Dart / pub load certificates from the system certificate store.

Am I understanding this wrong?

I have verified that the self-signed certificates used by the proxy are in the workstation certificate store but I am unable to determine if it is the system or machine store. I do not have administrator privileges so I can only see the certificates available to my user account.

I understand not wanting to ignore invalid HTTP certificates. I have seen so many posts in various places that have provided fruitless solutions to being behind a proxy that I am trying to get any changes made that will allow pub to work.

[JSanford42]

I'm guessing there will be a dev release available for testing next week -> https://dart.dev/tools/sdk/archive (2.14.0-85.0.dev as of writing doesn't look to have the possible fix)

I will pull down build 93 when it is available and test it out. I'll report my findings here.

[jonasfj]

I will pull down build 93 when it is available and test it out. I'll report my findings here.

Please do.. If we figure out a good solution for this we should document it here:
https://dart.dev/tools/pub/troubleshoot#pub-get-fails-from-behind-a-corporate-firewall

You're not the only one with peculiar network conditions.

[jonasfj]

@JSanford42
Can you tryout 2.14.0-90.0.dev from https://dart.dev/tools/sdk/archive#dev-channel

[JSanford42]

@jonasfj I pulled down build 90 this morning and it still does not work. Progress has been made though. It seems to be able to get the local issuer certificate now but it thinks it is expired. I went to https://pub.dartlang.org/api/packages/pedantic and checked all certificates in the browser and none of the ones in use are expired.

Previous Error

CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate

New Error

CERTIFICATE_VERIFY_FAILED: certificate has expired

I looked in the certificate store that I am able to see from the browser. I have several intermediate certificates in there that are expired but none of them are being used by the browser when I connect to the URL above.

[jonasfj]

@JSanford42, well, that's certainly progress.

I'm guessing we need to figure how to reproduce this. Can you create a custom HTTPS server on localhost using dart:io (or python) which uses a self-signed certificate that installed on the machine and works in the browser, but not in dart:io?

If we can do something like that, maybe we can ask some of the network people to look into it. Or maybe I can figure out how to build Dart SDK for windows and tweak: https://dart.googlesource.com/sdk//+/acf842c56f0c959588241d592dbcab18f1830322/runtime/bin/secure_socket_utils.h#22

---

Just guessing here.. maybe the certificate exists in multiple stores on the system, but one of them is expired, or maybe there is something else wrong. I think we need to reproduce on a clean Windows VM in the cloud to make much progress.

This could be certificate loading not working correctly, or it could be misconfiguration of the system.

[JSanford42]

I did some more digging now that all of the certificate stores are being used. I made a bare bones HTTP client for the pedantic package URL that prints the certificate information in the bad certificate callback. It shows that one of the intermediate certificates is expired.

I looked in the certificate store and found 3 with the same name (e.x. Signing CA 6). Two are expired and one is not.

Just saw your comment after this post. I think the certificate is loading correctly but the first one found is being used. I think, in this case, the first one found is expired.

[JSanford42]

I'm guessing we need to figure how to reproduce this. Can you create a custom HTTPS server on localhost using dart:io (or python) which uses a self-signed certificate that installed on the machine and works in the browser, but not in dart:io?

I created a new cert and added it to the current user certificate store as trusted. I created a small HTTP server in Dart and used the self signed cert and key.

As expected, I got the issuer certificate error when running under 2.12.4. I did not receive any certificate errors when running under 2.14.0-90.0.dev and saw the response text the server returns.

[jonasfj]

I think the certificate is loading correctly but the first one found is being used. I think, in this case, the first one found is expired.

Could the order in which they are inserted into the store be sufficient to reproduce this?
If so, that would be a great bug report for the Dart SDK.

I'm not sure if this is a problem in BoringSSL, how Dart does TLS, or maybe just because we're loading expired certificates when non-expired certificates are available.

/cc @aam

[aam]

When it comes to using Windows certificate stores dart runtime simply loads all certificates from current user and local machine certificate stores into boringssl(https://github.com/dart-lang/sdk/blob/master/runtime/bin/security_context_win.cc#L114) and relies on boringssl to do end-point certificate verification. Not sure why boringssl would be confused by presence of expired certificate if non-expired is present.
I would imagine though that end-point certificate specifies exact issuer certificate which effectively repeats the certificate verification process for the issuer certificate, all the way up the chain until it reaches a certificate that is present in trusted root store (in Windows store, in boringssl).

We also recently added an dart vm option --bypass_trusting_system_roots which bypasses use of Windows system certificate store altogether and relies on dart vm builtin list of certificates. 2.14.0-90.0.dev should have that change too.

[aam]

as @jonasfj pointed out bypassing system store might not be an option for you. In that case there is another command line option available: --root-certs-file=<filename> where filename is something that is given to boringssl via https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_load_verify_locations.html

See #1882 (comment)

[jonasfj]

Hmm, I was able to use --root-certs-file= as follows:

openssl s_client -showcerts -partial_chain -connect pub.dev:443 < /dev/null >> trusted-certificates.txt
openssl s_client -showcerts -partial_chain -connect storage.googleapis.com:443 < /dev/null > trusted-certificates.txt

dart --root-certs-file=trusted-certificates.txt pub upgrade --verbose

This is probably not the best of ideas, as I didn't isolate the root certificate I needed, I just dumped all the certificates into a trusted-certificates.txt file 🙈

---

But now that we have dart pub get, it's obvious that the old hack we had with DART_VM_OPTIONS no longer work (not that we really advertised it widely).

But perhaps it's time we highlight how to use --root-certs-file= on https://dart.dev/tools/pub/troubleshoot

[JSanford42]

I can't explain why but everything is working this morning with 2.14.0-90.0.dev. I checked my certificate stores and they are in the same state as yesterday. The expired certs that were causing pub to trip up are still present but they seem to not be interfering anymore.

I have not made any changes to my test code. The only thing I did this morning was test the --root-certs-file option on the command line. I verified that the certs in the file are the only ones used. The Dart cert bundle and the cert stores are ignored when a file is provided.

It might be worth keeping an eye out for others that may get the expired certificate issue in the future. There may be an intermittent issue with how either Dart or BoringSSL are handling the certificate inputs.

[JSanford42]

pub failed again this morning. I discovered that the proxy certs were updated but they had not been pushed to my machine yet. I got the latest cert bundle and ensured that all certs were installed. The expired certificate error returned after the update.

I also tested using the new cert bundle using --root-certs-file. The file uses has only the most recent certs. Some of the HTTPS connections change to using the previous cert, which is still valid for a few more days. This makes some package downloads throw the "unable to get local issuer certificate" error.

[JSanford42]

Prior to May 19, I had two certificates with the same name that were both valid and two that were expired. I thought that maybe having 2 valid certificates was causing an issue. I waited until only one was valid and the other 3 were expired before testing again. I am still getting the expired certificate issue.

I tested again with the cert bundle from my internal CA using --root-certs-file and I am getting the local issuer certificate error for some packages but not others.

[JSanford42]

I have been downloading development builds each time one is posted on dart.dev and this issue is still present. I can only conclude that this point that pub is getting tripped up on the first bad certificate.

Here is the code I am using to test the connection each time pub fails after a new build.

import 'dart:io';

void main(List<String> arguments) {
  var client = HttpClient();

  client.badCertificateCallback = (cert, host, port) {
    print('Bad certificate connecting to $host:$port:');
    _printCertificate(cert);
    print('');
    return true;
  };
  client
      .getUrl(Uri.parse('https://pub.dartlang.org/api/packages/pedantic'))
      .then((request) => request.close())
      .then((response) {
    print('Response certificate:');
    _printCertificate(response.certificate);
    response.drain();
    client.close();
  });
}

void _printCertificate(X509Certificate? cert) {
  if (cert != null) {
    print('${cert.issuer}');
    print('${cert.subject}');
    print('${cert.startValidity}');
    print('${cert.endValidity}');
  } else {
    print('Certificate is null');
  }
}

Running this code produces this result. (I had to mask some of the names with XXX due to policy.)

Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Intermediate CA 1
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
2020-01-06 17:59:49.000Z
2020-07-06 17:59:49.000Z

Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

Response certificate:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

The first certificate displayed is expired but the second one is not. I have tried this URL in the browser and the valid certificate here is the one being used.

[jonasfj]

I can only conclude that this point that pub is getting tripped up on the first bad certificate.

Yeah, I think dart in general will fail TLS connections if there is an expired certificate for the domain and this is picked first.

@JSanford42, so if you use --root-certs-file, and provide a file without the expired certificates then things should work, correct?

The issue about dart just picking the first certificate when certificates further down might not be expired, seems like something for dart-lang/sdk as a dart:io bug. Ideally, it's reproducible using --root-certs-file :D

[JSanford42]

@JSanford42, so if you use --root-certs-file, and provide a file without the expired certificates then things should work, correct?

When I run the example code with a cert bundle that does not have the expired certificates it does work. However, I cannot rely on that at all times. The certificates used by the proxy change frequently. The internal certificates are pushed to the machine store regularly in my environment but not published for download as frequently.

I will post this issue in the dart-lang/sdk repo and reference this issue.

[jonasfj]

When I run the example code with a cert bundle that does not have the expired certificates it does work. However, I cannot rely on that at all times.

Yeah, I can see that it's not a good solution.

[JSanford42]

This issue has been resolved by commit dart-lang/sdk@aa0a441 in the dart SDK. It has been merged to master and will be included in future releases.