Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname' #7660

danopia · 2020-09-24T18:37:55Z

HTTPS fetch with bare IPv4 address fails:

> deno run --allow-net https://deno.land/std@0.70.0/examples/curl.ts https://1.1.1.1/
error: Uncaught Http: error sending request for url (https://1.1.1.1/): error trying to connect: invalid dnsname
    at Object.jsonOpAsync (core.js:236:13)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1272:29)
    at async https://deno.land/std@0.70.0/examples/curl.ts:3:13

HTTPS fetch with bare IPv6 address fails:

> deno run --allow-net https://deno.land/std@0.70.0/examples/curl.ts https://[2606:4700:4700::1111]
error: Uncaught Http: error sending request for url (https://[2606:4700:4700::1111]/): error trying to connect: invalid dnsname
    at Object.jsonOpAsync (core.js:236:13)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1272:29)
    at async https://deno.land/std@0.70.0/examples/curl.ts:3:13

Real curl is fine:

> curl -sI https://[2606:4700:4700::1111] | head -n1
HTTP/1.1 200 OK

> curl -sI https://1.1.1.1/ | head -n1
HTTP/1.1 200 OK

HTTPS fetch with DNS name that resolves to said IP addresses works:

> deno run --allow-net https://deno.land/std@0.70.0/examples/curl.ts https://one.one.one.one | grep \\.\\.
............................................................
.........1............1............1............1...........
........11...........11...........11...........11...........
.......111..........111..........111..........111...........
......1111.........1111.........1111.........1111...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11....ooo....11....ooo....11....ooo....11...........
......111111..ooo..111111..ooo..111111..ooo..111111.........
............................................................

> deno --version
deno 1.4.1
v8 8.7.75
typescript 4.0.2

The text was updated successfully, but these errors were encountered:

junlarsen · 2020-10-01T15:06:28Z

I'm trying to do something similar and I believe this issue has its roots deep down the Rust ecosystem.

Deno's fetch() is implemented in the op_crates/fetch crate which uses reqwest to perform the http requests.

Inside that Cargo.toml, we see that the reqwest build uses rustls (https://github.com/denoland/deno/blob/master/op_crates/fetch/Cargo.toml#L18)

There is currently a limitation using Rustls when sending http requests to plain ip addresses (see rustls/rustls#281) which seems to stem from briansmith/webpki#54, an issue which has been open for 3 years.

It seems there's not much you can do right now to get around this limitation apart from using a third-party request library which doesn't use rustls

CGQAQ · 2020-10-02T01:47:23Z

https://github.com/http-rs/surf
This one doesn't rely on rustls, but it uses async-std instead of tokio

CGQAQ · 2020-10-02T01:56:56Z

https://github.com/async-email/async-native-tls
This one is more promising

CGQAQ · 2020-10-02T03:23:28Z

reqwest work perfect

Still won't work for GKE and other clusters until something happens for denoland/deno#7660

bobbyg603 · 2021-01-02T23:31:23Z

Any plans to support this with the native fetch implementation? This would be really nice for home automation projects.

Edit: in the meantime, anyone who runs into this can create an entry in their hosts file as a workaround.

somethingelseentirely · 2021-01-03T13:28:23Z

I think it's reasonable to look for alternatives to rustls that don't rely on webpki.

Even if the dns issue was resolved, there are other issues that seem to have just been abandoned, like support for self signed certificates.

lucacasonato · 2021-01-03T13:41:51Z

I think it's reasonable to look for alternatives to rustls that don't rely on webpki.

rustls is an essential part of Deno as it is the entire backbone of all of our TLS operations. Replacing it would be a large refactor. The only viable alternative to rustls is OpenSSL, but using that has some downsides and comes with questions itself (eg. do we statically or dynamically link it?).

there are other issues that seem to have just been abandoned, like support for self signed certificates.

Deno supports self signed CA certificates through all available TLS APIs: --cert and DENO_CERT for all internal HTTP requests, caData option in Deno.createHttpClient for fetch, certFile option for Deno.connectTls, certFile option for Deno.listenTls. What exactly are you missing?

Also I want to note that Brian seems to have continued work on this in WebPKI, so maybe we will see some progress soon.

somethingelseentirely · 2021-01-03T20:32:49Z

rustls is an essential part of Deno as it is the entire backbone of all of our TLS operations. Replacing it would be a large refactor. The only viable alternative to rustls is OpenSSL, but using that has some downsides and comes with questions itself (eg. do we statically or dynamically link it?).

Yeah there's no winning when it comes to TLS 😑😔

What exactly are you missing?

Don't get me wrong I think you guys did a great job! ❤️ ✨

My specific use case involves shipping a binary (thanks to deno compile! ✨ ), to end users
that might have "whoknowswhat almost broken, but accepted by browsers" certificates on their machines, and expect them to work. 😕

Native IP addresses not working makes this already pretty difficult 😅.
This is complicated by rustls being such a stickler.
It wasn't really clear from the merge request I linked, but webpki flat out rejects self signed certs with the CA flag, with a CAUsedAsEndEntity error.
For my own certs that's fine, but I fear that there's a large-ish number of end users with self signed CA certificates that just copy pasted them from some tutorial. 😩

Anyways, looks like there's just waiting that Brian becomes more responsive,
so let's hope for the best 😄😅.

lucacasonato · 2021-01-03T20:53:32Z

It wasn't really clear from the merge request I linked, but webpki flat out rejects self signed certs with the CA flag, with a CAUsedAsEndEntity error.
For my own certs that's fine, but I fear that there's a large-ish number of end users with self signed CA certificates that just copy pasted them from some tutorial.

This is untrue for Deno. We have tests validating that this works. You can use self signed CA certificates with Deno for all our TLS related APIs.

somethingelseentirely · 2021-01-03T20:59:22Z

We have tests validating that this works.

Famous last words 😆

$deno test --cert "/Users/jp/.minio/certs/public.crt" --allow-net test/test_triblemq.js
Check file:///Users/jp/Desktop/triblesspace/tribles-deno/$deno$test.ts
running 1 tests
test Check loopback. ... Connected to ws://127.0.0.1:8816.
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
[
  TypeError: error sending request for url (https://localhost:9000/denotest/a15bb5cc5dea3b83a94e42d391a2309f1b4523ac07bad6f84b8ea681fd0e695e): error trying to connect: invalid certificate: CAUsedAsEndEntity
    at processResponse (deno:core/core.js:223:11)
    at Object.jsonOpAsync (deno:core/core.js:240:12)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
    at async S3Bucket.putObject (bucket.ts:445:18),
  TypeError: error sending request for url (https://localhost:9000/denotest/9fd29d38aa625a97f70aa22f7dbec4eef1cf836587e28bc6f1eb236f8f06b241): error trying to connect: invalid certificate: CAUsedAsEndEntity
    at processResponse (deno:core/core.js:223:11)
    at Object.jsonOpAsync (deno:core/core.js:240:12)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
    at async S3Bucket.putObject (bucket.ts:445:18)
]
error: Uncaught (in promise) TypeError: error sending request for url (https://localhost:9000/denotest/2c22dc020c9fe79a0483e86dc1a23d6cf3a8f815e4802699599a8311f0142a2f): error trying to connect: invalid certificate: CAUsedAsEndEntity
    at processResponse (deno:core/core.js:223:11)
    at Object.jsonOpAsync (deno:core/core.js:240:12)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
    at async S3Bucket.putObject (bucket.ts:445:18)

danopia · 2021-01-03T21:11:35Z

Hello, this exact problem is probably off-topic for this thread, which is that Deno cannot connect to IP address hosts using HTTPS.

However, I can confirm that Deno has the above message when you try using a CA certificate directly on a server. I have a personal CA and just loaded up Deno HTTPS server using it. Resulting in:

> deno run --allow-net --cert /tmp/ca.crt https://deno.land/std@0.83.0/examples/curl.ts https://localhost:4443
Sending fatal alert BadCertificate
error: Uncaught (in promise) TypeError: error sending request for url (https://localhost:4443/): error trying to connect: invalid certificate: CAUsedAsEndEntity
    at processResponse (deno:core/core.js:223:11)
    at Object.jsonOpAsync (deno:core/core.js:240:12)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
    at async https://deno.land/std@0.83.0/examples/curl.ts:3:13

That being said, Google Chrome is also refusing to establish a connection:

This site can’t be reached
The webpage at https://localhost:4443/ might be temporarily down or it may have moved permanently to a new web address.
ERR_SSL_KEY_USAGE_INCOMPATIBLE

as well as curl:

> curl --cacert /tmp/ca.crt https://localhost:4443 
curl: (60) SSL certificate problem: unsupported certificate purpose
More details here: https://curl.se/docs/sslcerts.html

Wow, it looks like wget's only problem is the hostname being wrong 😅

> wget --ca-certificate /tmp/ca.crt https://localhost:4443
--2021-01-03 22:12:24--  https://localhost:4443/
Loaded CA certificate '/tmp/ca.crt'
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:4443... connected.
The certificate's owner does not match hostname ‘localhost’

Surely I'd be able to golf a bit with the certificate's usage flags to get one that works elsewhere, and while I'm there I could fix the CA flag as well.

It's unfortunate that the rust TLS ecosystem is acting a bit stagnant. I personally am quite blocked by the IP address issue, to the point where I'm writing code like Deno.run({cmd: ["curl", "https://....."]}). Hopefully some movement happens with the recent work on webpki's repo.

i1u5 · 2021-03-28T22:31:34Z

Any updates about the progress of a possible fix for this issue, it seems to have been slept on.

lucacasonato · 2021-03-28T22:39:50Z

Any updates about the progress of a possible fix for this issue, it seems to have been slept on.

This has not been slept on. This is blocked on upstream missing support in rustls / webpki. If you want to accelerate this work I suggest you sponsor @ctz and @briansmith.

lucacasonato · 2021-03-28T22:40:56Z

We have tests validating that this works.

Famous last words

@somethingelseentirely CA cert and TLS cert can not be the same - in you case it looks like they are.

cryptogohan · 2021-05-24T16:08:57Z

Going to see if I can drive this forward a little bit, because.. well.. I love Deno, but this was rather painful for me.

So as I understand it, connecting to a Postgres Database with no DNS, just a hostname, using TLS, currently fails.

Small code sample:

await Deno.connectTls({
  hostname: "34.76.80.151",
  port: 5432,
  certFile: "server-ca.pem",
});

Let's see if we can get that one step closer to working 💪 !

cryptogohan · 2021-05-24T16:10:25Z

Straight off it's kinda odd that the docs give examples that don't work, and the default argument to Deno.connectTls will result in an error but rather than create PRs for that let's keep pushing in the other direction 😅 , making it work!

danopia · 2021-08-25T13:54:30Z

in the meantime, anyone who runs into this can create an entry in their hosts file as a workaround.

Based on this quote and my own use-case for this feature, perhaps a more practical step forward would be adding a parameter to replicate the servername option in Node.JS. The goal being that I could tell Deno what DNS name I expect the certificate to be good for. Then rustls should be able to validate successfully from there.

In the GKE (Kubernetes) situation I can expect that a cluster IP address certificate is also always good for kubernetes.default.svc.
In home automation perhaps the certificate is also good for localhost or similar.

I'd be ok telling Deno those servernames ^^ if it means avoiding 'invalid dnsname'.

Of course the title issue should still be fixed, but upstream movement on it seems to be pretty slow.

base698 · 2022-04-05T16:52:16Z

If a DNS record has multiple entries can you force Deno in the fetch to try another entry?

berkant · 2022-04-27T18:35:58Z

If a DNS record has multiple entries can you force Deno in the fetch to try another entry?

I suppose the answer is no for the time being. We'll have to wait before Deno.CreateHttpClientOptions matures and gets a field to override the DNS resolution process for fetch calls.

danopia · 2023-01-25T08:59:03Z

I've been quiet for a year or so :) just checking in.

It looks like the error message is different nowadays (Deno 1.29.1) for IPv4: presented server name type wasn't supported

$ deno run --allow-net https://deno.land/std@0.70.0/examples/curl.ts https://1.1.1.1/
Sending fatal alert BadCertificate
error: Uncaught (in promise) TypeError: error sending request for url (https://1.1.1.1/): error trying to connect: presented server name type wasn't supported

Interestingly, IPv6 IPs still just report invalid dnsname 🤔

And, in fact, the request does now work with --unsafely-ignore-certificate-errors given to Deno, though I don't plan on using that flag in practice. Providing an alternate servername would be a safer workaround ^_^

Looking upstream, it seems rustls 0.21.0 will include a step towards this issue. I look forward to seeing where that goes:

Can't connect to a server presenting a certificate with an IP address in its common name rustls/rustls#184 (comment)

danopia · 2023-01-30T11:40:46Z

I forgot to mention a workaround: Deno.connect and Deno.startTls can be used together to provide different hostname values for the TCP and TLS aspects of a connection, instead of Deno.connectTls which uses the same hostname for both aspects.

Minimal example with https://1.1.1.1:

const tcp = await Deno.connect({ hostname: '1.1.1.1', port: 443 }); // the host we want to connect to
const tls = await Deno.startTls(tcp, { hostname: 'one.one.one.one' }); // the host we want to handshake with

// Send a raw HTTP request, and print the raw HTTP response:
await tls.write(new TextEncoder().encode('GET /healthz HTTP/1.1\r\nHost: 1.1.1.1\r\nConnection: close\r\n\r\n'));
for await (const text of tls.readable.pipeThrough(new TextDecoderStream())) {
  console.log(text);
}

Also, I published /x/socket_fetch which implements a small subset of HTTP/1.1 in Typescript and can leverage Deno.startTls() as demonstrated above. Minimal example, again with https://1.1.1.1:

import { fetchUsing, TlsDialer } from "https://deno.land/x/socket_fetch@v0.1/mod.ts";

const dialer = new TlsDialer({ hostname: "one.one.one.one" });
const resp = await fetchUsing(dialer, "https://1.1.1.1/");
console.log(await resp.text());

This socket_fetch module is very limited (no POST, no socket reuse, etc) but works for some basic use-cases. I've been using it for HTTP GETs within Kubernetes clusters for the past year.

ry · 2023-03-30T01:40:10Z

https://www.memorysafety.org/blog/rustls-new-features/

The first big feature is support for TLS certificates containing IP addresses. Rustls can now be used to set up TLS connections addressed by IP rather than a domain name. This is useful for things like Kubernetes pods, which often use IP addresses instead of domain names, and for DNS over HTTPS/TLS which need an IP address for the server to avoid circular dependency on name resolution. TLS certificates for IP addresses have been the most heavily requested feature for quite a while now and it's great to have it completed.

This commit adds support for connecting to IP addresses over HTTPS. This is done by updating "rustls" to "0.21.0" and other related crates. Closes #7660 Closes #17967 --------- Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>

johnspurlock · 2023-05-17T01:05:55Z

thank you!!

This commit adds support for connecting to IP addresses over HTTPS. This is done by updating "rustls" to "0.21.0" and other related crates. Closes #7660 Closes #17967 --------- Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>

kitsonk added bug Something isn't working correctly cli related to cli/ dir labels Sep 24, 2020

alexanderGalushka mentioned this issue Sep 25, 2020

bug: Connection refused (os error 111) #7686

Open

junlarsen mentioned this issue Oct 1, 2020

deno: invalid hostname when using 127.0.0.1 junlarsen/league-connect#36

Closed

CGQAQ mentioned this issue Oct 2, 2020

fix #7660 #7789

Closed

junlarsen mentioned this issue Oct 2, 2020

Fetching any plain ip address over https fails due to rustls error #7796

Closed

kidonng mentioned this issue Oct 29, 2020

error sending request for welcome.ts : error trying to connect: invalid dnsname #5296

Closed

danopia mentioned this issue Dec 25, 2020

feat: Passing certificate data directly to HttpClient #8190

Closed

danopia added a commit to cloudydeno/deno-kubernetes_client that referenced this issue Dec 25, 2020

kubeconfig: Add the actual client code

23aedd8

Still won't work for GKE and other clusters until something happens for denoland/deno#7660

bobbyg603 mentioned this issue Jan 2, 2021

webpki/cert: extend verify_is_valid_for to iPAddress SAN briansmith/webpki#54

Open

metaquanta mentioned this issue Jan 27, 2021

Attempt to open TLS connection with self-signed certificate fails with "InvalidData: invalid certificate: BadDER" #9296

Closed

cryptogohan mentioned this issue May 21, 2021

Add flag to ignore SSL certificate issues #1371

Closed

lucacasonato added the upstream Changes in upstream are required to solve these issues label Jul 20, 2021

albnnc mentioned this issue May 14, 2022

Ability to ignore IP-address certificate errors #14609

Closed

danopia mentioned this issue Feb 9, 2023

Missing parameter servername in interface Deno.ConnectTlsOptions, which may be used for dns over tls. #14530

Closed

loynoir mentioned this issue Mar 26, 2023

npm:better-sqlite3 npm:node-libcurl not working #18444

Closed

levex self-assigned this Mar 28, 2023

bartlomieju mentioned this issue Mar 30, 2023

fix: support "fetch" over HTTPS for IP addresses #18499

Merged

bartlomieju closed this as completed in #18499 May 17, 2023

quite4work added a commit to quite4work/deno-kubernetes_client that referenced this issue Jun 3, 2023

Remove check for denoland/deno#7660

ffff1dd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname' #7660

Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname' #7660

danopia commented Sep 24, 2020 •

edited

Loading

junlarsen commented Oct 1, 2020

CGQAQ commented Oct 2, 2020

CGQAQ commented Oct 2, 2020

CGQAQ commented Oct 2, 2020

bobbyg603 commented Jan 2, 2021 •

edited

Loading

somethingelseentirely commented Jan 3, 2021

lucacasonato commented Jan 3, 2021 •

edited

Loading

somethingelseentirely commented Jan 3, 2021

lucacasonato commented Jan 3, 2021

somethingelseentirely commented Jan 3, 2021

danopia commented Jan 3, 2021 •

edited

Loading

i1u5 commented Mar 28, 2021 •

edited

Loading

lucacasonato commented Mar 28, 2021

lucacasonato commented Mar 28, 2021

cryptogohan commented May 24, 2021

cryptogohan commented May 24, 2021

danopia commented Aug 25, 2021 •

edited

Loading

base698 commented Apr 5, 2022

berkant commented Apr 27, 2022 •

edited

Loading

danopia commented Jan 25, 2023 •

edited

Loading

danopia commented Jan 30, 2023

ry commented Mar 30, 2023

johnspurlock commented May 17, 2023

Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname' #7660

Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname' #7660

Comments

danopia commented Sep 24, 2020 • edited Loading

junlarsen commented Oct 1, 2020

CGQAQ commented Oct 2, 2020

CGQAQ commented Oct 2, 2020

CGQAQ commented Oct 2, 2020

bobbyg603 commented Jan 2, 2021 • edited Loading

somethingelseentirely commented Jan 3, 2021

lucacasonato commented Jan 3, 2021 • edited Loading

somethingelseentirely commented Jan 3, 2021

lucacasonato commented Jan 3, 2021

somethingelseentirely commented Jan 3, 2021

danopia commented Jan 3, 2021 • edited Loading

i1u5 commented Mar 28, 2021 • edited Loading

lucacasonato commented Mar 28, 2021

lucacasonato commented Mar 28, 2021

cryptogohan commented May 24, 2021

cryptogohan commented May 24, 2021

danopia commented Aug 25, 2021 • edited Loading

base698 commented Apr 5, 2022

berkant commented Apr 27, 2022 • edited Loading

danopia commented Jan 25, 2023 • edited Loading

danopia commented Jan 30, 2023

ry commented Mar 30, 2023

johnspurlock commented May 17, 2023

danopia commented Sep 24, 2020 •

edited

Loading

bobbyg603 commented Jan 2, 2021 •

edited

Loading

lucacasonato commented Jan 3, 2021 •

edited

Loading

danopia commented Jan 3, 2021 •

edited

Loading

i1u5 commented Mar 28, 2021 •

edited

Loading

danopia commented Aug 25, 2021 •

edited

Loading

berkant commented Apr 27, 2022 •

edited

Loading

danopia commented Jan 25, 2023 •

edited

Loading