DNS issue resolving external domains #18
Comments
|
GCP Logging have the same issue |
|
Http vs https shouldn't be the issue as we don't hijack those hostnames. Can you run "dig cloudtrace.googleapis.com" or see what IP it resolves to in the container on Cloud Run? |
|
I used your example code and results below. From logs |
|
Looks like
|
|
I'm asking these because I have an example container that runs and when I run it, curl works fine:
|
func print(w http.ResponseWriter, req *http.Request) {
file, err := os.Open("/etc/resolv.conf")
if err != nil {
fmt.Fprintf(w, "dig failed: %v\n", err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
b, err := ioutil.ReadAll(file)
fmt.Fprint(w, string(b))
}Output of file is Dig from example from this code repo Curl output |
|
Weird enough I have the exact
and Do you spot any differences? |
|
The only one difference I see is region us-central1 vs europe-west1 and in your screenshot there is no |
|
I updated my image, works with --http2 as well:
|
|
Do you mind trying another region? |
|
Sure, on us-central1 not working too, the same results, only /etc/resolv.conf is different (as expected). I'll try to run this on my private account, not corp one which is in organization but at this moment I don't see reason of this issue |
|
I deployed it on my private account on private project and the same result, not working. I was thinking that maybe it's Docker image issue but now I'm not sure. Image with example I'm building locally with docker-compose on wsl2. |
|
When I used dig with @8.8.8.8 I got results some results /dig?domain=cloudtrace.googleapis.com@8.8.8.8 |
|
that's because it didn't parse it correctly. :) @8.8.8.8 is a separate argument. :) |
|
Oh, I was hoping that we moved closer with an issue:D For now I have no idea what's going on. |
|
Can you perhaps try deploying to |
|
I did previously and on my private account it was us-central1. |
|
Building on Cloud Build does not change anything. Dig for google.com |
|
I cloned repo into cloudshell, slightly modified regions file to let me run this, built docker and dig and curl from example was working. I triend it also with runtime runsc to try gvisor but it's also working. Pushed this example into container registry, created revision on cloud run and nope, not working. |
|
Hi, do you have any update regarding dns fix on google side? edit: |
|
Sorry, I've been debating internally defending this is a bug. So far I have not convinced people to fix this and it required me to go low level to prove that this is how DNS works and our bug makes it incompatible. My only other option is to never proxy the queries to "google.internal." Zone in runsd resolve metadata.google.internal lookup myself. I am not sure what other side effects would be and it wouldn't be a long-term solution. Maybe we can try. |
ahmetb
changed the title
|
At this moment it works on my 2 projects. One with hello example and one on my dev environment so maybe something has changed. But until it's not confirmed internally I won't use it. I was considering using serverless connector as you mentioned earlier but it is at least $16 per month.
|
|
The problem wasnt resolving internal domains? It was resolving external domains I think? Example.com shouldn't be working? |
|
Hm, internally it was not working too but maybe it was other reason and I just don't remember. Do you think that https://dns.google might help? Query this and cache when getting SERVFAIL or pattern does not match run.app. Just loose considerations. |
|
We can't blindly replace the nameserver we recurse into. People who use VPC Connector actually use internal DNS names of VMs and it works from Cloud Run. Thats why we have to proxy to 169.254.169.254. However the "google.Internal." search domain is giving us problems iterating over candidates. That's the problem. |
If we let "{nonexisting}.google.internal." get handled by the Cloud Run host
nameserver, it returns a SERVFAIL, which prevents trying other "search" domains
from being tried.
Adding a temporary workaround that _only_ handles "metadata.google.internal."
for A question (and ignoring other question types) and properly NXDOMAIN-ing
the non-existing domains.
This is to address #18.
Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
|
I've just pushed a tag |
|
It seems like the fix solved it. |
I just tried to use runsd with GCP tracking and got this error
I assume the reason is that it's using
httpswhile for runsd it should behttp. Is there a way to makie it working without changing urls in client object?The text was updated successfully, but these errors were encountered: