muslc-based images -> alpine based git-init fail to resolve github.com

[cruwe]

First and foremost, thank you for developing, maintaining and publishing the knative framework.

Expected Behavior

At the root issue, alpine-based images to be able to resolve with "higher" ndots and a number of search domains set in /etc/resolv.conf. At the intermediate issue, git-init should be able to resolve domains such as github.com. At the more immediate issue, git-init should be able to just checkout a given repo.

Actual Behavior

Checking out repos fails with (prettied the json up for readability)

[
  {
    "level": "warn",
    "ts": 1552040816.4585717,
    "logger": "fallback-logger",
    "caller": "logging/config.go:65",
    "msg": "Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"
  },
  {
    "level": "fatal",
    "ts": 1552040821.4823895,
    "logger": "fallback-logger",
    "caller": "git-init/main.go:52",
    "msg": "Unexpected error running git [fetch --depth=1 --recurse-submodules=yes origin master]: exit status 128\nfatal: unable to access 'https://github.com/knative/build.git/': Could not resolve host: github.com\n",
    "stacktrace": "main.runOrFail\n\t/go/src/github.com/knative/build/cmd/git-init/main.go:52\nmain.main\n\t/go/src/github.com/knative/build/cmd/git-init/main.go:89\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"
  }
]

Steps to Reproduce the Problem

1. Run a self-provisioned cluster (kubespray calling kubeadm) with ndots 5 and a couple of search domains (default.svc.k8s.cruwe.local svc.k8s.cruwe.local k8s.cruwe.local [...].cruwe.de)
2. Run an arbitrary build with a scm-ceckout.

Additional Info

Being new to both knative and istio, it took me a day to properly bisect the problem to get to the root cause, which is, admittedly, not related to knative, but rather on how muslc does the resolving. The issue seems to be rather old, perhaps even sporting some notoriety, gliderlabs/docker-alpine#8 being the most prominent discussion.

I have verified with garden variety alpine images in my cluster, all fail to properly resolve github.com, they resolve github.com. (with the root-zone dot) using getent hosts, but resolve just fine in both cases using dig from the bind-utils.

I am unsure how to classify this issue. Having been nastily bitten by muslc/alpine peculiarities in 2015, I walk around that particular distro since then. Apparently (I searched knative issues), either no-one else has clusters built as I have (would'nt have thought that my taste is that strange) or everyone else expects this behaviour, which I cannot quite fathom.

I have "resolved" the problem for my own setting just re-basing the git-init container on a debian:jessie-slim. I am unsure if to really ask to modify or revert #179 or at the very least aks for a polite hint at https://github.com/knative/docs/blob/master/install/Knative-custom-install.md or at https://github.com/knative/docs/blob/master/build/creating-builds.md.

I am curious to hear (well, read) your thoughts and thank you again for the indeed very practical knative framework.

Cheers, Christopher

Edit: Found the about same issue referenced with kubernetes upstream as in kubernetes/kubernetes/issues/64924

[sfxworks]

This also occurs on a Kaniko build using the Kaniko Build template