Sync does not work in containers without the tar command

[corneliusweig]

Expected behavior

Changed files are synced to the container. The example below does not make much sense, but it does also affect distroless containers.

Actual behavior

Sync fails with

DEBU[0007] Running command: [kubectl exec getting-started --namespace default -c getting-started -i -- tar xmf - -C / --no-same-owner] 
WARN[0007] Skipping deploy due to sync error: copying files: exit status 126

Information

• Skaffold version: >0.24 (since Optimise sync #1641)

Steps to reproduce the behavior

Based on the examples/getting-started example

1. Modify Dockerfile to

FROM golang:1.10.1-alpine3.7 as builder
COPY main.go .
RUN go build -o /app main.go

FROM scratch
ENTRYPOINT ["./app"]
COPY . /unused
COPY --from=builder /app .

2. Modify skaffold.yaml

apiVersion: skaffold/v1beta7
kind: Config
build:
  artifacts:
  - image: gcr.io/k8s-skaffold/skaffold-example
    sync:
    - '*.go': /
deploy:
  kubectl:
    manifests:
      - k8s-*

3. skaffold dev -v debug
4. Change main.go and wait for sync.

[corneliusweig]

A way out is obviously to offer a fallback from the tar-optimized copy to a single-file copy (see next comment). I would like to fix this, and I can think of two options:

1. try an opportunistic tar-sync. If that fails, change the syncer from tar-based to single-file for the rest of this skaffold session. This has the advantage of not adding yet another configuration option which users don't really want to bother about. On the other hand, if the sync fails for a different reason, it will be stuck on the slow syncer.
2. add a new configuration to switch between tar-based and single-file syncer. In that case, what is a good name? Has the drawback that users need to be aware of this option and makes the configuration even bigger.

What do you think?

[corneliusweig]

So I just tried the fallback version but it will not work, because kubectl cp also builds a tar 🙄
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/cp/cp.go#L245

I also had a look at the distroless images from https://github.com/GoogleContainerTools/distroless/ and it looks like they do not package any executable like cat or tar (that's what they are made for, after all). So that rules out kubectl cp for distroless and scratch.

A way out could be to package a statically linked tar binary with skaffold and copy this into the container. However, handling this edge case may not be worth it. And maybe this is something which should be handled by the kubectl folks. What do you think? (<- chicken-egg problem)

EDIT: kubernetes/kubernetes#58512

[corneliusweig]

Looks like this is completely blocked until kubectl implements a fallback that works without tar in the container.

[tstromberg]

Leaving open, because this is still apparently an issue blocked on kubectl cp improvements.

[aaron-prindle]

kubectl cp still requires tar (11/28/2022)

[aaron-prindle]

Related k8s issue (mentioned above in the thread) - kubernetes/kubernetes#58512