Feature request for Bazel support: Support containers built with rules_oci

[aran]

rules_oci is a new container builder for Bazel. Compared to rules_docker, it is under active development and recently released 1.0.

Instead of one container_image rule that also produces a .tar target for docker, rules_oci separates two rules: oci_image and oci_tarball.

Despite the fact that docker (or in my case, colima) can load and run the output of rules_oci, when using skaffold, something breaks with the following error: build [hello-image] failed: tagging the image: Error parsing reference: "" is not a valid repository/tag: invalid reference format See longer version below, and attached small workspace. rules_oci_demo.tar.gz

aran@MacBook-Air rules_oci % skaffold dev               
Generating tags...
 - hello-image -> hello-image:latest
Checking cache...
 - hello-image: Not found. Building
Starting build...
Found [colima] context, using local docker daemon.
Building [hello-image]...
Target platforms: [linux/arm64]
Loading: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //:hello.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //:hello.tar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //:hello.tar up-to-date:
  bazel-bin/hello.tar/tarball.tar
INFO: Elapsed time: 0.074s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Loaded image: hello:latest
Cleaning up...
 - No resources found
build [hello-image] failed: tagging the image: Error parsing reference: "" is not a valid repository/tag: invalid reference format
aran@MacBook-Air rules_oci % docker load --input $(bazel cquery --output=files //:hello.tar)     
Loading: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //:hello.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //:hello.tar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: Elapsed time: 0.070s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
Loaded image: hello:latest
aran@MacBook-Air rules_oci % docker run hello:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2023/06/15 19:21:46 Server listening on port 8080
^C%    

[ar3s3ru]

Just to add some context to the issue, I'm having a similar problem on my setup, though it might not be directly correlated to rules_oci.

Here is a condensed version of my skaffold.yaml:

apiVersion: skaffold/v4beta5
kind: Config

build:
  platforms: ["linux/amd64"]
  artifacts:
  - image: backend
    bazel:
      target: //apps/backend:image.tar


profiles:
- name: local
  manifests:
    rawYaml:
    - apps/**/deploy.local.yaml
    - tools/k3d/manifests/*.yaml
  build:
    local:
      push: true
  deploy:
    kubectl: {}

- name: prod
  manifests:
    rawYaml:
    - apps/**/deploy.prod.yaml
  deploy:
    cloudrun:
      projectid: <PROJECT>
      region: <REGION>

To specify the right Docker registry, I have wrapped Skaffold with a Makefile, and passing --default-repo based on the environment:

DOCKER_REPO_local := k3d-registry.local:9000
DOCKER_REPO_prod  := <REGION>-docker.pkg.dev/<PROJECT>/docker
SKAFFOLD_ARGS     :=

ifdef DOCKER_REPO_$(env)
SKAFFOLD_ARGS += --default-repo $(DOCKER_REPO_$(env))
endif

skaffold.run: 
	skaffold run -p $(env) $(SKAFFOLD_ARGS)

Hopefully I can remove this if/when #8920 is closed.

What I have observed:

1. When my local Kubernetes cluster (k3d) is off, I'm able to build and deploy my Cloud Run images/manifests just fine,
2. When k3d is on, I'm not able to move past image tagging, same error message as OP.

I suspect that the reason for 2. is that Skaffold inspects the Kubernetes context of the local configuration, and extracts some info from it (e.g. default-repo):

skaffold/pkg/skaffold/runner/runcontext/context.go

Lines 430 to 438 in 637fe0d

	cluster, err := config.GetCluster(ctx, config.GetClusterOpts{
	ConfigFile: opts.GlobalConfig,
	DefaultRepo: opts.DefaultRepo,
	MinikubeProfile: opts.MinikubeProfile,
	DetectMinikube: opts.DetectMinikube,
	})
	if err != nil {
	return nil, fmt.Errorf("getting cluster: %w", err)
	}

)

skaffold/pkg/skaffold/config/util.go

Lines 367 to 397 in 88e1c17

	func discoverLocalRegistry(ctx context.Context, kubeContext string) (*string, error) {
	clientset, err := kubeclient.Client(kubeContext)
	if err != nil {
	return nil, err
	}
	configMap, err := clientset.CoreV1().ConfigMaps("kube-public").Get(ctx, "local-registry-hosting", api_v1.GetOptions{})
	statusErr := &api_errors.StatusError{}
	switch {
	case errors.As(err, &statusErr) && statusErr.Status().Code == http.StatusNotFound:
	return nil, nil
	case err != nil:
	return nil, err
	}
	data, ok := configMap.Data["localRegistryHosting.v1"]
	if !ok {
	return nil, errors.New("invalid local-registry-hosting ConfigMap")
	}
	dst := struct {
	Host *string `yaml:"host"`
	}{}
	if err := yaml.Unmarshal([]byte(data), &dst); err != nil {
	return nil, errors.New("invalid local-registry-hosting ConfigMap")
	}
	return dst.Host, nil
	}

Which then gets used during the run command:

skaffold/cmd/skaffold/app/cmd/run.go

Lines 45 to 49 in 88e1c17

	return withRunner(ctx, out, func(r runner.Runner, configs []util.VersionedConfig) error {
	bRes, err := r.Build(ctx, out, targetArtifacts(opts, configs))
	if err != nil {
	return fmt.Errorf("failed to build: %w", err)
	}

[ericzzzzzzz]

• If you use a remote registry, oci_rule should just work
• If you store your image in local docker daemon, there is a workaround, but quite hacky
  • in oci_tarball call, repo_tags should contains a tag in the format bazel:${target-name-without-.tar} , example

          oci_tarball(
    name = "ccc.tar",
    image = "@hello//:hello",
    repo_tags = ["bazel:ccc", "hello:latest"],
)

other tags are optional.

we're currently a bit understaffed, the official support for oci rule may not be prioritized.

[aran]

Is there a reason it would be non-deterministic, or dependent on some aspect of docker state? I thought had it working with this workaround, then it stopped working. Using relative targets ":foo" instead of "//abs/path/to:foo" seems to get it to work again. I am wondering if that's because it avoids

skaffold/pkg/skaffold/build/bazel/build.go

Line 163 in d019fdd

if strings.Contains(imageTag, ":") {

. That makes me wonder if there might be a small and easy fix.

[a9b3]

Confirming that the workaround of putting the target on the same level as skaffold.yaml and using the relative target results in a successful build. (Note: it does not work if the target is //some/path:aaa.tar must be //:aaa.tar)

Example below

# ./skaffold.yaml
kind: Config
build:
  artifacts:
    - image: aaa
      bazel:
        target: //:aaa.tar

# ./BUILD.bazel
oci_tarball(
    name = "aaa.tar",
    image = "//path/to/whatever:image",
    repo_tags = [
        "bazel:aaa",
    ],
)

[ericzzzzzzz]

Is there a reason it would be non-deterministic, or dependent on some aspect of docker state? I thought had it working with this workaround, then it stopped working. Using relative targets ":foo" instead of "//abs/path/to:foo" seems to get it to work again. I am wondering if that's because it avoids

Not sure why this part of code was implemented in this way, I don't know much about bazel, but it looks like that with the original docker build rule, there is no dedicated place to specify image tag when bazel builds a image tarball, it deduces image name/tag from the target name and wrap it in the tarball, when the tarball is loaded to a docker daemon the tag is used, skaffold is kind doing something similar to construct the tag, and then get the imageID with the tag from docker daemon.

For oci_tarball rule, you can specify your tag in repo_tags. That's why we need to add an extra tag that skaffold will deduce from target name, for something like //:abc.tar //: and .tarare removed, and bazel: is prepend. For name like //sss/bbb/ccc:abc.tar , skaffold doesn't remove ":", instead it build a string like bazel/sss/bbb/ccc:abc as the tag.
So for case like this, the extra tag like "bazel/{package_name}:{targetname_without_.tar}" should be added to repo_tags to make this hack work.

Also, this feature request has been prioritized, should be available after one or two releases.