Ability to run Earthly on an M1 computer that uses a remote x86 Docker

[vladaionescu]

Setup

• User has an M1 laptop
• User uses a remote Docker daemon, which is running on an x86

Problem

User wants to run Earthly locally, but as Earthly attempts to start the buildkit daemon in Docker, it pulls the arm64 version of the image, instead of the x86 version of it. This causes a crash as buildkit cannot be emulated.

What should happen instead

• Earthly should detect that the Docker daemon is running on x86 (which may be different than the current host's architecture) and thus use an x86 version of the Earthly image.
• Earthly builds run on this kind of setup should default to use platform x86, when no platform is specified (it's possible that the new platform logic behind a feature flag already does this)

[vladaionescu]

Possible workaround

• Start an x86 Buildkit daemon via Docker using the instructions here: https://docs.earthly.dev/ci-integration/remote-buildkit
• Point Earthly to that daemon by setting EARTHLY_BUILDKIT_HOST env var

[dchw]

We actually already expose this information, too! Its in this structure:

earthly/util/containerutil/types.go

Lines 53 to 62 in b0e5a0f

	type FrontendInfo struct {
	ClientVersion string
	ClientAPIVersion string
	ClientPlatform string
	ServerVersion string
	ServerAPIVersion string
	ServerPlatform string
	ServerAddress string
	}

And you get it (regardless of frontend) with this method on a given frontend:

earthly/util/containerutil/frontend.go

Line 21 in b0e5a0f

Information(ctx context.Context) (*FrontendInfo, error)

Implementations:

• Docker:

  earthly/util/containerutil/docker.go

  Lines 77 to 112 in b0e5a0f

  	func (dsf *dockerShellFrontend) Information(ctx context.Context) (*FrontendInfo, error) {
  	output, err := dsf.commandContextOutput(ctx, "version", "--format={{json .}}")
  	if err != nil {
  	return nil, err
  	}
  	type versionInfo struct {
  	Version string
  	APIVersion string
  	OS string
  	Arch string
  	}
  	type info struct {
  	Client versionInfo
  	Server versionInfo
  	}
  	allInfo := info{}
  	json.Unmarshal([]byte(output.string()), &allInfo)
  	host, exists := os.LookupEnv("DOCKER_HOST")
  	if !exists {
  	host = "/var/run/docker.sock"
  	}
  	return &FrontendInfo{
  	ClientVersion: allInfo.Client.Version,
  	ClientAPIVersion: allInfo.Client.APIVersion,
  	ClientPlatform: fmt.Sprintf("%s/%s", allInfo.Client.OS, allInfo.Client.Arch),
  	ServerVersion: allInfo.Server.Version,
  	ServerAPIVersion: allInfo.Server.APIVersion,
  	ServerPlatform: fmt.Sprintf("%s/%s", allInfo.Server.OS, allInfo.Server.Arch),
  	ServerAddress: host,
  	}, nil
  	}

• Podman:

  earthly/util/containerutil/podman.go

  Lines 78 to 131 in b0e5a0f

  	func (psf *podmanShellFrontend) Information(ctx context.Context) (*FrontendInfo, error) {
  	output, err := psf.commandContextOutput(ctx, "info", "--format={{.Host.RemoteSocket.Exists}}")
  	if err != nil {
  	return nil, err
  	}
  	hasRemote, err := strconv.ParseBool(output.string())
  	if err != nil {
  	return nil, errors.Wrapf(err, "info returned invalid value %s", output.string())
  	}
  	args := []string{"version", "--format=json"}
  	if hasRemote {
  	args = append([]string{"-r"}, args...)
  	}
  	output, err = psf.commandContextOutput(ctx, args...)
  	if err != nil {
  	return nil, err
  	}
  	type versionInfo struct {
  	Version string
  	APIVersion string
  	OSArch string
  	}
  	type info struct {
  	Client versionInfo
  	Server versionInfo
  	}
  	allInfo := info{}
  	json.Unmarshal([]byte(output.string()), &allInfo)
  	host := "daemonless"
  	if hasRemote {
  	output, err = psf.commandContextOutput(ctx, "info", "--format={{.Host.RemoteSocket.Path}}")
  	if err != nil {
  	return nil, err
  	}
  	host = string(output.string())
  	}
  	return &FrontendInfo{
  	ClientVersion: allInfo.Client.Version,
  	ClientAPIVersion: allInfo.Client.APIVersion,
  	ClientPlatform: allInfo.Client.OSArch,
  	ServerVersion: allInfo.Server.Version,
  	ServerAPIVersion: allInfo.Server.APIVersion,
  	ServerPlatform: allInfo.Server.OSArch,
  	ServerAddress: host,
  	}, nil
  	}

[vladaionescu]

Oh great - that's really useful! I'll play with this this weekend. EDIT: Didn't have time.

[vladaionescu]

Note that interactive mode might not work immediately in this setup and more tinkering may be necessary. Also, we should test that local registry-based outputs still work fine in this case.

[r-LaForge]

Digging into the problem more I believe @alexcb is correct and we need to use that frontend information.
We are using an incorrect assumption when running the buildkit image.

• We assume the frontend host platform is the same as the current machine's platform.
• We assume the message will contain "Unable to find image" if and only if it supports the frontend.

The below shows the second assumption is not true either.

$ docker run --rm --platform fake/arch scratch
Unable to find image 'scratch:latest' locally
docker: Error response from daemon: 'scratch' is a reserved name.
See 'docker run --help'.