Add kind-loadbalancer

[SchahinRohani]

Description

Provide an envoy loadbalancer container to access the local kind cluster. The loadbalancer adds the ability to reach the local kind cluster trough localhost, so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080 -> hubble-ui
localhost:8081 -> tekton-dashboard
localhost:8082/eventlistener -> nativelink-eventlistener

Type of change

Please delete options that aren't relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to
  not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please also list any relevant details for your test configuration

aarch64-darwin

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[CLAassistant]

All committers have signed the CLA.

[SchahinRohani]

@aaronmondal Can you check this on linux?

[aaronmondal]

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, and 1 discussions need to be resolved

---

native-cli/components/loadbalancer.go line 240 at r1 (raw file):

				&docker.ContainerMountArgs{
					Source: pulumi.String(absolutePath),
					Target: pulumi.String("/etc/envoy/envoy.yaml"),

I'm getting this:

[2024-05-24 02:39:47.817][1][critical][main] [source/server/server.cc:112] error initializing configuration '/etc/envoy/envoy.yaml': unable to read file: /etc/envoy/envoy.yaml
[2024-05-24 02:39:47.817][1][info][main] [source/server/server.cc:855] exiting unable to read file: /etc/envoy/envoy.yaml

It looks like the envoy.yaml doesn't make it into the kind-loadbalancer container.

[SchahinRohani]

native-cli/components/loadbalancer.go line 240 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

I'm getting this:

[2024-05-24 02:39:47.817][1][critical][main] [source/server/server.cc:112] error initializing configuration '/etc/envoy/envoy.yaml': unable to read file: /etc/envoy/envoy.yaml
[2024-05-24 02:39:47.817][1][info][main] [source/server/server.cc:855] exiting unable to read file: /etc/envoy/envoy.yaml

It looks like the envoy.yaml doesn't make it into the kind-loadbalancer container.

Fixed. The file permissions of the created envoy config werent allowing to read the file in the container.

[aaronmondal]

Reviewed 10 of 14 files at r1, 9 of 9 files at r2, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and 22 discussions need to be resolved

---

native-cli/clusters/config.yaml line 5 at r2 (raw file):

apiVersion: kind.x-k8s.io/v1alpha4
networking:
  # the default CNI will not be installed

nit

Suggestion:

# Don't install the default CNI.

---

native-cli/clusters/localcluster.go line 87 at r2 (raw file):

	Nodes                   []Node     `yaml:"nodes"`
	ContainerdConfigPatches []string   `yaml:"containerdConfigPatches"`
}

Can we make this work in the same way that the envoy templating works? I.e. have the kind config.yaml a kind-config.template.yaml and use a simila approach as for envoy.template.yaml.

---

native-cli/clusters/localcluster.go line 101 at r2 (raw file):

	var config ClusterConfig

	config.get()

Does config := ClusterConfig.get() work?

Code quote:

	var config ClusterConfig

	config.get()

---

native-cli/clusters/localcluster.go line 103 at r2 (raw file):

	config.get()

	gitRoot := gitSrcRoot()

You can inline this.

---

native-cli/clusters/localcluster.go line 114 at r2 (raw file):

				},
			)
			if runtime.GOOS == "linux" {

Let's add a "TODO(username): Implement nix caching for MacOS." to make it clear that this doesn't work yet.

---

native-cli/components/loadbalancer.go line 89 at r2 (raw file):

		waitGroup.Add(1)
		// Assign the function to a variable and then invoke it in the go statement
		checkAndAddGateway := func(name string) {

nit: You can remove this variable and instead do:

go func(name string) {
...
}(name)

Code quote:

checkAndAddGateway :=

---

native-cli/components/loadbalancer.go line 121 at r2 (raw file):

}

func addGatewayIfNeeded(

Missing docstring.

---

native-cli/components/loadbalancer.go line 166 at r2 (raw file):

	return true
}

Since this is only used once, inline the function.

Code quote:

func allGatewaysFound(requiredGateways map[string]bool) bool {
	for _, found := range requiredGateways {
		if !found {
			return false
		}
	}

	return true
}

---

native-cli/components/loadbalancer.go line 168 at r2 (raw file):

}

func getHomeDir() (string, error) {

This function is unnecessary because the way to call it would look like this:

home, err := getHomeDir()
if err != nil {
  return "", fmt.Errorf("xxxxxx"),
}

That's equivalent to how this function looks like, so we might as well just write this code out directly.

---

native-cli/components/loadbalancer.go line 177 at r2 (raw file):

}

func buildKubeConfig(kubeconfig *string) (*rest.Config, error) {

Same as above.

---

native-cli/components/loadbalancer.go line 186 at r2 (raw file):

}

func createGatewayClientset(

Same as above.

---

native-cli/components/loadbalancer.go line 313 at r2 (raw file):

	}

	requiredGateways := map[string]bool{

You can inline this variable.

---

native-cli/components/loadbalancer.go line 315 at r2 (raw file):

	requiredGateways := map[string]bool{
		"scheduler-gateway": false,
		"cache-gateway":     false,

At the moment the loadbalancer doesn't yet require (or expose) the scheduler and cache gateways. Let's comment these two out and put a TODO that we want them enabled.

Code quote:

		"scheduler-gateway": false,
		"cache-gateway":     false,

---

native-cli/components/loadbalancer.go line 337 at r2 (raw file):

	}

	ports := prepareDockerContainerPorts(component.Ports)

Inline this variable and call the function dockerContainerPorts.

---

native-cli/components/nativelink-routes.go line 22 at r2 (raw file):

var nativeLinkRoutesYaml string

// Install sets up the Gateways for the NativeLink deployment.

The comments in this file need to be updated.

---

native-cli/components/embedded/envoy_template.yaml line 16 at r2 (raw file):

                stat_prefix: ingress_http
                upgrade_configs:
                  - upgrade_type: websocket

I don't think we're making use of websockets at the moment.

---

native-cli/components/embedded/envoy_template.yaml line 26 at r2 (raw file):

                        - match: { prefix: "/eventlistener" }
                          route: { cluster: el-gateway }
                        # TODO(SchahinRohani) working grpc proxy

nit

Suggestion:

Make gRPC proxying work.

---

native-cli/components/embedded/envoy_template.yaml line 31 at r2 (raw file):

                        # - match: { prefix: "/scheduler" }
                        #   route: { cluster: scheduler-gateway }
                        {{- else if eq $port.InternalPort 80 }}

Can we move these ports to go so that we have them all in the same place?

I'm thinking something like ... if eq $port.InternalPort HubblePort and then set HubblePort in Go. Same for the others.

---

native-cli/components/embedded/envoy_template.yaml line 41 at r2 (raw file):

                  # - name: envoy.filters.http.grpc_http1_bridge
                  #   typed_config:
                  #     "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_http1_bridge.v3.Config

Let's remove this since we already have the todo above.

---

native-cli/components/embedded/nativelink-gateways.yaml line 16 at r2 (raw file):

      allowedRoutes:
        namespaces:
          from: All

This allows routes from all namespaces to attach to the gateways. Can we make this an exact matching? There might be some issues with cilium, but ideally we'd only allow namespaces whose routes we actually expect.

---

native-cli/components/embedded/nativelink-routes.yaml line 2 at r2 (raw file):

# TODO(aaronmondal): There should just be a single gateway. But that's currently
#                    bugged: https://github.com/cilium/cilium/issues/29099

Misplaced coment

---

native-cli/components/embedded/nativelink-routes.yaml line 3 at r2 (raw file):

# TODO(aaronmondal): There should just be a single gateway. But that's currently
#                    bugged: https://github.com/cilium/cilium/issues/29099
# ---

The --- mark the beginning of the yaml file and shouldn't be commented out.

Suggestion:

---

[SchahinRohani]

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 19 discussions need to be resolved

---

native-cli/clusters/localcluster.go line 87 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Can we make this work in the same way that the envoy templating works? I.e. have the kind config.yaml a kind-config.template.yaml and use a simila approach as for envoy.template.yaml.

Done.

---

native-cli/clusters/localcluster.go line 101 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Does config := ClusterConfig.get() work?

Done.

---

native-cli/clusters/localcluster.go line 103 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

You can inline this.

Done.

---

native-cli/clusters/localcluster.go line 114 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Let's add a "TODO(username): Implement nix caching for MacOS." to make it clear that this doesn't work yet.

Done.

---

native-cli/components/loadbalancer.go line 121 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Missing docstring.

Done.

---

native-cli/components/loadbalancer.go line 166 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Since this is only used once, inline the function.

Done.

---

native-cli/components/loadbalancer.go line 168 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

This function is unnecessary because the way to call it would look like this:

home, err := getHomeDir()
if err != nil {
  return "", fmt.Errorf("xxxxxx"),
}

That's equivalent to how this function looks like, so we might as well just write this code out directly.

Done.

---

native-cli/components/loadbalancer.go line 177 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Same as above.

Done.

---

native-cli/components/loadbalancer.go line 186 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Same as above.

Done.

---

native-cli/components/loadbalancer.go line 313 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

You can inline this variable.

Done.

---

native-cli/components/loadbalancer.go line 315 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

At the moment the loadbalancer doesn't yet require (or expose) the scheduler and cache gateways. Let's comment these two out and put a TODO that we want them enabled.

Lets keep them to ensure that we have all gateway ips in place.

---

native-cli/components/loadbalancer.go line 337 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Inline this variable and call the function dockerContainerPorts.

Done.

---

native-cli/components/nativelink-routes.go line 22 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

The comments in this file need to be updated.

Done.

---

native-cli/components/embedded/nativelink-gateways.yaml line 16 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

This allows routes from all namespaces to attach to the gateways. Can we make this an exact matching? There might be some issues with cilium, but ideally we'd only allow namespaces whose routes we actually expect.

Done.

---

native-cli/components/embedded/nativelink-routes.yaml line 2 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Misplaced coment

Done.

---

native-cli/components/embedded/nativelink-routes.yaml line 3 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

The --- mark the beginning of the yaml file and shouldn't be commented out.

Done.

---

native-cli/components/embedded/envoy_template.yaml line 16 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

I don't think we're making use of websockets at the moment.

Done.

---

native-cli/components/embedded/envoy_template.yaml line 31 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Can we move these ports to go so that we have them all in the same place?

I'm thinking something like ... if eq $port.InternalPort HubblePort and then set HubblePort in Go. Same for the others.

Done.

---

native-cli/components/embedded/envoy_template.yaml line 41 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Let's remove this since we already have the todo above.

Done.

[aaronmondal]

One minor nit to merge Route into RouteConfig, otherwise looks good! Great work!

Reviewed 9 of 9 files at r3, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and 4 discussions need to be resolved

---

native-cli/components/loadbalancer.go line 39 at r3 (raw file):

	Cluster       string
	PrefixRewrite string
}

nit: It looks like we can move the Cluster and PrefixRewrite fields directly into the RouteConfig.

Code quote:

type Route struct {
	Cluster       string
	PrefixRewrite string
}

---

native-cli/components/embedded/nativelink-gateways.yaml line 50 at r3 (raw file):

      allowedRoutes:
        namespaces:
          from: Same

nit: This can be removed since Same is the default.

---

native-cli/components/embedded/nativelink-gateways.yaml line 65 at r3 (raw file):

      allowedRoutes:
        namespaces:
          from: Same

nit: ditto

---

native-cli/programs/local.go line 133 at r3 (raw file):

					Cluster: "hubble-gateway",
				},
			},

When the Route is changed as above these become:

Suggestion:

			{
				Prefix: "/",
                Cluster: "hubble-gateway",
			},

[SchahinRohani]

Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable, and 1 discussions need to be resolved

---

native-cli/programs/local.go line 133 at r3 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

When the Route is changed as above these become:

Done.

[aaronmondal]

Reviewed 4 of 4 files at r4, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Cargo Dev / macos-13