Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Makes controller run in-cluster #1

Merged
merged 3 commits into from
Jun 10, 2020
Merged

Makes controller run in-cluster #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2eddc47
Makes controller run in-cluster
somtochiama Jun 2, 2020
3be60ae
Documentation for running operator in kinder cluster
somtochiama Jun 2, 2020
b965cf3
Seperates manager.yaml into patches
somtochiama Jun 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion kubeproxy/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
FROM ubuntu:latest as kubectl
RUN apt-get update
RUN apt-get install curl -y
RUN curl -fsSL https://dl.k8s.io/release/v1.17.4/bin/linux/amd64/kubectl > /usr/bin/kubectl
RUN chmod a+rx /usr/bin/kubectl
# Build the manager binary
FROM golang:1.12.5 as builder

Expand All @@ -19,9 +24,11 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
COPY --from=kubectl /usr/bin/kubectl /usr/bin/kubectl
COPY channels/ channels/
USER nonroot:nonroot

ENTRYPOINT ["/manager"]
53 changes: 53 additions & 0 deletions kubeproxy/InCluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
## This Readme documents how to run the KubeProxy operator in a kinder cluster

# 1. Create a kinder cluster
Ensure kinder is installed. [Installation docs](https://github.com/kubernetes/kubeadm/blob/master/kinder/README.md)

```bash
kinder create cluster --image=kindest/node:v1.18.0

kinder do kubeadm-config
kinder do loadbalancer

docker exec -it kind-control-plane-1 /kind/bin/kubeadm init --skip-phases="addon/kube-proxy" --ignore-preflight-errors="FileContent--proc-sys-net-bridge-bridge-nf-call-iptables,Swap,SystemVerification" --config /kind/kubeadm.conf
kinder exec @all -- sysctl -w net.ipv4.conf.all.rp_filter=1

kinder cp @cp1:/etc/kubernetes/admin.conf $(kinder get kubeconfig-path)
export KUBECONFIG=$(kinder get kubeconfig-path)
```

You might have set the server ip in the KUBECONFIG to use localhost to reach the cluster, `insecure-skip-tls-verify` to true, and delete the ca certificate. To find the port, run `docker ps | grep kind` and check the port

> insecure-skip-tls-verify: true
> server: https://127.0.0.1:<port>

2. Set the Kubernetes Service host and port in manager.yaml
ssh into the node and get the host and port.
The command below should give the host.
```bash
docker inspect kind-control-plane-1 | grep IPAddress
```

Replace it in the `manager.yaml`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run make manifests will this get overwritten? If so, it may make more sense to have the user edit the patch (patches/apiserver_endpoint.patch.yaml)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. That should change


>- name: KUBERNETES_SERVICE_HOST
> value: "172.17.0.2"
>- name: KUBERNETES_SERVICE_PORT
> value: "6443"


3. Build and deploy Docker image
```bash
make docker-build

make deploy
```

4. Install CRD

```bash
make install
kubectl apply -f config/samples/
```

5. KubeProxy should be up and running
2 changes: 1 addition & 1 deletion kubeproxy/api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 2 additions & 3 deletions kubeproxy/channels/packages/kubeproxy/1.15.0/manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ spec:
- description: Project Homepage
url: "https://github.com/kubernetes/kubernetes"
---
apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-proxy
Expand Down Expand Up @@ -60,7 +60,7 @@ spec:
command:
- /bin/sh
- -c
- kube-proxy --resource-container="" --oom-score-adj=-998 {{params}}
- kube-proxy --resource-container="" --oom-score-adj=-998 {{params}}
#1>>/var/log/kube-proxy.log 2>&1
securityContext:
privileged: true
Expand Down Expand Up @@ -104,4 +104,3 @@ roleRef:
kind: ClusterRole
name: system:node-proxier
apiGroup: rbac.authorization.k8s.io

8 changes: 5 additions & 3 deletions kubeproxy/config/crd/bases/addons.x-k8s.io_kubeproxies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.2.4
creationTimestamp: null
name: kubeproxies.addons.x-k8s.io
spec:
Expand All @@ -12,20 +14,20 @@ spec:
listKind: KubeProxyList
plural: kubeproxies
singular: kubeproxy
scope: ""
scope: Namespaced
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a notable change -- I think it does make sense.
The Namespace of the CustomResource can be used to patch the manifests.
Most clusters will only have 1 instance of this CR, but containing it within a namespace like kube-system could make RBAC and ownership more clear.
You could manage kube-proxy daemonsets for different Node Groups in separate Namespaces.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is inline with the existing operators (eg coredns). I think I generated this CRD incorrectly by not picking Namespaced from the onset.

Really great point about different node groups.

validation:
openAPIV3Schema:
description: KubeProxy is the Schema for the API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
Expand Down
9 changes: 9 additions & 0 deletions kubeproxy/config/manager/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,11 @@
resources:
- manager.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: controller
newName: controller
newTag: latest
patchesStrategicMerge:
- patches/apiserver_endpoint.patch.yaml
- patches/tolerations_controlplane.patch.yaml
2 changes: 2 additions & 0 deletions kubeproxy/config/manager/manager.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,14 @@ spec:
labels:
control-plane: controller-manager
spec:
hostNetwork: true
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a reasonable default 👍
users can override for esoteric deploys

containers:
- command:
- /manager
args:
- --enable-leader-election
image: controller:latest
imagePullPolicy: IfNotPresent
name: manager
resources:
limits:
Expand Down
15 changes: 15 additions & 0 deletions kubeproxy/config/manager/patches/apiserver_endpoint.patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller-manager
namespace: system
spec:
template:
spec:
containers:
- name: manager
env:
- name: KUBERNETES_SERVICE_HOST
value: "172.17.0.2"
- name: KUBERNETES_SERVICE_PORT
value: "6443"
15 changes: 15 additions & 0 deletions kubeproxy/config/manager/patches/tolerations_controlplane.patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller-manager
namespace: system
spec:
template:
spec:
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
Empty file added kubeproxy/config/manager/role.yaml
View file
Open in desktop
Empty file.
24 changes: 16 additions & 8 deletions kubeproxy/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,22 @@ module addon-operators/kubeproxy
go 1.12

require (
github.com/appscode/jsonpatch v0.0.0-20190108182946-7c0e3b262f30 // indirect
github.com/go-logr/logr v0.1.0
github.com/onsi/ginkgo v1.7.0
github.com/onsi/gomega v1.4.3
k8s.io/apimachinery v0.0.0-20190404173353-6a84e37a896d
k8s.io/client-go v11.0.1-0.20190409021438-1a26190bd76a+incompatible
k8s.io/klog v0.3.0
github.com/gobuffalo/envy v1.6.10 // indirect
github.com/imdario/mergo v0.3.7 // indirect
github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a // indirect
github.com/onsi/ginkgo v1.10.1
github.com/onsi/gomega v1.7.0
github.com/petar/GoLLRB v0.0.0-20130427215148-53be0d36a84c // indirect
k8s.io/apimachinery v0.17.0
k8s.io/client-go v0.17.0
k8s.io/klog v1.0.0
k8s.io/kubeadm v0.0.0-20191014153037-d541f020334c // indirect
k8s.io/kubeadm/kinder v0.0.0-20191014153037-d541f020334c // indirect
sigs.k8s.io/controller-runtime v0.2.2
sigs.k8s.io/kubebuilder-declarative-pattern v0.0.0-20190926123507-e845b6c6f25a
sigs.k8s.io/cluster-addons/dashboard v0.0.0-20200515184536-657eb5be7e85
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why the dashboard operator and kind are getting pulled into this file? Maybe a weird local cache issue?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's weird. I would look into it

sigs.k8s.io/controller-runtime v0.4.0
sigs.k8s.io/controller-tools v0.1.6 // indirect
sigs.k8s.io/kind v0.1.0 // indirect
sigs.k8s.io/kubebuilder-declarative-pattern v0.0.0-20200226054827-748a6481b2a4
sigs.k8s.io/kustomize v2.0.3+incompatible // indirect
)
Loading