New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support antrea as network policy provider in kube-up #100736
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,262 @@ | ||
kind: DaemonSet | ||
apiVersion: apps/v1 | ||
metadata: | ||
labels: | ||
app: antrea | ||
addonmanager.kubernetes.io/mode: Reconcile | ||
component: antrea-node-init | ||
name: antrea-node-init | ||
namespace: kube-system | ||
spec: | ||
selector: | ||
matchLabels: | ||
app: antrea | ||
component: antrea-node-init | ||
template: | ||
metadata: | ||
labels: | ||
app: antrea | ||
component: antrea-node-init | ||
spec: | ||
nodeSelector: | ||
antrea.io/ds-ready: "true" | ||
kubernetes.io/os: linux | ||
hostPID: true | ||
hostNetwork: true | ||
containers: | ||
- name: node-init | ||
image: gcr.io/google-containers/startup-script:v1 | ||
imagePullPolicy: IfNotPresent | ||
securityContext: | ||
privileged: true | ||
volumeMounts: | ||
- mountPath: /etc/default/ | ||
name: host-etc-default | ||
env: | ||
- name: STARTUP_SCRIPT | ||
value: | | ||
#! /bin/bash | ||
set -o errexit | ||
set -o pipefail | ||
set -o nounset | ||
|
||
echo "Node initialization start" | ||
sed 's/^ip_aliases = .*/ip_aliases = false/g' -i /etc/default/instance_configs.cfg | ||
|
||
# kill restart google-network-daemon then systemd on host will restart it | ||
killall google_network_daemon | ||
|
||
echo "Node initialization complete" | ||
volumes: | ||
- hostPath: | ||
path: /etc/default | ||
name: host-etc-default | ||
--- | ||
apiVersion: apps/v1 | ||
kind: DaemonSet | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. /cc @jkh52 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also the commit references windows but I notice this says linux in the path. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I repurposed this to support linux as well. @jkh52 what is the concern about the number of DaemonSets? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In my experience with DaemonSets, when there are enough nodes (example: hundreds or more) and HA clusters (3 masters) the proxy-server could have runaway memory ballooning (at times like rolling master restarts, when we expect many agent grpc connections). I found the root cause to be: the agent authentication path in proxy-server was getting throttled by client-go and accumulating resources. I found stability improvements by tuning:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are there node startup scripts that this could be placed in? Or is this solving a chicken/egg problem where Antrea network fabric is not known to be selected before the node boots? |
||
metadata: | ||
labels: | ||
app: antrea | ||
addonmanager.kubernetes.io/mode: Reconcile | ||
component: antrea-agent | ||
name: antrea-agent | ||
namespace: kube-system | ||
spec: | ||
selector: | ||
matchLabels: | ||
app: antrea | ||
component: antrea-agent | ||
template: | ||
metadata: | ||
labels: | ||
app: antrea | ||
component: antrea-agent | ||
spec: | ||
containers: | ||
- args: | ||
- --config | ||
- /etc/antrea/antrea-agent.conf | ||
- --logtostderr=false | ||
- --log_dir=/var/log/antrea | ||
- --alsologtostderr | ||
- --log_file_max_size=100 | ||
- --log_file_max_num=4 | ||
- --v=0 | ||
command: | ||
- antrea-agent | ||
env: | ||
- name: POD_NAME | ||
valueFrom: | ||
fieldRef: | ||
fieldPath: metadata.name | ||
- name: POD_NAMESPACE | ||
valueFrom: | ||
fieldRef: | ||
fieldPath: metadata.namespace | ||
- name: NODE_NAME | ||
valueFrom: | ||
fieldRef: | ||
fieldPath: spec.nodeName | ||
image: projects.registry.vmware.com/antrea/antrea-ubuntu:v1.0.0 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should these images be hosted from a canonical Kubernetes repository? |
||
livenessProbe: | ||
exec: | ||
command: | ||
- /bin/sh | ||
- -c | ||
- container_liveness_probe agent | ||
failureThreshold: 5 | ||
initialDelaySeconds: 5 | ||
periodSeconds: 10 | ||
timeoutSeconds: 5 | ||
name: antrea-agent | ||
ports: | ||
- containerPort: 10350 | ||
name: api | ||
protocol: TCP | ||
readinessProbe: | ||
failureThreshold: 5 | ||
httpGet: | ||
host: localhost | ||
path: /readyz | ||
port: api | ||
scheme: HTTPS | ||
initialDelaySeconds: 5 | ||
periodSeconds: 10 | ||
timeoutSeconds: 5 | ||
resources: | ||
requests: | ||
cpu: 200m | ||
securityContext: | ||
privileged: true | ||
volumeMounts: | ||
- mountPath: /etc/antrea/antrea-agent.conf | ||
name: antrea-config | ||
readOnly: true | ||
subPath: antrea-agent.conf | ||
- mountPath: /var/run/antrea | ||
name: host-var-run-antrea | ||
- mountPath: /var/run/openvswitch | ||
name: host-var-run-antrea | ||
subPath: openvswitch | ||
- mountPath: /var/lib/cni | ||
name: host-var-run-antrea | ||
subPath: cni | ||
- mountPath: /var/log/antrea | ||
name: host-var-log-antrea | ||
- mountPath: /host/proc | ||
name: host-proc | ||
readOnly: true | ||
- mountPath: /host/var/run/netns | ||
mountPropagation: HostToContainer | ||
name: host-var-run-netns | ||
readOnly: true | ||
- mountPath: /run/xtables.lock | ||
name: xtables-lock | ||
- args: | ||
- --log_file_max_size=100 | ||
- --log_file_max_num=4 | ||
command: | ||
- start_ovs | ||
image: projects.registry.vmware.com/antrea/antrea-ubuntu:v1.0.0 | ||
livenessProbe: | ||
exec: | ||
command: | ||
- /bin/sh | ||
- -c | ||
- timeout 10 container_liveness_probe ovs | ||
failureThreshold: 5 | ||
initialDelaySeconds: 5 | ||
periodSeconds: 10 | ||
timeoutSeconds: 10 | ||
name: antrea-ovs | ||
resources: | ||
requests: | ||
cpu: 200m | ||
securityContext: | ||
capabilities: | ||
add: | ||
- SYS_NICE | ||
- NET_ADMIN | ||
- SYS_ADMIN | ||
- IPC_LOCK | ||
volumeMounts: | ||
- mountPath: /var/run/openvswitch | ||
name: host-var-run-antrea | ||
subPath: openvswitch | ||
- mountPath: /var/log/openvswitch | ||
name: host-var-log-antrea | ||
subPath: openvswitch | ||
dnsPolicy: ClusterFirstWithHostNet | ||
hostNetwork: true | ||
initContainers: | ||
- command: | ||
- install_cni | ||
image: projects.registry.vmware.com/antrea/antrea-ubuntu:v1.0.0 | ||
name: install-cni | ||
resources: | ||
requests: | ||
cpu: 100m | ||
securityContext: | ||
capabilities: | ||
add: | ||
- SYS_MODULE | ||
volumeMounts: | ||
- mountPath: /etc/antrea/antrea-cni.conflist | ||
name: antrea-config | ||
readOnly: true | ||
subPath: antrea-cni.conflist | ||
- mountPath: /host/etc/cni/net.d | ||
name: host-cni-conf | ||
- mountPath: /host/opt/cni/bin | ||
name: host-cni-bin | ||
- mountPath: /lib/modules | ||
name: host-lib-modules | ||
readOnly: true | ||
- mountPath: /var/run/antrea | ||
name: host-var-run-antrea | ||
nodeSelector: | ||
antrea.io/ds-ready: "true" | ||
kubernetes.io/os: linux | ||
priorityClassName: system-node-critical | ||
serviceAccountName: antrea-agent | ||
tolerations: | ||
- key: CriticalAddonsOnly | ||
operator: Exists | ||
- effect: NoSchedule | ||
operator: Exists | ||
- effect: NoExecute | ||
operator: Exists | ||
volumes: | ||
- configMap: | ||
name: antrea-config-5ct9ktdt77 | ||
name: antrea-config | ||
- hostPath: | ||
path: /etc/cni/net.d | ||
name: host-cni-conf | ||
- hostPath: | ||
path: /home/kubernetes/bin | ||
name: host-cni-bin | ||
- hostPath: | ||
path: /proc | ||
name: host-proc | ||
- hostPath: | ||
path: /var/run/netns | ||
name: host-var-run-netns | ||
- hostPath: | ||
path: /var/run/antrea | ||
type: DirectoryOrCreate | ||
name: host-var-run-antrea | ||
- hostPath: | ||
path: /var/log/antrea | ||
type: DirectoryOrCreate | ||
name: host-var-log-antrea | ||
- hostPath: | ||
path: /lib/modules | ||
name: host-lib-modules | ||
- hostPath: | ||
path: /run/xtables.lock | ||
type: FileOrCreate | ||
name: xtables-lock | ||
updateStrategy: | ||
type: RollingUpdate | ||
--- |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -167,7 +167,7 @@ export ENABLE_DOCKER_REGISTRY_CACHE=true | |
|
||
# Optional: Deploy a L7 loadbalancer controller to fulfill Ingress requests: | ||
# glbc - CE L7 Load Balancer Controller | ||
export ENABLE_L7_LOADBALANCING="${KUBE_ENABLE_L7_LOADBALANCING:-glbc}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bad merge? Why is this getting deleted? Or is the comment above meant to be preserved? |
||
|
||
|
||
# Optional: Enable Metrics Server. Metrics Server should be enable everywhere, | ||
# since it's a critical component, but in the first release we need a way to disable | ||
|
@@ -198,6 +198,17 @@ export MASTER_NODE_LABELS="${KUBE_MASTER_NODE_LABELS:-}" | |
NON_MASTER_NODE_LABELS="${KUBE_NON_MASTER_NODE_LABELS:-}" | ||
WINDOWS_NON_MASTER_NODE_LABELS="${WINDOWS_NON_MASTER_NODE_LABELS:-}" | ||
|
||
# Network Policy plugin specific settings for Linux. | ||
# none - No network policy plugin installed on Linux | ||
# calico - Install calico on Linux nodes to provide network policy support | ||
# antrea - Install antrea on Linux nodes to provide network policy support | ||
NETWORK_POLICY_PROVIDER="${NETWORK_POLICY_PROVIDER:-none}" | ||
|
||
# Network Policy plugin specific settings for Windows. | ||
# none - No network policy plugin installed on Windows | ||
# antrea - Install Antrea on Windows nodes to provide network policy support | ||
export WINDOWS_NETWORK_POLICY_PROVIDER="${WINDOWS_NETWORK_POLICY_PROVIDER:-none}" | ||
|
||
if [[ "${PREEMPTIBLE_MASTER}" == "true" ]]; then | ||
NODE_LABELS="${NODE_LABELS},cloud.google.com/gke-preemptible=true" | ||
WINDOWS_NODE_LABELS="${WINDOWS_NODE_LABELS},cloud.google.com/gke-preemptible=true" | ||
|
@@ -211,6 +222,8 @@ fi | |
# Windows nodes do not support Calico. | ||
if [[ ${NETWORK_POLICY_PROVIDER:-} == "calico" ]]; then | ||
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}projectcalico.org/ds-ready=true" | ||
elif [[ ${NETWORK_POLICY_PROVIDER:-} == 'antrea' ]] || [[ "${WINDOWS_NETWORK_POLICY_PROVIDER:-}" == "antrea" ]]; then | ||
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}antrea.io/ds-ready=true" | ||
fi | ||
|
||
# Optional: Enable netd. | ||
|
@@ -402,9 +415,6 @@ STORAGE_BACKEND=${STORAGE_BACKEND:-} | |
# Networking plugin specific settings. | ||
NETWORK_PROVIDER="${NETWORK_PROVIDER:-kubenet}" # none, kubenet | ||
|
||
# Network Policy plugin specific settings. | ||
NETWORK_POLICY_PROVIDER="${NETWORK_POLICY_PROVIDER:-none}" # calico | ||
|
||
export NON_MASQUERADE_CIDR="0.0.0.0/0" | ||
|
||
# How should the kubelet configure hairpin mode? | ||
|
@@ -566,3 +576,18 @@ export WINDOWS_ENABLE_DSR="${WINDOWS_ENABLE_DSR:-false}" | |
# TLS_CIPHER_SUITES defines cipher suites allowed to be used by kube-apiserver. | ||
# If this variable is unset or empty, kube-apiserver will allow its default set of cipher suites. | ||
export TLS_CIPHER_SUITES="" | ||
|
||
# Optional: URL to download antrea-cni.exe for Windows node | ||
export WINDOWS_ANTREA_CNI_BINARY_URL="${WINDOWS_ANTREA_CNI_BINARY_URL:-https://github.com/vmware-tanzu/antrea/releases/download/v0.13.1/antrea-cni-windows-x86_64.exe}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we want CNI and Agent at the same version? If so it may make sense to use a common version or URL base. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Most likely, but I am not assuming that. It could be different |
||
|
||
# Optional: URL to download antrea-agent.exe for Windows node | ||
export WINDOWS_ANTREA_AGENT_BINARY_URL="${WINDOWS_ANTREA_AGENT_BINARY_URL:-https://github.com/vmware-tanzu/antrea/releases/download/v0.13.1/antrea-cni-windows-x86_64.exe}" | ||
|
||
# Optional: URL to a script that downloads and installs OVS for Windows node | ||
export WINDOWS_OVS_INSTALLER_URL="${WINDOWS_OVS_INSTALLER_URL:-https://raw.githubusercontent.com/vmware-tanzu/antrea/v0.13.1/hack/windows/Install-OVS.ps1}" | ||
|
||
# Optional: Image project for Windows node | ||
WINDOWS_NODE_IMAGE_PROJECT=${WINDOWS_NODE_IMAGE_PROJECT:-windows-cloud} | ||
|
||
# Optional: Image name for Windows node | ||
WINDOWS_NODE_IMAGE=${WINDOWS_NODE_IMAGE:-} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use v2? v1 was built in 2016.