Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: AWS marketplace & EKS add on integration (WIP) #86

Open
wants to merge 81 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
5842bac
feat: new chart for aws marketplace product. The product supports lic…
Sep 6, 2022
fbc595c
fix: qoute productSKU value; add logs for checkout license
Sep 6, 2022
20852b4
refactor: upgrade values.yaml with updated image pushed to aws mp pro…
Sep 6, 2022
ed91e2a
fix: support region, product, fingerprint and debug env variables; re…
Sep 7, 2022
44b1b82
fix: run license cron job on @daily. fix values naming bug
Sep 7, 2022
4012723
fix: upgrade image in aws mp helm chart values
Sep 7, 2022
0ae52ee
fix: remove awsmp-... chart. datree-admission-webhook chart should ho…
Sep 12, 2022
f61c9c7
fix: add * to ignore luanch.json file
Sep 12, 2022
32e1e4e
fix: remove ** to ignore luanch.json file
Sep 12, 2022
754366a
feat: added validation for value.yaml file
Sep 12, 2022
2beb9fa
fix: combine charts and added values.yaml for each option
Sep 19, 2022
cd9ef82
feat: script for packaing
Sep 19, 2022
17d8e07
fix: added script package for free-datree
Sep 19, 2022
3b5fa6d
feat: added chart
Sep 20, 2022
5ca2d41
fix: accidently deleted gh-pages values file with awsmp values file. …
Sep 21, 2022
be37723
fix: added aws.values
Sep 21, 2022
916151b
feat: added validation for value.yaml file
Sep 12, 2022
7dcfbbe
fix: combine charts and added values.yaml for each option
Sep 19, 2022
80c87f7
feat: script for packaing
Sep 19, 2022
8cc5a94
fix: added script package for free-datree
Sep 19, 2022
33758ae
feat: added chart
Sep 20, 2022
2eb9637
feat: charts file structure
Sep 20, 2022
7e86a32
fix: move tempaltes to lib template
shmu3l Sep 28, 2022
538ff06
Merge branch 'datree-lib-chart' into DAT-feat-datree-lib-chart
shmu3l Sep 28, 2022
ebab8d6
fix: remove templates use include from lib
shmu3l Sep 28, 2022
9002818
fix: remove templates use include from lib
shmu3l Sep 28, 2022
25ced28
fix: add dynamic webhook server alt name to cert
shmu3l Oct 2, 2022
c7eb9ae
fix: clear datree webhook values and update schema
shmu3l Oct 2, 2022
3a92aac
feat: awsmp datree use datree lib
shmu3l Oct 2, 2022
dc2b784
Merge pull request #99 from datreeio/DAT-awsmp-datree-use-lib
shmu3l Oct 2, 2022
31214ab
fix: awsmp file structure
shmu3l Oct 2, 2022
91ff33e
feat: add release admission webhook script
shmu3l Oct 3, 2022
a3f36c9
fix: release admission webhook
shmu3l Oct 3, 2022
3fa719e
fix: bump chart version
shmu3l Oct 3, 2022
68d2b7f
release chart 0.1.3
shmu3l Oct 3, 2022
9d53630
fix: release admission webhook helm script update
shmu3l Oct 6, 2022
4e8ef26
fix: update .gitignore
Oct 6, 2022
a7c79e8
fix: change messages
Oct 6, 2022
5f2fe54
fix: release admission webhook path scripts
shmu3l Oct 6, 2022
4a16912
fix: release admission webhook path scripts
shmu3l Oct 6, 2022
0aefb7b
fix: release admission webhook path scripts
shmu3l Oct 6, 2022
eb6f8e5
fix: rename script
Oct 6, 2022
ffbccfb
fix: change the order of helm file creations
Oct 6, 2022
6b7605e
fix: remove files
Oct 6, 2022
121fc8c
fix: release file index yaml
Oct 6, 2022
e775214
fix: dont stash pop
Oct 6, 2022
27c833d
feat: certificate alternative names use datree.namespace template.
Oct 18, 2022
5c19e06
fix: bump lib chart version
Oct 18, 2022
3c9e662
Merge pull request #142 from datreeio/DAT-support-certificate-release…
noaabarki Oct 18, 2022
d9dd178
fix: added dynamic namespace installation. Rename chart according to …
Oct 24, 2022
890d179
fix: aligned charts with aws-marketplace ECR repositories.
Nov 3, 2022
19585b5
feat: pull bitnami-kubectl from private ecr registry
Nov 3, 2022
4e04618
fix: updated bitnami in free offer chart
Nov 3, 2022
49aff40
feat: added new binary in cmd folder for init container
Nov 7, 2022
6930693
wip
Nov 8, 2022
c08df99
wip: poc of webhook race condition
Nov 9, 2022
6055484
fix: working product on minikube. NOTE: very slow
Nov 10, 2022
8d1eb85
fix: working version
Nov 13, 2022
976d208
fix: wokring production version on minikube
Nov 13, 2022
89e7bda
fix: working production version. Minikube. 3minutes sleep time
Nov 13, 2022
674558d
fix: working version ECR repositories on Faragate.
Nov 13, 2022
b986d9b
feat: support uninstall Datree product
Nov 13, 2022
8fd5105
fix: working version in AWS Marketplace. Datree product
Nov 13, 2022
790d488
feat: working version. Datree Free product. version 1.0.1-rc.1
Nov 13, 2022
332ecbd
feat: check for webhook existence, wait for running pods. Refactor
Nov 15, 2022
14277c3
fix: wait for all deployment replicas to be ready. updated ecr images
Nov 15, 2022
811dd1b
fix: update aws mp Chart
Nov 16, 2022
5e6d6d7
fix: remove .vscode luanch.json
Nov 16, 2022
acc7591
fix: update Dockerfiles
Nov 16, 2022
64eb1c5
fix: remove comments
Nov 16, 2022
04ace8c
Merge pull request #179 from datreeio/DAT-custom-validation-webhook
noaabarki Nov 16, 2022
b2999c6
test: added tests for cert-generator
Nov 16, 2022
1c47849
test: wip
Nov 20, 2022
74e9c60
test: fixed table testing for k8sClient
Nov 20, 2022
feb8e8d
test: cert-generator tests
Nov 21, 2022
c766a92
fix: structure packages in webhook-init to not use one another, only …
Nov 21, 2022
43a6ea2
fix: updated makefile
Nov 21, 2022
3c0a416
test: ensure empty env variables has defaults
Nov 21, 2022
ddeddfd
fix: bump images versions, working product on minikube.
Nov 21, 2022
0c9ca9d
Merge pull request #92 from datreeio/DAT-feat-datree-lib-chart
noaabarki Nov 21, 2022
08bef3a
Merge pull request #91 from datreeio/DAT-chart-values-validation
noaabarki Nov 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
wip
  • Loading branch information
Noaa Barki authored and Noaa Barki committed Nov 8, 2022
commit 69306939356c838aebaa536b64a45ab4b5b53351
11 changes: 6 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -2,7 +2,6 @@ FROM golang:1.18-alpine AS builder

ARG BUILD_ENVIRONMENT
ARG WEBHOOK_VERSION
ARG BINARY_PATH

WORKDIR /go/src/app

@@ -11,10 +10,12 @@ COPY go.* .
RUN go mod download

COPY . .
# cache the build
RUN --mount=type=cache,target=/root/.cache/go-build go build ./cmd/cert-generator -tags $BUILD_ENVIRONMENT -ldflags="-X github.com/datreeio/admission-webhook-datree/pkg/config.WebhookVersion=$WEBHOOK_VERSION" -o webhook-datree
# cache the build,
## map the /root/.cache/go-build to your host go build cache folder
# RUN --mount=type=cache,target=/root/.cache/go-build go build ./cmd/webhook-datree -tags $BUILD_ENVIRONMENT -ldflags="-X github.com/datreeio/admission-webhook-datree/pkg/config.WebhookVersion=$WEBHOOK_VERSION" -o webhook-datree
RUN go build ./cmd/webhook-server

FROM alpine:3.14
COPY --from=builder /go/src/app/webhook-datree /
COPY --from=builder /go/src/app/webhook-server /
EXPOSE 8443
ENTRYPOINT ["/webhook-datree"]
ENTRYPOINT ["/webhook-server"]
24 changes: 24 additions & 0 deletions Dockerfile.init
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
FROM golang:1.18-alpine AS builder

ARG BUILD_ENVIRONMENT
ARG WEBHOOK_VERSION

WORKDIR /go/src/app



# download dependencies in a separate step to allow caching
COPY go.* .
RUN go mod download

COPY . .
# cache the build
# RUN --mount=type=cache,target=/root/.cache/go-build go build cmd/webhook-server -o webhook-server
RUN go build ./cmd/cert-generator

# RUN --mount=type=cache,target=/root/.cache/go-build go build -tags $BUILD_ENVIRONMENT -ldflags="-X github.com/datreeio/admission-webhook-datree/pkg/config.WebhookVersion=$WEBHOOK_VERSION" -o webhook-datree

FROM alpine:3.14
COPY --from=builder /go/src/app/cert-generator /
EXPOSE 8443
ENTRYPOINT ["/cert-generator"]
Binary file modified charts/datree-admission-webhook-awsmp/charts/datree-lib-0.1.1.tgz
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# {{- include "datree-lib.namespace-post-delete" .}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# {{- include "datree-lib.namespace-post-install" .}}
152 changes: 78 additions & 74 deletions charts/datree-admission-webhook-awsmp/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -6,8 +6,11 @@ metadata:
labels: {{ include "datree.labels" . | nindent 4 }}
owner: datree
app: "datree-webhook-server"
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-weight": "5"
{{- if .Values.customAnnotations }}
annotations: {{- toYaml .Values.customAnnotations | nindent 4 }}
{{- toYaml .Values.customAnnotations | nindent 4 }}
{{- end }}
spec:
replicas: {{ .Values.replicaCount }}
@@ -23,88 +26,89 @@ spec:
{{- end }}
spec:
serviceAccountName: {{.Values.rbac.serviceAccount.name}}
initContainers:
image: <webhook init-image name>
imagePullPolicy: IfNotPresent
name: webhook-init
volumeMounts:
- mountPath: /etc/webhook/certs
name: webhook-certs
env:
- name: WEBHOOK_NAMESPACE
value: {{ template "datree.namespace" . }}
- name: WEBHOOK_SERVICE
value: "datree-webhook-server"
containers:
- name: server
securityContext: {{- toYaml .Values.securityContext | nindent 12 }}
livenessProbe:
- name: server
securityContext: {{- toYaml .Values.securityContext | nindent 12 }}
livenessProbe:
httpGet:
path: /health
port: 8443
scheme: HTTPS
path: /health
port: 8443
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
readinessProbe:
httpGet:
path: /ready
port: 8443
scheme: HTTPS
path: /ready
port: 8443
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 10
resources: {{- toYaml .Values.resources | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{.Values.image.pullPolicy}}
ports:
- containerPort: 8443
name: webhook-api
# caution: don't change the order of the environment variables
# changing the order will harm resource patching
env:
- name: DEBUG
value: "{{ .Values.debug }}"
# Datree webhook varaibles
- name: DATREE_TOKEN
value: {{.Values.datree.token}}
- name: DATREE_POLICY
value: {{.Values.datree.policy}}
- name: DATREE_VERBOSE
value: {{.Values.datree.verbose}}
- name: DATREE_OUTPUT
value: {{.Values.datree.output}}
- name: DATREE_NO_RECORD
value: {{.Values.datree.noRecord}}
# AWS Marketplace varaibles
- name: AWS_MP_PRODUCT_ID
value: {{ .Values.aws.productId }}
- name: AWS_MP_KEY_FINGERPRINT
value: {{ .Values.aws.issuerKey }}
- name: AWS_MP_ENABLE_CHECK_ENTITLEMENT
value: "{{.Values.aws.enableCheckEntitlement}}"
- name: AWS_MP_REGION
value: {{.Values.aws.region}}
{{- if .Values.aws.licenseConfigSecretName }}
- name: AWS_WEB_IDENTITY_REFRESH_TOKEN_FILE
value: "/var/run/secrets/awsmp-product-license/license_token"
- name: AWS_ROLE_ARN
valueFrom:
secretKeyRef:
name: {{ .Values.aws.licenseConfigSecretName }}
key: iam_role
{{- end}}
# add aws marketplace license config to the licensed container application env
volumeMounts:
- mountPath: /etc/webhook/certs
name: webhook-certs
readOnly: true
{{- if .Values.aws.licenseConfigSecretName }}
- name: awsmp-product-license
mountPath: "/var/run/secrets/awsmp-product-license"
{{- end}}
resources: {{- toYaml .Values.resources | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{.Values.image.pullPolicy}}
ports:
- containerPort: 8443
name: webhook-api
# caution: don't change the order of the environment variables
# changing the order will harm resource patching
env:
- name: DEBUG
value: "{{ .Values.debug }}"
# Datree webhook varaibles
- name: DATREE_TOKEN
value: {{.Values.datree.token}}
- name: DATREE_POLICY
value: {{.Values.datree.policy}}
- name: DATREE_VERBOSE
value: {{.Values.datree.verbose}}
- name: DATREE_OUTPUT
value: {{.Values.datree.output}}
- name: DATREE_NO_RECORD
value: {{.Values.datree.noRecord}}
# AWS Marketplace varaibles
- name: AWS_MP_PRODUCT_ID
value: {{ .Values.aws.productId }}
- name: AWS_MP_KEY_FINGERPRINT
value: {{ .Values.aws.issuerKey }}
- name: AWS_MP_ENABLE_CHECK_ENTITLEMENT
value: "{{.Values.aws.enableCheckEntitlement}}"
- name: AWS_MP_REGION
value: {{.Values.aws.region}}
{{- if .Values.aws.licenseConfigSecretName }}
- name: AWS_WEB_IDENTITY_REFRESH_TOKEN_FILE
value: "/var/run/secrets/awsmp-product-license/license_token"
- name: AWS_ROLE_ARN
valueFrom:
secretKeyRef:
name: {{ .Values.aws.licenseConfigSecretName }}
key: iam_role
{{- end}}
# add aws marketplace license config to the licensed container application env
volumeMounts:
- mountPath: /etc/webhook/certs
name: webhook-certs
readOnly: true
{{- if .Values.aws.licenseConfigSecretName }}
- name: awsmp-product-license
mountPath: "/var/run/secrets/awsmp-product-license"
{{- end}}
volumes:
- name: webhook-certs
emptyDir: {}
{{- if .Values.aws.licenseConfigSecretName }}
{{- if .Values.aws.licenseConfigSecretName }}
- name: awsmp-product-license
secret:
secretName: {{ .Values.aws.licenseConfigSecretName }}
{{- end}}
{{- end}}
initContainers:
- name: webhook-init
image: "{{ .Values.initContainer.repository }}@{{ .Values.initContainer.sha }}"
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /etc/webhook/certs
name: webhook-certs
env:
- name: WEBHOOK_NAMESPACE
value: {{ template "datree.namespace" . }}
- name: WEBHOOK_SERVICE
value: "datree-webhook-server"

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

8 changes: 6 additions & 2 deletions charts/datree-admission-webhook-awsmp/values.yaml
Original file line number Diff line number Diff line change
@@ -44,12 +44,16 @@ datree:
# The Datree webhook-server image to use.
image:
# Image repository
repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/datree/datree-admission-webhook-awsmp
repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/datree/awsmp-datree-admission-webhook
# Image tag
tag: 0.1.24-rc.1
tag: 0.1.25-rc.3
# Image pull policy
pullPolicy: Always

initContainer:
repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/datree/awsmp-datree-admission-webhook
sha: sha256:694cc1a664315582bb9414f3ef255f3cbd730d361b6dfa2bea2be45dd1801d37

# Security context for the containers
securityContext:
allowPrivilegeEscalation: false
7 changes: 7 additions & 0 deletions charts/datree-lib/templates/_clusterrole.yaml
Original file line number Diff line number Diff line change
@@ -11,12 +11,19 @@ metadata:
rules:
- apiGroups:
- ""
- "admissionregistration.k8s.io"
resources:
- "nodes"
- "namespaces"
- "validatingwebhookconfigurations"
verbs:
- "get"
- "list"
- "create"
- "delete"
- "patch"
- "update"
- "watch"
{{- end}}
---
apiVersion: rbac.authorization.k8s.io/v1
27 changes: 17 additions & 10 deletions cmd/cert-generator/main.go
Original file line number Diff line number Diff line change
@@ -22,15 +22,22 @@ import (
)

func main() {
loggerUtil.Log("starting cert-generator")
caCert, _ := generateSelfSignedCAAndSignWebhookServerCertificate()
createValidationWebhookConfig(caCert)
loggerUtil.Log(fmt.Sprintf("created ca certificate, caCert: %v", caCert))
err := createValidationWebhookConfig(caCert)
if err != nil {
loggerUtil.Log(fmt.Sprintf("failed to create validation webhook config, err: %v", err))
} else {
loggerUtil.Log("created validating webhook configuration")
}
}

func createValidationWebhookConfig(caCert *bytes.Buffer) {
func createValidationWebhookConfig(caCert *bytes.Buffer) error {
config := ctrl.GetConfigOrDie()
kubeClient, err := kubernetes.NewForConfig(config)
if err != nil {
panic("failed to set go -client")
return err // panic("failed to set go -client")
}

webhookNamespace, _ := os.LookupEnv("WEBHOOK_NAMESPACE")
@@ -49,7 +56,7 @@ func createValidationWebhookConfig(caCert *bytes.Buffer) {
ClientConfig: admissionregistrationv1.WebhookClientConfig{
CABundle: caCert.Bytes(), // CA bundle created earlier
Service: &admissionregistrationv1.ServiceReference{
Name: webhookService,
Name: webhookService, // datree-webhook-server
Namespace: webhookNamespace,
Path: &path,
},
@@ -79,8 +86,10 @@ func createValidationWebhookConfig(caCert *bytes.Buffer) {
}

if _, err = kubeClient.AdmissionregistrationV1().ValidatingWebhookConfigurations().Create(context.Background(), validationWebhookConfig, metav1.CreateOptions{}); err != nil {
panic(err)
return err
}

return nil
}

func generateSelfSignedCAAndSignWebhookServerCertificate() (*bytes.Buffer, error) {
@@ -100,11 +109,9 @@ func generateSelfSignedCAAndSignWebhookServerCertificate() (*bytes.Buffer, error
Bytes: caBytes,
})

dnsNames := []string{"datree-webhook-server",
"datree-webhook-server.default",
"datree-webhook-server.default.svc",
"datree-webhook-server.datree",
"datree-webhook-server.datree.svc",
webhookNamespace, _ := os.LookupEnv("WEBHOOK_NAMESPACE")
dnsNames := []string{
fmt.Sprintf("datree-webhook-server.%s.svc", webhookNamespace),
}

commonName := "/CN=datree-webhook-server.datree.svc"
3 changes: 3 additions & 0 deletions cmd/webhook-server/main.go
Original file line number Diff line number Diff line change
@@ -9,6 +9,7 @@ import (
"github.com/datreeio/admission-webhook-datree/pkg/controllers"
"github.com/datreeio/admission-webhook-datree/pkg/errorReporter"
"github.com/datreeio/admission-webhook-datree/pkg/k8sMetadataUtil"
"github.com/datreeio/admission-webhook-datree/pkg/loggerUtil"
"github.com/datreeio/admission-webhook-datree/pkg/server"
"github.com/datreeio/datree/pkg/cliClient"
"github.com/datreeio/datree/pkg/deploymentConfig"
@@ -43,6 +44,7 @@ func start(port string) {
}
}()

loggerUtil.Log("initializing k8s metadata")
k8sMetadataUtil.InitK8sMetadataUtil()

certPath, keyPath, err := server.ValidateCertificate()
@@ -58,6 +60,7 @@ func start(port string) {
http.HandleFunc("/ready", healthController.Ready)

// start server
loggerUtil.Log("strting server")
if err := http.ListenAndServeTLS(":"+port, certPath, keyPath, nil); err != nil {
http.ListenAndServe(":"+port, nil)
}
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ module github.com/datreeio/admission-webhook-datree
go 1.18

require (
github.com/datreeio/datree v1.7.1
github.com/datreeio/datree v1.6.6
github.com/ghodss/yaml v1.0.0
github.com/lithammer/shortuuid v3.0.0+incompatible
github.com/stretchr/testify v1.7.1
2 changes: 1 addition & 1 deletion pkg/server/server.go
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ import (
)

func ValidateCertificate() (certPath string, keyPath string, err error) {
tlsDir := `/etc/webhook/certs/`
tlsDir := `/etc/webhook/certs`
tlsCertFile := `tls.crt`
tlsKeyFile := `tls.key`

3 changes: 2 additions & 1 deletion pkg/services/validationService.go
Original file line number Diff line number Diff line change
@@ -3,12 +3,13 @@ package services
import (
"encoding/json"
"fmt"
"github.com/datreeio/admission-webhook-datree/pkg/loggerUtil"
"net/http"
"os"
"strings"
"time"

"github.com/datreeio/admission-webhook-datree/pkg/loggerUtil"

cliDefaultRules "github.com/datreeio/datree/pkg/defaultRules"

"k8s.io/utils/strings/slices"