SDP Operator

SDP Operator is a Kubernetes operator to configure an Appgate SDP system.

Table of Contents

• Getting Started
  • Prerequisite
  • Installing the Normal Operator
  • Installing the Reverse Operator
    • GitHub
    • GitLab
  • Installing the Git Operator
• Examples
• Helm Values
• Advanced Usage
  • External Source for Secrets and Files
    • Secret Source
      • Hashicorp Vault
    • File Source
      • HTTP
      • S3
  • Encrypt Secrets with Fernet Key
  • CA Certificates
  • Dump Entities into YAML
  • Validate Entities against OpenAPI Spec

Getting Started

Prerequisite

The following tools are required to install the SDP Operator

• helm v3.8.0+ - https://helm.sh/docs/intro/install/
• kubectl - https://kubernetes.io/docs/tasks/tools/#kubectl

Browse available versions by navigating to the SDP Operator Releases

1. Add appgate/sdp-operator to your Helm repository

   helm repo add appgate https://appgate.github.io/sdp-operator/
   helm repo update

2. Install the SDP Operator CRD

   helm install sdp-operator-crd appgate/sdp-operator-crd \
       --namespace sdp-system \
       --create-namespace

Installing the Normal Operator

Standard Mode pushes entities on Kubernetes to an Appgate SDP system

1. Create a secret containing Admin API credentials

   kubectl create secret generic sdp-operator-secret \
       --from-literal=appgate-operator-user="<USERNAME>" \
       --from-literal=appgate-operator-password="<PASSWORD>" \
       --namespace sdp-system

2. Install the SDP Operator in Standard Mode

   sdp:
     operators:
       - sdp-operator
     sdpOperator:
       version: v18
       host: "https://sdp.appgate.com:8443"
       deviceId: "00000000-1111-2222-3333-44444444"
       secret: sdp-operator-secret
       reverseMode: false

   helm install normal-operator appgate/sdp-operator --values normal-operator.yaml --namespace sdp-system

Installing the Reverse Operator

Reverse Mode pulls SDP entities from an Appgate SDP system into Kubernetes

1. Create a secret containing Admin API credentials

   kubectl create secret generic sdp-operator-secret \
       --from-literal=appgate-operator-user="<USERNAME>" \
       --from-literal=appgate-operator-password="<PASSWORD>" \
       --namespace sdp-system

2. Install the SDP Operator in Reverse Mode

   sdp:
     operators:
       - sdp-operator
     sdpOperator:
       version: v18
       host: "https://sdp.appgate.com:8443"
       deviceId: "00000000-1111-2222-3333-44444444"
       secret: sdp-operator-secret
       reverseMode: true

   helm install reverse-operator appgate/sdp-operator --values reverse-operator.yaml --namespace sdp-system

Installing the Git Operator

Git Operator pushes SDP entities on Kubernetes to a Git repository and create pull requests on GitHub or GitLab

GitHub

1. Create a secret containing SSH key and GitHub token

   kubectl create secret generic github-operator-secret \
       --from-literal=github-token="<GITHUB_TOKEN>" \
       --from-file=git-ssh-key="<SSH_KEY_PATH>"

2. Install the Git Operator for GitHub

   sdp:
     operators:
       - git-operator
     gitOperator:
       version: v18
       secret: github-operator-secret
       git:
         vendor: github
         mainBranch: main
         baseBranch: main
         repository: <ORGANIZATION_OR_USER/REPOSITORY>

   helm install github-operator appgate/sdp-operator --values github-operator.yaml --namespace sdp-system

GitLab

GitLab (SaaS)

1. Create a secret containing SSH key and GitLab token

   kubectl create secret generic gitlab-operator-secret \
       --from-literal=gitlab-token="<GITLAB_TOKEN>" \
       --from-file=git-ssh-key="<SSH_KEY_PATH>"

2. Install the Git Operator for GitLab (SaaS)

   sdp:
     operators:
       - git-operator
     gitOperator:
       version: v18
       secret: gitlab-operator-secret
       git:
         vendor: gitlab
         mainBranch: main
         baseBranch: main
         repository: <ORGANIZATION_OR_USER/REPOSITORY>

   helm install gitlab-operator appgate/sdp-operator --values gitlab-operator.yaml --namespace sdp-system

GitLab (Self-Hosted)

1. Generate a file containing the host key fingerprint of the self-hosted GitLab

   ssh-keyscan <GITLAB_HOSTNAME> > <SSH_HOST_KEY_FINGERPRINT_PATH>

2. Create a secret containing SSH key and GitLab token

   kubectl create secret generic gitlab-operator-secret \
       --from-literal=gitlab-token="<GITLAB_TOKEN>" \
       --from-file=git-ssh-key="<SSH_KEY_PATH>" \
       --from-file=git-ssh-host-key-fingerprint="<SSH_HOST_KEY_FINGERPRINT_PATH>"

3. Install the Git Operator for self-hosted GitLab

   sdp:
     operators:
       - git-operator
     gitOperator:
       version: v18
       secret: gitlab-operator-secret
       git:
         vendor: gitlab
         mainBranch: main
         baseBranch: main
         repository: <ORGANIZATION_OR_USER/REPOSITORY>
         hostname: <GITLAB_HOSTNAME>

   helm install gitlab-operator appgate/sdp-operator --values gitlab-operator.yaml --namespace sdp-system

Examples

• Normal Operator
• Reverse Operator
• GitOps with SDP Operator
• Sync from Collective A to Collective B

Helm Values

For the list of available Helm parameters, please go to Helm Values

Advanced Usage

Configuring what entities to sync

The SDP Operator supports different of ways of specifying what to sync and how:

1. Configuring what entity tags to sync :: To filter based on the tags of the actual entities
2. Configuring what entity kinds to sync :: To only fetch entities of specific kinds

Note that both can be used together.

Configuring what tags to sync

We can specify what tags the operator will work on via the environemnt variable APPGATE_OPERATOR_TARGET_TAGS. This variable supports a list of tags (separated by comma) to sync. This means that entities that have any of those tags will be managed by the operator.

Example:

APPGATE_OPERATOR_TARGET_TAGS="devops,dev"

With that environemnt variable defined the operator will only create, modify or delete entities that any of the tags devops or dev (ot both at the same time).

We can also specify explicitly that we don't want to manage entities with specific tags. This can be done with the environemnt variable APPGATE_OPERATOR_EXCLUDE_TAGS. Entities with any of the tags listed there won't be deleted.

Note that if an entity has both tags it won't be deleted or if a tag is in both sets it will prevent entities with that tag to be deleted.

Configuring what entity kinds to sync

We saw how to use tags to decide what entities the operator should manage. In some occasions we don't want to load entities of some specific kind at all. To achive that we have 2 more environment variables:

• APPGATE_OPERATOR_INCLUDE_ENTITIES :: list of entity kinds we want to manage
• APPGATE_OPERATOR_EXCLUDE_ENTITIES :: list of entity kinds we don't want to manage

The list of entity kinds to use will be computed from the contents of those two environment variables. Note that APPGATE_OPERATOR_INCLUDE_ENTITIES has always precedence, this means that if we have the same entity kind in both sets then it will be included.

When the list of entity kinds is computed the operator will only create clients (SDP client, k8s watchers and git clients) for those entity kinds there. In practice that means that it won't load entities belonging to another kind.

Example:

APPGATE_OPERATOR_INCLUDE_ENTITIES="TrustedCertificate,Entitlement,Policy"

If the operator runs with that environment variable it will only create 3 SDP clients (for TrustedCertificate, Entitlements and Policies) and 3 K8S watchers.

External Source for Secrets and Files

By design, the Admin API does not return sensitive information and binary data in its response. That means, you cannot sync any entity from Collective A to Collective B if it contains any secret or file fields.

Suppose you want to sync a LocalUser from Collective A to Collective B. The reverse operator will sync the LocalUser entity from Collective A into Kubernetes without the password field (because the response to GET /localuser does not contain such field). If this entity is pushed to Collective B as-is, it would fail because password is a required value. To fix this issue, you can configure the normal operator to fetch and load the password value in memory before pushing it to Collective B.

The same logic applies to binary data. DeviceScript is an entity that contains a binary data in field file. By configuring an external source for files, you can load the binary data before pushing the entity to target collective.

SDP Operator expects secrets and files to be uploaded to the store in a specific format:

{Entity Type}-{API Version}/{Name}/{Key}

Configuring an External Secret Source

The following secret source are supported:

• Hashicorp Vault

Vault

Create a secret containing the Vault token

kubectl create secret generic sdp-operator-vault-secret \
    --from-literal=vault-token="<VAULT_TOKEN>"

Install the SDP Operator with Vault

sdp:    
  externalSecret:
    enabled: true
    type: vault
    source:
      vault:
        address: "https://vault.example.com:8200"
        tokenSecret: sdp-operator-vault-secret

# Note: Other required helm values are omitted for simplicity

SDP Operator expects secrets to be stored under /data/secret/sdp

Initialize the sdp path by uploading data for v18 LocalUser (username=john.doe password=password123)

vault kv put secret/sdp localuser-v18/john.doe/password="password123"

On subsequent uploads, you can use vault kv patch to update the secret. For example, to upload another v18 LocalUser (username=jane.doe, password=password123), run the following:

vault kv patch secret/sdp localuser-v18/jane.doe/password="password123"

Configuring an External File Source

The following external file source are supported:

• HTTP
• S3

HTTP

sdp:
  externalFile:
    enabled: true
    type: http
    source:
      http:
        address: "https://example.com:8000"

# Note: Other required helm values are omitted for simplicity

SDP Operator expects the files to be stored at the root of the filesystem.

Upload the file example.sh for v18 DeviceScript under the following path:

/devicescript-v18/example/file

S3

Create secret containing the access key and secret key for the S3 storage

kubectl create secret generic sdp-operator-s3-secret \
    --from-literal=access-key="<S3_ACCESS_KEY>"
    --from-literal=secret-key="<S3_SECRET_KEY>"

Install the SDP Operator with S3

sdp:
  externalFile:
    enabled: true
    type: s3
    source:
      s3:
        address: "http://example.com:8000"
        tokenSecret: sdp-operator-http-secret

# Note: Other required helm values are omitted for simplicity

SDP Operator expects the files to be stored under bucket sdp

Create a bucket /sdp on the root of the object store

Upload the file example.sh for v18 DeviceScript under the following path:

/sdp/devicescript-v18/example/file

Encrypt Secrets with Fernet Key

Sensitive information can be stored in the YAML as encrypted secrets using fernet (symmetric encryption).

To generate a new fernet key, run:

python3 -c 'from cryptography.fernet import Fernet;print(Fernet.generate_key().decode())'

To generate a secret with the key, run:

export SECRET="super-sensitive-information"
export KEY="dFVzzjKCa9mWbeig8dprliGLCXwnwE5Fbycz4Xe2ptk="
python3 -c 'from cryptography.fernet import Fernet;import os;print(Fernet(os.getenv("KEY")).encrypt(bytes(os.getenv("SECRET").encode())))'

After generating the secret, the encrypted value can be safely stored inside the YAML file as plain-text. When the operator encounters such field, it will read the value of environment variable APPGATE_OPERATOR_FERNET_KET to decrypt and read secrets in entities.

sdp:
  sdpOperator:
    fernetKey: "dFVzzjKCa9mWbeig8dprliGLCXwnwE5Fbycz4Xe2ptk="

CA Certificates

CA certificate in PEM format can be stored in env APPGATE_OPERATOR_CACERT. We recommend storing the contents of the PEM as a base64 encoded string.

sdp: 
  sdpOperator:
    caCert: "$(cat cert.ca)"

Dump Entities into YAML

dump-entities command will read the entities from an existing SDP system and dump them into YAML.

$ python3 -m appgate --spec-directory /appgate/appgate/api_specs/v15 dump-entities

The above command will generate a directory that contains the YAML

$ ls -l example-v15/
total 52
-rw-r--r-- 1 root root 1756 Jul 20 23:03 administrativerole.yaml
-rw-r--r-- 1 root root 3537 Jul 20 23:03 appliance.yaml
-rw-r--r-- 1 root root  143 Jul 20 23:03 clientconnection.yaml
-rw-r--r-- 1 root root  184 Jul 20 23:03 condition.yaml
-rw-r--r-- 1 root root  194 Jul 20 23:03 criteriascripts.yaml
-rw-r--r-- 1 root root  332 Jul 20 23:03 globalsettings.yaml
-rw-r--r-- 1 root root 1405 Jul 20 23:03 identityprovider.yaml
-rw-r--r-- 1 root root  423 Jul 20 23:03 ippool.yaml
-rw-r--r-- 1 root root  251 Jul 20 23:03 localuser.yaml
-rw-r--r-- 1 root root  479 Jul 20 23:03 mfaprovider.yaml
-rw-r--r-- 1 root root 1054 Jul 20 23:03 policy.yaml
-rw-r--r-- 1 root root  695 Jul 20 23:03 ringfencerule.yaml
-rw-r--r-- 1 root root  602 Jul 20 23:03 site.yaml

Validate Entities against an OpenAPI Spec

validate-entities command will validate the compatibility of entities against a version of the OpenAPI specification. This is useful for verifying if entities dumped from one SDP system is compatible with another SDP system.

$ python3 -m appgate --spec-directory /appgate/appgate/api_specs/v17 validate-entities examples-v15/

In the example above, we validated the v15 entities (generated by dump-entities command) to a v17 OpenAPI specification. The command will attempt to load all entities defined in examples-v15-entities/ as a v17 entities, reporting errors if encountered any.

Supporting new API versions

To support a new version of the API, you need to update bin/get-open-spec.sh and bin/unzip-open-spec.sh. Once it is updated, run

$ make api-specs

to download the new API specs. Then, run

$ VERSION=<new API version> make dump-crd

which will read the new specification, generate the CRDs, and dump them into k8s/crd/templates/<new API version>.yaml

How It Works

Custom Resource Definitions (CRD) on Kubernetes allow the operator to represent Appgate SDP entities as YAMLs. Each instance of an entity is stored as a Custom Resource. The operator reads each instance's spec and syncs the entity with the controller using the Admin API. SDP Operator consumes a version from Appgate SDP OpenAPI Spec to generate entities for a given version of the controller.

For each entity configured, the operator begins a timer and an event loop to listen for any changes happening on the cluster. Every time an event is received, the timer resets. After the timeout period has expired (in other words, when no event has newly arrived), the operator proceeds to compute a Plan. A Plan represents the difference between the current state on the controller vs the desired state defined in Kubernetes - it outlines what entities will be created/updated/deleted on the controller. After the plan is computed, the operator to execute the Plan on the controller using the API in order to produce the desired state.

It is important to note that, by design, any state defined in Kubernetes wins over state in the SDP system - any external changes made outside the operator will be overwritten. For example, if an administrator makes a change to the Policy via admin UI, the operator will determine the change as 'out-of-sync- and undoes the change.