Skip to content

Commit

Permalink
Add best practices documentation and update other docs
Browse files Browse the repository at this point in the history 
* Added general and security best practices sections.
* Updated FAQ to discuss CVE patching.
* Moved to use more visible warnings/notes in ASOv1 -> ASOv2 migration
  guide.

This fixes #3261.
  • Loading branch information
@matthchr
matthchr committed Apr 16, 2024
1 parent 5f3c5d0 commit f56ca7b
Show file tree
Hide file tree
Showing 9 changed files with 339 additions and 111 deletions.
152 changes: 102 additions & 50 deletions docs/hugo/content/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,52 @@ cert-manager-webhook-c4b5687dc-x66bj 1/1 Running 0 1m

(Alternatively, you can wait for cert-manager to be ready with `cmctl check api --wait=2m` - see the [cert-manager documentation](https://cert-manager.io/docs/usage/cmctl/) for more information about `cmctl`.)

2. Create an Azure Service Principal. You'll need this to grant Azure Service Operator permissions to create resources in your subscription.
2. Install [the latest **v2+** Helm chart](https://github.com/Azure/azure-service-operator/tree/main/v2/charts):

{{< tabpane text=true left=true >}}
{{% tab header="**Shell**:" disabled=true /%}}
{{% tab header="bash" %}}
``` bash
$ helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
$ helm upgrade --install aso2 aso2/azure-service-operator \
--create-namespace \
--namespace=azureserviceoperator-system \
--set crdPattern='resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*'
```
Note: **bash** requires the value for `crdPattern` to be quoted with `'` to avoid expansion of the wildcards.
{{% /tab %}}
{{% tab header="PowerShell" %}}
``` powershell
PS> helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
PS> helm upgrade --install aso2 aso2/azure-service-operator `
--create-namespace `
--namespace=azureserviceoperator-system `
--set crdPattern=resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*
```
{{% /tab %}}
{{% tab header="CMD" %}}
``` cmd
C:\> helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
C:\> helm upgrade --install aso2 aso2/azure-service-operator ^
--create-namespace ^
--namespace=azureserviceoperator-system ^
--set crdPattern=resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*
```
{{% /tab %}}
{{< /tabpane >}}

{{% alert title="Warning" color="warning" %}}
Make sure to set the `crdPattern` variable to include the CRDs you are interested in using.
You can use `--set crdPattern=*` to install all the CRDs, but be aware of the
[limits of the Kubernetes you are running](https://github.com/Azure/azure-service-operator/issues/2920). `*` is **not**
recommended on AKS Free-tier clusters.
See [CRD management](https://azure.github.io/azure-service-operator/guide/crd-management/) for more details.
{{% /alert %}}

Alternatively you can install from the [release YAML directly](https://azure.github.io/azure-service-operator/guide/installing-from-yaml/).

3. Create an Azure Service Principal. You'll need this to grant Azure Service Operator permissions to create resources
in your subscription.

First, set the following environment variables to your Azure Tenant ID and Subscription ID with your values:

Expand All @@ -78,13 +123,13 @@ C:\> SET AZURE_SUBSCRIPTION_ID=<your-subscription-id-goes-here>
{{% /tab %}}
{{< /tabpane >}}

You can find these values by using the Azure CLI: `az account show`
You can find these values by using the Azure CLI: `az account show`

Next, create a service principal with Contributor permissions for your subscription.
Next, create a service principal with Contributor permissions for your subscription.

You can optionally use a service principal with a more restricted permission set
(for example contributor to just a Resource Group), but that will restrict what you can
do with ASO. See [using reduced permissions](https://azure.github.io/azure-service-operator/guide/authentication/reducing-access/#using-a-credential-for-aso-with-reduced-permissions) for more details.
You can optionally use a service principal with a more restricted permission set
(for example contributor to just a Resource Group), but that will restrict what you can
do with ASO. See [using reduced permissions](https://azure.github.io/azure-service-operator/guide/authentication/reducing-access/#using-a-credential-for-aso-with-reduced-permissions) for more details.

{{< tabpane text=true left=true >}}
{{% tab header="**Shell**:" disabled=true /%}}
Expand Down Expand Up @@ -144,63 +189,69 @@ C:\> SET AZURE_CLIENT_SECRET=<your-client-secret>
{{% /tab %}}
{{< /tabpane >}}

3. Install [the latest **v2+** Helm chart](https://github.com/Azure/azure-service-operator/tree/main/v2/charts):
Then create a secret named `aso-credential` in the namespace you'd like to create ASO resources in.

{{< tabpane text=true left=true >}}
{{% tab header="**Shell**:" disabled=true /%}}
{{% tab header="bash" %}}
``` bash
$ helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
$ helm upgrade --install aso2 aso2/azure-service-operator \
--create-namespace \
--namespace=azureserviceoperator-system \
--set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
--set azureTenantID=$AZURE_TENANT_ID \
--set azureClientID=$AZURE_CLIENT_ID \
--set azureClientSecret=$AZURE_CLIENT_SECRET \
--set crdPattern='resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*'
```bash
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: aso-credential
namespace: default
stringData:
AZURE_SUBSCRIPTION_ID: "$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID: "$AZURE_TENANT_ID"
AZURE_CLIENT_ID: "$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET: "$AZURE_CLIENT_SECRET"
EOF
```
Note: **bash** requires the value for `crdPattern` to be quoted with `'` to avoid expansion of the wildcards.
{{% /tab %}}

{{% tab header="PowerShell" %}}
``` powershell
PS> helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
PS> helm upgrade --install aso2 aso2/azure-service-operator `
--create-namespace `
--namespace=azureserviceoperator-system `
--set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID `
--set azureTenantID=$AZURE_TENANT_ID `
--set azureClientID=$AZURE_CLIENT_ID `
--set azureClientSecret=$AZURE_CLIENT_SECRET `
--set crdPattern=resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*
```powershell
@"
apiVersion: v1
kind: Secret
metadata:
name: aso-credential
namespace: default
stringData:
AZURE_SUBSCRIPTION_ID: "$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID: "$AZURE_TENANT_ID"
AZURE_CLIENT_ID: "$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET: "$AZURE_CLIENT_SECRET"
"@ | kubectl apply -f -
```
{{% /tab %}}

{{% tab header="CMD" %}}
``` cmd
C:\> helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
C:\> helm upgrade --install aso2 aso2/azure-service-operator ^
--create-namespace ^
--namespace=azureserviceoperator-system ^
--set azureSubscriptionID=%AZURE_SUBSCRIPTION_ID% ^
--set azureTenantID=%AZURE_TENANT_ID% ^
--set azureClientID=%AZURE_CLIENT_ID% ^
--set azureClientSecret=%AZURE_CLIENT_SECRET% ^
--set crdPattern=resources.azure.com/*;containerservice.azure.com/*;keyvault.azure.com/*;managedidentity.azure.com/*;eventhub.azure.com/*

Create a file named `secret.yaml` with the following content. Replace each of the variables such as
`%AZURE_SUBSCRIPTION_ID%` with the subscription ID, `%AZURE_CLIENT_SECRET%` with the client secret and so on.

```
apiVersion: v1
kind: Secret
metadata:
name: aso-credential
namespace: default
stringData:
AZURE_SUBSCRIPTION_ID: "%AZURE_SUBSCRIPTION_ID%"
AZURE_TENANT_ID: "%AZURE_TENANT_ID%"
AZURE_CLIENT_ID: "%AZURE_CLIENT_ID%"
AZURE_CLIENT_SECRET: "%AZURE_CLIENT_SECRET%"
```
{{% /tab %}}
{{< /tabpane >}}

{{% alert title="Warning" color="warning" %}}
Make sure to set the `crdPattern` variable to include the CRDs you are interested in using.
You can use `--set crdPattern=*` to install all the CRDs, but be aware of the
[limits of the Kubernetes you are running](https://github.com/Azure/azure-service-operator/issues/2920). `*` is **not**
recommended on AKS Free-tier clusters.
See [CRD management](https://azure.github.io/azure-service-operator/guide/crd-management/) for more details.
{{% /alert %}}
Then run: `kubectl apply -f secret.yaml`

Alternatively you can install from the [release YAML directly](https://azure.github.io/azure-service-operator/guide/installing-from-yaml/).
{{% /tab %}}
{{< /tabpane >}}

To learn more about other authentication options, see the [authentication documentation](https://azure.github.io/azure-service-operator/guide/authentication/).
To learn more about other authentication options, see the
[authentication documentation](https://azure.github.io/azure-service-operator/guide/authentication/).

### Usage

Expand All @@ -218,7 +269,8 @@ To view the logs for the running ASO controller, take note of the pod name shown
$ kubectl logs -n azureserviceoperator-system azureserviceoperator-controller-manager-5b4bfc59df-lfpqf manager
```

Let's create an Azure ResourceGroup in westcentralus with the name "aso-sample-rg". Create a file called `rg.yaml` with the following contents:
Let's create an Azure ResourceGroup in westcentralus with the name "aso-sample-rg". Create a file called `rg.yaml`
with the following contents:

```yaml
apiVersion: resources.azure.com/v1api20200601
Expand Down
4 changes: 4 additions & 0 deletions docs/hugo/content/guide/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,5 +94,9 @@ How does ASO manage a [large number of CRDs]( {{< relref "crd-management" >}})?

ASO exposes [metrics]( {{< relref "metrics" >}}) for Prometheus.

### Best practices

See [best practices]( {{< relref "best-practices" >}}) for ASO best practices.

{{< /card >}}
{{< /cardpane >}}
104 changes: 65 additions & 39 deletions docs/hugo/content/guide/asov1-asov2-migration/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,23 +33,34 @@ kubectl api-resources --api-group cert-manager.io`
```
should show `v1` resources.

> ℹ️ **Note:** We strongly recommend ensuring that you're running the latest version of cert-manager
> (1.14.4 at the time this article was written).
> ℹ️ **Note:** We also recommend ensuring that ASOv1 is configured to use `cert-manager.io/v1` resources.
> You can run `helm upgrade` and pass `--set certManagerResourcesAPIVersion=cert-manager.io/v1` to ensure ASOv1 is using
> the v1 versions of the cert-manager resources.
{{% alert title="Warning" color="warning" %}}
Ensure that you've verified each ASOv1 resource you delete has the `skipreconcile=true` annotation before you delete it
in Kubernetes or else the deletion will propagate to Azure and delete the underlying Azure resource as well, which you do not want.
{{% /alert %}}

{{% alert title="Note" %}}
We strongly recommend ensuring that you're running the latest version of cert-manager
(1.14.4 at the time this article was written).
{{% /alert %}}

{{% alert title="Note" %}}
We also recommend ensuring that ASOv1 is configured to use `cert-manager.io/v1` resources.
You can run `helm upgrade` and pass `--set certManagerResourcesAPIVersion=cert-manager.io/v1` to ensure ASOv1 is using
the v1 versions of the cert-manager resources.
{{% /alert %}}

### Install ASOv2

Follow the [standard instructions](../../#installation). We recommend you use the same credentials as ASOv1 is currently using.

> ℹ️ **Note:** Make sure to follow the [guidelines](../crd-management/) for setting the `--crdPattern`. Configure ASOv2
> to only manage the CRDs which you need.
>
> For example: if you are using storage accounts, resource groups, redis cache's, cosmosdbs, eventhubs, and SQL Azure,
> then you would configure
> `--set crdPattern='resources.azure.com/*;cache.azure.com/*;documentdb.azure.com/*;eventhub.azure.com/*;sql.azure.com/*;storage.azure.com/*'`
{{% alert title="Note" %}}
Make sure to follow the [guidelines](../crd-management/) for setting the `--crdPattern`. Configure ASOv2
to only manage the CRDs which you need.

For example: if you are using storage accounts, resource groups, redis cache's, cosmosdbs, eventhubs, and SQL Azure,
then you would configure
`--set crdPattern='resources.azure.com/*;cache.azure.com/*;documentdb.azure.com/*;eventhub.azure.com/*;sql.azure.com/*;storage.azure.com/*'`
{{% /alert %}}

### Stop ASOv1 reconciliation

Expand All @@ -68,8 +79,10 @@ kubectl annotate $(kubectl api-resources -o name | grep azure.microsoft.com | pa

Once you have annotated the resources, double-check that they all have the `skipreconcile` annotation.

> ⚠️ **Important**: If you delete an ASOv1 resource that does not have this annotation it will delete the underlying
> Azure resource which may cause downtime or an outage.
{{% alert title="Warning" color="warning" %}}
If you delete an ASOv1 resource that does not have this annotation it _**will**_ delete the underlying
Azure resource which may cause downtime or an outage.
{{% /alert %}}

### Use `asoctl` to import the resources from Azure into ASOv2

Expand Down Expand Up @@ -147,13 +160,17 @@ spec:

Once you have `resources.yaml` locally, examine it to ensure that it has the resources you expect.

> ℹ️ **Note**: `asoctl` is importing the resources from Azure. This means it will not preserve any Kubernetes labels/annotations
> you have on your ASOv1 resources. If your resources need certain labels or annotations, add those to `resources.yaml`
> manually or use the `--annotation` and `--label` arguments to `asoctl`.
{{% alert title="Note" %}}
`asoctl` is importing the resources from Azure. This means it cannot preserve any Kubernetes labels/annotations
you have on your ASOv1 resources. If your resources need certain labels or annotations, add those to `resources.yaml`
manually or use the `--annotation` and `--label` arguments to `asoctl`.
{{% /alert %}}

> ℹ️ **Note**: By default `asoctl` imports everything into the `default` namespace. Before applying any YAML
> created by `asoctl`, ensure that the namespaces for the resources are correct. You can do this by manually
> modifying the YAML, using a tool such as Kustomize, or using the `--namespace` argument to `asoctl`.
{{% alert title="Note" %}}
By default `asoctl` imports everything into the `default` namespace. Before applying any YAML
created by `asoctl`, ensure that the namespaces for the resources are correct. You can do this by manually
modifying the YAML, using a tool such as Kustomize, or using the `--namespace` argument to `asoctl`.
{{% /alert %}}

### Configure `resources.yaml` to export the secrets you need from Azure

Expand All @@ -164,11 +181,13 @@ ASOv1 exports secrets from Azure according to the
rules. ASOv1 always exports these secrets, there is no user configuration required on either the name of the secret or its values.
In contrast, ASOv2 requires the user to opt-in to secret export, via the `spec.operatorSpec.secrets` property.

> ℹ️ **Note**: ASOv2 also splits secrets/configuration into two categories: Secrets provided by you, and secrets generated
> by Azure. Exporting secrets from Azure is done via the `spec.operatorSpec.secrets`, while supplying secrets to Azure
> is done by passing the reference to a secret key/value, such as via the
> [administratorLoginPassword field of Azure SQL](../../reference/sql/v1api20211101#sql.azure.com/v1api20211101.Server_Spec).
> This means that there may be cases where a single secret in ASOv1 becomes 2 secrets in ASOv2, one for inputs and one for outputs.
{{% alert title="Note" %}}
ASOv2 also splits secrets into two categories: Secrets provided by you, and secrets generated
by Azure. Exporting secrets from Azure is done via the `spec.operatorSpec.secrets`, while supplying secrets to Azure
is done by passing the reference to a secret key/value, such as via the
[administratorLoginPassword field of Azure SQL](../../reference/sql/v1api20211101#sql.azure.com/v1api20211101.Server_Spec).
This means that there may be cases where a single secret in ASOv1 becomes 2 secrets in ASOv2, one for inputs and one for outputs.
{{% /alert %}}

You may need to configure `resouces.yaml` to export corresponding secrets. To determine if, for a given namespace,
there are secrets being written by ASOv1 and consumed by your applications, check the following two things:
Expand Down Expand Up @@ -206,13 +225,16 @@ Once you've identified the set of secrets which are exported by ASOv1 and which
configure ASOv2 to export similar secrets by using the `spec.operatorSpec.secrets` field. See [examples](#examples) for
examples of various resource types.

> ℹ️ **Note:** It is strongly recommended that you export the ASOv2 secrets to a _different_ secret name than the
> ASOv1 secret. ASOv2 will not allow you to overwrite an existing secret with `spec.operatorSpec.secrets`.
> You'll get an error that looks like
> "cannot overwrite Secret ns1/storageaccount-cutoverteststorage1 which is not owned by StorageAccount.storage.azure.com
> ns1/aso-migration-test-cutoverteststorage1".
> Instead of overwriting the same secret, create a different secret with ASOv2 (recommend including a suffix to
> identify it such as -asov2). Then, when you're ready, swap your deployment to use the new ASOv2 secrets.

{{% alert title="Note" %}}
It is strongly recommended that you export the ASOv2 secrets to a _different_ secret name than the
ASOv1 secret. ASOv2 will not allow you to overwrite an existing secret with `spec.operatorSpec.secrets`.
You'll get an error that looks like
"cannot overwrite Secret ns1/storageaccount-cutoverteststorage1 which is not owned by StorageAccount.storage.azure.com
ns1/aso-migration-test-cutoverteststorage1".
Instead of overwriting the same secret, create a different secret with ASOv2 (recommend including a suffix to
identify it such as -asov2). Then, when you're ready, swap your deployment to use the new ASOv2 secrets.
{{% /alert %}}

### Configure `resources.yaml` to source secret values from Kubernetes

Expand Down Expand Up @@ -246,11 +268,13 @@ This can be done automatically by `asoctl` during resource import with the
`-annotation serviceoperator.azure.com/reconcile-policy=skip` argument or manually by modifying `resources.yaml`
afterward.

> ⚠️ **Important:** Be _very_ careful issuing deletions of ASOv2 resources once you've imported them if they do not have
> the serviceoperator.azure.com/reconcile-policy=skip annotation. Just like ASOv1,
> ASOv2 will delete the underlying Azure resource by default!
> It is _strongly_ recommended that you use asoctl's `-annotation serviceoperator.azure.com/reconcile-policy=skip` flag
> and only remove that annotation a few resources at a time to ensure things are working as you expect.
{{% alert title="Warning" color="warning" %}}
Be _very_ careful issuing deletions of ASOv2 resources once you've imported them if they do not have
the `serviceoperator.azure.com/reconcile-policy=skip` annotation. Just like ASOv1,
ASOv2 _**will**_ delete the underlying Azure resource by default!
It is _strongly_ recommended that you use asoctl's `-annotation serviceoperator.azure.com/reconcile-policy=skip` flag
and only remove that annotation a few resources at a time to ensure things are working as you expect.
{{% /alert %}}

### Apply `resources.yaml` to your cluster

Expand All @@ -276,8 +300,10 @@ to rely on the ASOv2 secrets.
Once you've migrated the ASOv1 resources to ASOv2 and been running successfully for a while, you can delete the ASOv1 resources
in Kubernetes with `kubectl delete`.

> ⚠️ **Important:** Make sure that each ASOv1 resource you delete has the `skipreconcile=true` annotation before you delete it
> in Kubernetes or else the deletion will propagate to Azure and delete the underlying Azure resource as well, which you do not want.
{{% alert title="Warning" color="warning" %}}
Make sure that each ASOv1 resource you delete has the `skipreconcile=true` annotation before you delete it
in Kubernetes or else the deletion will propagate to Azure and delete the underlying Azure resource as well, which you do not want.
{{% /alert %}}

## Examples

Expand Down
Loading

0 comments on commit f56ca7b

Please sign in to comment.