[k8s-configuration] Update configuration CLI to v1.7.0 to support Azu…

…re Blob Storage (Azure#5472) * Create pull.yml * Update pull.yml * Update azure-pipelines.yml * Initial commit of k8s-extension * Update CODEOWNERS * Update azure-pipelines.yml * Create pull.yml * Update pull.yml * Update pull.yml * Update pipelines file * Update k8s-configuration name * Update test script params * Update pipeline file * Remove codeowners * Update pipelines file * Update CODEOWNERS * Update private preview pipelines * Remove open service mesh from public release * Update pipeline files * Update custom pipelines files * Add publish step to k8s-configuration * Update pipeline to publish extension * Update public extension pipeline * Change condition variable * Update pipeline naming * Add version to public preview/private preview * Update pipelines * Add different testing based on private branch * Add annotations to extension model * Update k8s-custom-pipelines.yml * Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13) * Update sdks with updated swagger spec * Update version and history rst * Reorder release history timeline * Fix ExtensionInstanceForCreate for import * remove py2 bdist support * Add custom table formatting * Remove unnecessary files * Fix style issues * Fix branch based on comments * Update identity piece manually * Don't handle defaults at the CLI level * Remove defaults from CLI client * Check null target namespace with namespace scope * Update style * Add cassandra operator and location to model * Stage Public Version of k8s-extension 0.2.0 for official release (#15) * Create pull.yml * Update pull.yml * Update azure-pipelines.yml * Initial commit of k8s-extension * Update pipelines file * Update CODEOWNERS * Update private preview pipelines * Remove open service mesh from public release * Update pipeline files * Update public extension pipeline * Change condition variable * Add version to public preview/private preview * Update pipelines * Add different testing based on private branch * Add annotations to extension model * Update k8s-custom-pipelines.yml * Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13) * Update sdks with updated swagger spec * Update version and history rst * Reorder release history timeline * Fix ExtensionInstanceForCreate for import * remove py2 bdist support * Add custom table formatting * Remove unnecessary files * Fix style issues * Fix branch based on comments * Update identity piece manually * Don't handle defaults at the CLI level * Remove defaults from CLI client * Check null target namespace with namespace scope * Update style * Add cassandra operator and location to model Co-authored-by: action@github.com <Action - Fork Sync> * Remove custom pipelines file * Update extension description, remove private const * Update pipeline file * Disable check ref docs * Disable refs docs * Update to include better create warning logs and remove update context (#20) * Update to include better create warning logs and remove update context * Remove help text for update * Fix spelling error * Update message * Fix k8s-extension conflict with private version * Fix style errors * Fix filename * add customization for microsoft.azureml.kubernetes (#23) * add customization for microsoft.azureml.kubernetes * Update release history Co-authored-by: Yue Yu <yuyu3@microsoft.com> Co-authored-by: jonathan-innis <jonathan.innis.ji@gmail.com> * Add E2E Testing from Separate branch into internal code (#26) * Add internal e2e testing * Change to testing folder * Inference CLI validation for Scoring FE (#24) * cli validation starter * added the call to the fe validation function * nodeport validation not required * test fix Co-authored-by: Jonathan Innis <jonathan.innis.ji@gmail.com> * legal warning added (#27) * Remove deprecated method logger.warn * Update k8s-custom-pipelines.yml for Azure Pipelines * Update k8s-custom-pipelines.yml for Azure Pipelines * Add Azure Defender to E2E testing (#28) * Add azure defender testing to e2e * Remove the debug flag * Add configuration testing * Fix pipeline failures * Make test script more intuitive * Remove parameter from testing * Add some debug * Fix wrong location for k8s config whl * Fix pip install upgrade issue * Fix pip install upgrade issue * Add Check for Provider Registration and Refactor (#19) * Add check for provider registration and refactor * Fix bug in checking registration * Add license header to utils * Update private key check and error messaging * Update based on refactoring * Fix failing tests * Add provider registration check * Create a test for uppercase url, address comments * Add blank line to fix style check * Testing increase to ubuntu-latest * Update k8s-configuration Models to Track2 (#63) * Update models to track2 * Increase k8s-configuration version number * Update kind version * Change error to warning because of DSA failure * Upgrade helm operator chart version (#75) * Pin helm version * Bump version * Migrate pipeline (#90) * Disable updates on configuration tests (#89) * Release k8s-configuration v1.2.0 for Flux v2 Public Preview (#86) * Scaffold out the k8s-config package * Base implementation of CLI commands * Add create scenario and cleanup in consts * Add help text to commands, params * Add other clients to client factory * Automatically installing the flux extension * Move flux and extension into modules * Updated the versioned sdks * Push working command for testing * Update to multi api versioned sdk * Support other extension methods * Fix nullity check * Add source control provider * Add scc commands * Add defer logic for create with cache * Use default extension with identity * Fix identity creation * Add kustomization caching * Add formatters * Add scc provider * Add help text for k8s-config fluxv1 * Add help text for extension * Allow force delete of extension and fluxconfiguration * Add location to the extension model * Update with latest from k8s-extension * Add k8s-config testing * Add license header * Fix all style issues * Update codeowners file * Validate data before checking cluster compliance * No kustomizations warning * Fix identity issue in 2020-07-01 * Fix k8s regex * Fix configuration name regex validation * Fix name length validation * Adding some validation warnings * Add protected settings to request * Exclude private test path * Add suspend functionality * Add correct values to build Kustomization * Add no_wait * Fix style issues * Use base64 encoded httpsUser * Fix formatting error and base64 encoding error * Fix style issues * Fix force * Updated help text * Style fixes * Increase namespace maximum len * Add managed cluster support to k8s-config * Custom confirmation when prune is enabled * Add flux commands to existing k8s-configuration * Remove extension provider from CLI * Fix style issues * Override extension variables * Strip newlines from known_hosts file * Update help text and validators * Strip newlines from known hosts * Add provisioning state check for flux extension * Pin helm version * Remove validation from create command * Add patch support with new SDK * Add implementation for CRUD of source and kustomization * Fix errors on patch * Fix some bugs in patching properties * Add fixes for patch in k8s-configuraiton * Change duration formatting in table output * Add validation and conversion for durations * Bump verison and fix typo * Fix bug with dependencies * Fix linter and style issues * Fix delete prune check * Add flux testing * Create separate jobs for scenarios * Update error text * Fix filepath suggestion from CLI team * Fix unneeded file edit * Add a Deployed Object List to the Flux CLI (#91) * Enable a deployed object list on the CLI * Show detail when extension install fails * Bump version * Update deployed object format (#93) * Fix help text for consistency * Enable Bucket Support in the CLI (#92) * Update vendored_sdks * Enable source kind generation factories and use kwargs to pass to provider * Add better validation logic to source generator * Move away from classes in the provider directory * Fix style issues using black auto-formatter * Fix linter failures * Update identity with api version and rp in same function * Bucket Testing for E2E Testing (#96) * Bucket testing * Add switching kind test to update CLI * Bump version * Fix help text and parameter naming for bucket (#100) * Fix default help text for parameters (#101) * Edit history with breaking change * Prepare GA FluxConfiguration 2022-03-01 (#103) * Add vendored sdks for new api-version * Update the new depends on definition * Update table formatting * BucketDefinition to BucketPatchDefinition * Remove http url warning * Update vendored_sdks * Update action file to add depenencies * Remove preview from command groups * Update changelog * Make dependencies none when not specified * Use KustomizationPatch instead of Kustomization for internal rep (#117) * Only test k8s-configuration in azdev test (#140) * modify codeowners for k8s-configuration (#157) Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> * [k8s-configuration] add support for provisionedClusters (#146) * [k8s-configuration] add support for provisionedClusters * resolve cli errors * remove sourceControlConfiguration support for provisionedClusters * code cleanup * updates hybridcontainerservice api version * change description for cluster_type Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> * bump k8s-configuration version to 1.6.0 * CI fix[k8s-configuration]: deprecate python version 3.6 use in testing (#174) * CI fix[k8s-configuration]: deprecate python version 3.6 use in testing * correct the python version for package install Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> * Support Azure blob as source (#156) * [k8s-configuration] Update configuration CLI to v1.7.0 to support Azure Blob Storage * change help message Co-authored-by: Jonathan Innis <jonathan.innis.ji@gmail.com> Co-authored-by: action@github.com <Action - Fork Sync> Co-authored-by: yuyue9284 <15863499+yuyue9284@users.noreply.github.com> Co-authored-by: Yue Yu <yuyu3@microsoft.com> Co-authored-by: Lia Kazakova <58274127+liakaz@users.noreply.github.com> Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> Co-authored-by: Summer Hasama <69527370+summerhasama@users.noreply.github.com>
AzureArcForKubernetes · Oct 20, 2022 · d78a840 · d78a840
1 parent 7679096
commit d78a840
Show file tree

Hide file tree

Showing 40 changed files with 9,459 additions and 19 deletions.
diff --git a/src/k8s-configuration/HISTORY.rst b/src/k8s-configuration/HISTORY.rst
@@ -3,6 +3,10 @@
 Release History
 ===============
 
+1.7.0
+++++++++++++++++++
+* Add support for Azure Blob Storage
+
 1.6.0
 ++++++++++++++++++
 * Add support for provisionedClusters

diff --git a/src/k8s-configuration/azext_k8s_configuration/_help.py b/src/k8s-configuration/azext_k8s_configuration/_help.py
@@ -94,6 +94,14 @@
           --kind bucket --url https://bucket-provider.minio.io \\
           --bucket-name my-bucket --kustomization name=my-kustomization \\
           --bucket-access-key my-access-key --bucket-secret-key my-secret-key
+      - name: Create a Kubernetes v2 Flux Configuration with Azure Blob Source Kind
+        text: |-
+          az k8s-configuration flux create --resource-group my-resource-group \\
+          --cluster-name mycluster --cluster-type connectedClusters \\
+          --name myconfig --scope cluster --namespace my-namespace \\
+          --kind azblob --url https://mystorageaccount.blob.core.windows.net \\
+          --container-name my-container --kustomization name=my-kustomization \\
+          --account-key my-account-key
 """
 
 helps[
@@ -108,11 +116,16 @@
           --cluster-name mycluster --cluster-type connectedClusters --name myconfig \\
           --url https://github.com/Azure/arc-k8s-demo --branch main \\
           --kustomization name=my-kustomization path=./my/new-path
-      - name: Update a Flux v2 Kubernetse configuration with Bucket Source Kind to connect insecurely
+      - name: Update a Flux v2 Kubernetes configuration with Bucket Source Kind to connect insecurely
         text: |-
           az k8s-configuration flux update --resource-group my-resource-group \\
           --cluster-name mycluster --cluster-type connectedClusters --name myconfig \\
           --bucket-insecure
+      - name: Update a Flux v2 Kubernetes configuration with Azure Blob Source Kind with another container name
+        text: |-
+          az k8s-configuration flux update --resource-group my-resource-group \\
+          --cluster-name mycluster --cluster-type connectedClusters --name myconfig \\
+          --container-name other-container
 """
 
 helps[

diff --git a/src/k8s-configuration/azext_k8s_configuration/_params.py b/src/k8s-configuration/azext_k8s_configuration/_params.py
@@ -67,7 +67,7 @@ def load_arguments(self, _):
         )
         c.argument(
             "kind",
-            arg_type=get_enum_type([consts.GIT, consts.BUCKET]),
+            arg_type=get_enum_type([consts.GIT, consts.BUCKET, consts.AZBLOB]),
             help="Source kind to reconcile",
         )
         c.argument(
@@ -178,6 +178,62 @@ def load_arguments(self, _):
             help="Define kustomizations to sync sources with parameters ['name', 'path', 'depends_on', 'timeout', 'sync_interval', 'retry_interval', 'prune', 'force']",
             nargs="+",
         )
+        c.argument(
+            "container_name",
+            help="Name of the Azure Blob Storage container to sync",
+        )
+        c.argument(
+            "sp_client_id",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-client-id", "--service-principal-client-id"],
+            help="The client ID for authenticating a service principal with Azure Blob, required for this authentication method",
+        )
+        c.argument(
+            "sp_tenant_id",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-tenant-id", "--service-principal-tenant-id"],
+            help="The tenant ID for authenticating a service principal with Azure Blob, required for this authentication method",
+        )
+        c.argument(
+            "sp_client_secret",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-client-secret", "--service-principal-client-secret"],
+            help="The client secret for authenticating a service principal with Azure Blob",
+        )
+        c.argument(
+            "sp_client_cert",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-client-cert", "--service-principal-client-certificate"],
+            help="The Base64 encoded client certificate for authenticating a service principal with Azure Blob",
+        )
+        c.argument(
+            "sp_client_cert_password",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-cert-password", "--service-principal-client-certificate-password"],
+            help="The password for the client certificate used to authenticate a service principal with Azure Blob",
+        )
+        c.argument(
+            "sp_client_cert_send_chain",
+            arg_group="Azure Blob Auth",
+            options_list=["--sp-cert-send-chain", "--service-principal-client-certificate-send-chain"],
+            help="Specify whether to include x5c header in client claims when acquiring a token to enable subject name / issuer based authentication for the client certificate",
+        )
+        c.argument(
+            "account_key",
+            arg_group="Azure Blob Auth",
+            help="The Azure Blob Shared Key for authentication ",
+        )
+        c.argument(
+            "sas_token",
+            arg_group="Azure Blob Auth",
+            help="The Azure Blob SAS Token for authentication ",
+        )
+        c.argument(
+            "mi_client_id",
+            arg_group="Azure Blob Auth",
+            options_list=["--mi-client-id", "--managed-identity-client-id"],
+            help="The client ID of the managed identity for authentication with Azure Blob",
+        )
 
     with self.argument_context("k8s-configuration flux update") as c:
         c.argument(

diff --git a/src/k8s-configuration/azext_k8s_configuration/consts.py b/src/k8s-configuration/azext_k8s_configuration/consts.py
@@ -8,8 +8,8 @@
 # API VERSIONS -----------------------------------------
 
 SOURCE_CONTROL_API_VERSION = "2022-03-01"
-FLUXCONFIG_API_VERSION = "2022-03-01"
-EXTENSION_API_VERSION = "2022-03-01"
+FLUXCONFIG_API_VERSION = "2022-07-01"
+EXTENSION_API_VERSION = "2022-07-01"
 
 # ERROR/HELP TEXT DEFINITIONS -----------------------------------------
 
@@ -41,7 +41,30 @@
 REQUIRED_BUCKET_VALUES_MISSING_HELP = (
     "Provide either both of '--secret-key' and '--access-key' or '--local-auth-ref'"
 )
-
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_VALUES_MISSING_ERROR = (
+    "Error! Service principal is invalid because it is missing value(s)"
+)
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_VALUES_MISSING_HELP = (
+    "Provide '--sp-client-id', '--sp-tenant-id', and either '--sp-client-secret' or '--sp-client-cert'"
+)
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_AUTH_ERROR = (
+    "Error! Too many authentication methods provided for service principal"
+)
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_AUTH_HELP = (
+    "Provide either '--sp-client-secret' or '--sp-client-cert'"
+)
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_CERT_VALUES_MISSING_ERROR = (
+    "Error! Service principal certificate password is invalid"
+)
+REQUIRED_AZURE_BLOB_SERVICE_PRINCIPAL_CERT_VALUES_MISSING_HELP = (
+    "Provide '--sp-client-id', '--sp-tenant-id', and '--sp-client-cert' with your '--sp-cert-password"
+)
+REQUIRED_AZURE_BLOB_AUTH_ERROR = (
+    "Error! Too many authentication methods provided for Azure Blob"
+)
+REQUIRED_AZURE_BLOB_AUTH_HELP = (
+    "Specify one of the available authentication methods from the list: '--local-auth-ref', '--account-key', '--sas-token', '--mi-client-id', or service principal with '--sp-client-id', '--sp-tenant-id', and either '--sp-client-secret' or '--sp-client-cert'"
+)
 EXTRA_VALUES_PROVIDED_ERROR = (
     "Error! Invalid properties [{}] were specified for kind '{}'"
 )
@@ -213,6 +236,24 @@
     "local_auth_ref",
 }
 
+AZUREBLOB_REQUIRED_PARAMS = {"url", "container_name"}
+AZUREBLOB_VALID_PARAMS = {
+    "url",
+    "container_name",
+    "sync_interval",
+    "timeout",
+    "account_key",
+    "local_auth_ref",
+    "sp_tenant_id",
+    "sp_client_id",
+    "sp_client_cert",
+    "sp_client_cert_password",
+    "sp_client_secret",
+    "sp_client_cert_send_chain",
+    "sas_token",
+    "mi_client_id",
+}
+
 DEPENDENCY_KEYS = ["dependencies", "depends_on", "dependsOn", "depends"]
 SYNC_INTERVAL_KEYS = ["interval", "sync_interval", "syncInterval"]
 RETRY_INTERVAL_KEYS = ["retryInterval", "retry_interval"]
@@ -222,12 +263,16 @@
 VALID_DURATION_REGEX = r"((?P<hours>\d+?)h)?((?P<minutes>\d+?)m)?((?P<seconds>\d+?)s)?"
 VALID_GIT_URL_REGEX = r"^(((http|https|ssh)://)|(git@))"
 VALID_BUCKET_URL_REGEX = r"^(((http|https)://))"
+VALID_AZUREBLOB_URL_REGEX = r"^(((http|https)://))"
 
 VALID_KUBERNETES_DNS_SUBDOMAIN_NAME_REGEX = r"^[a-z0-9]([\.\-a-z0-9]*[a-z0-9])?$"
 VALID_KUBERNETES_DNS_NAME_REGEX = r"^[a-z0-9]([\-a-z0-9]*[a-z0-9])?$"
 
 GIT = "git"
 BUCKET = "bucket"
+BUCKET_CAPS = "Bucket"
+AZBLOB = "azblob"
+AZURE_BLOB = "AzureBlob"
 GIT_REPOSITORY = "GitRepository"
 
 CONNECTED_CLUSTER_TYPE = "connectedclusters"