Skip to content

Releases: crossplane/upjet

v1.3.0

25 Apr 13:26
577bfa7
Compare
Choose a tag to compare

Release Highlights:

1. New Feature: Introduce MR Metrics

We are excited to introduce a new set of managed resource (MR) metrics in this release. These metrics are pivotal for monitoring the state and performance of resources managed through Crossplane. This enhancement follows the implementation guidelines from the provider-kubernetes and incorporates changes from crossplane-runtime. The newly available metrics include:

  • crossplane_managed_resource_exists{"gvk"}
  • crossplane_managed_resource_ready{"gvk"}
  • crossplane_managed_resource_synced{"gvk"}
  • crossplane_managed_resource_first_time_to_reconcile_seconds{"gvk"}
  • crossplane_managed_resource_first_time_to_readiness_seconds{"gvk"}
  • crossplane_managed_resource_deletion_seconds{"gvk"}
  • crossplane_managed_resource_drift_seconds{"gvk"}

For a detailed overview of the implementation, please refer to this PR in crossplane-runtime.

2. Bug Fix: Immediate Sync Status Update on Async Failures

This update addresses a critical bug related to asynchronous operation failures (Issue #1164). With this fix, the "Synced" status will immediately reflect as "False" upon any failure in asynchronous operations, ensuring compliance with the XRM contract and improving reliability in status reporting. Previously, the error handling only updated the LastAsyncOperation status without properly setting the Synced condition to False. For more information, please see the detailed PR description.

What's Changed

New Contributors

Full Changelog: v1.2.4...v1.3.0

v1.1.0

01 Feb 19:07
a8e5f39
Compare
Choose a tag to compare

The v1.1.0 introduces several new features and enhancements

Multi-version Custom Resource Definitions (CRDs) generation and CRD conversion webhooks

The generation pipeline and the resource configuration framework now allow the generation of CRDs with multiple versions and conversion webhooks by conversion.Hub and conversion.Convertible implementations for the resource.Terraformed resources. The generated webhooks will invoke a chain of Conversions while converting from/to Hub versions to/from the spoke versions. There are currently two conversion types, conversion.PavedConversion and conversion.ManagedConversion.

We are also providing some high-level Conversions to implement some common API conversion tasks. conversion.NewFieldRenameConversion returns a new PavedConversion that implements a field renaming conversion from a specified source version to a specified target version of an API. conversion.NewCustomConverter can be used to initialize a new ManagedConversion that invokes a specified conversion function on a source and destination resource.Managed pair.

Terraform Plugin Framework resources support

This release introduces two new managed.Connecters and two new managed.ExternalClients (controller.terraformPluginFrameworkExternalClient and the controller.terraformPluginFrameworkAsyncExternalClient) which bring support for reconciling Terraform Plugin Framework resources in the new architecture without forking any Terraform CLI or Terraform provider processes. Consumers can specify resources in provider.TerraformPluginFrameworkIncludeList to generate them with Terraform Plugin Framework external clients. The nofork architecture now has the native implementations for both the Terraform plugin SDK v2 resources and the Terraform plugin framework resources. In this release, the NoFork terminology was replaced by TerraformPluginSDK.

What's Changed

  • Add server-side apply merge strategy markers by @negz in #301
  • Put default and configured timeouts to InstanceState for using the timeouts while Observe calls by @sergenyalcin in #309
  • Call the registered schema.CustomizeDiffFunc functions in the Terraform SDK-based external client by @ulucinar in #311
  • Make main-tf contents exported by @therealmitchconnors in #223
  • Ignore specific error that returned by expandWildcard function by @sergenyalcin in #312
  • Cache the Terraform instance state returned from schema.Resource.Apply even if the returned diagnostics contain errors by @ulucinar in #313
  • Add config.Resource. ServerSideApplyMergeStrategies to configure the SSA Merge Strategies by @ulucinar in #308
  • Pass full state to GetExternalNameFn function to access field other than ID by @sergenyalcin in #316
  • Set the RawPlan for a new terraform.InstanceState returned from an observation by @ulucinar in #317
  • Set diff state's Attributes to nil if the resource does not exist by @ulucinar in #318
  • Add reference fields to the InitProvider by @sergenyalcin in #315
  • Set RawConfig on the diff state by @ulucinar in #319
  • fix image links in configuring-a-resource by @smcavallo in #322
  • Add a nil check for the calculated instanceDiff while Observe by @sergenyalcin in #324
  • Bump hashicorp/terraform-plugin-sdk to v2.30.0 by @ulucinar in #320
  • fix broken link in docs by @mbbush in #328
  • Fix the return type so it uses Upjet by @jake-ciolek in #330
  • Generate a standalone "zz_generated.terraformed.go" file for each resource by @ulucinar in #326
  • Add a transformer to remove API group imports for cross-resource references by @ulucinar in #331
  • fix(environment): add missing environment specific patches TO/FROM by @haarchri in #332
  • Multiversion CRDs & Conversion Webhooks by @ulucinar in #321
  • Add configuration API parameter for overriding the field names by @sergenyalcin in #335
  • terraform plugin framework external client & connectors by @erhancagirici in #329
  • fix diff state not being set to freshly observed state for non-existing resources by @erhancagirici in #334
  • Set the ID string to empty string on Terraform state for calculating the diffs by @ulucinar in #337
  • Replace NoFork Terminology with TerraformPluginSDK by @ulucinar in #338
  • Set CRD storage version to the latest generated version by @ulucinar in #339

New Contributors

Full Changelog: v1.0.0...v1.1.0

v1.0.0

16 Nov 15:24
da16191
Compare
Choose a tag to compare

v1.0.0 contains major performance improvements for reconciling Terraform resources.

This release introduces two new managed.Connecters and two new managed.ExternalClients (controller.noForkExternal and the controller.noForkAsyncExternal) for reconciling Terraformed resources without forking any Terraform CLI or Terraform provider processes.

The new external client implementation consumes the Terraform provider's Go provider schema and invokes the CRUD functions registered in that schema via the terraform-plugin-sdk utility functions.
The new external clients employ an in-memory Terraform state cache as these clients no longer construct Terraform workspaces while reconciling their resources.

New external clients provide significant CPU and memory footprint reduction compared to traditional external clients. Below is an example result from Upbound Official provider-aws using the new external client for 1K and 10K MR. These results are shared not to provide a Sizing Guide but as a reference to understand the improvements in performance metrics. Also, the experiment sets of tests are shared:

  • Objective: 10K UserPolicyAttachment MRs

  • Parallelization: 500 reconcile workers

  • Poll period: 10min

  • Node: m5.2xlarge (8vCPU, 32GiB RAM)

    Average TTR Peak TTR Average Memory Peak Memory Average CPU Peak CPU
    4.40 secs 79 secs 775.41 MiB 1185.62 MiB 15.25% 20.90%
  • Objective: 1K UserPolicyAttachment MRs

  • Parallelization: 500 reconcile workers

  • Poll period: 10min

  • Node: m5.2xlarge (8vCPU, 32GiB RAM)

    Average TTR Peak TTR Average Memory Peak Memory Average CPU Peak CPU
    4.15 secs 44 secs 630.39 MiB 948.35 MiB 9.06% 12.69%

What's Changed

New Contributors

Full Changelog: v0.10.0...v1.0.0

v0.10.0

23 Aug 09:07
a5cc50f
Compare
Choose a tag to compare

The v0.10.0 release contains some important bug fixes regarding the granular management policies, a fix in the reconciliation logic of the upjet runtime and improvements in the core migration framework. You can bump your provider's upjet dependency by running go get github.com/upbound/upjet@v0.10.0 in your provider's root directory.

What's Changed

  • feat(docs): fix doc generation by @haarchri in #247
  • Set a default EventHandler if the provider's main module does not supply one by @ulucinar in #249
  • Change spec.owner in catalog-info.yaml by @Piotr1215 in #252
  • Add some common utility functions to migration framework by @sergenyalcin in #251
  • revert leaving optional for required init fields by @lsviben in #257
  • Fixes the requeueing bug which requeues a reconcile request in a wrong workqueue by @ulucinar in #259
  • remove overwrite from initProvider-forProvider merge by @lsviben in #260
  • Escape CEL Reserved Keywords when Generating Validation Rules by @dalton-hill-0 in #255
  • Add the inUse count to SharedProviderScheduler log statements by @toastwaffle in #261
  • Queue a reconcile request after marking the resource as ready by @ulucinar in #262

New Contributors

Full Changelog: v0.9.0...v0.10.0

v0.9.1

04 Aug 11:59
8d86393
Compare
Choose a tag to compare

This is a patch release that brings backwards-compatibility for upjet-based generated providers so that if they do not explicitly configure an event handler, a default one is configured. Please refer to the discussion in #250 for further details.

What's Changed

  • Set a default EventHandler if the provider's main module does not supply one by @ulucinar in #250

Full Changelog: v0.9.0...v0.9.1

v0.9.0

01 Aug 17:23
06bdecc
Compare
Choose a tag to compare

This release adds support for the spec.initProvider API and for the granular management policies alpha feature detailed here.

The generated example manifests from Terraform registry no longer contain the trailing YAML document separator (---).

There are numerous improvements and new features introduced to the migration framework to support some new migration cases, such the monolithic official providers to the provider families migration.

The external client for Terraformed resources now explicitly requeue, up to 20 retries, a reconciliation request if a shared provider has expired. And only after 20 retries it propagates the error down to the managed reconciler. The ttl-expired error message has also been improved to hint at the --provider-ttl command-line option.

Also status updates and updates to certain annotations (crossplane.io/external-create-failed & crossplane.io/external-create-pending) no longer queue reconciliation requests, which decreases the resource utilization of upjet-based providers. This is especially important when errors happen during the external connecter's Create call, or in general, when an MR is failing to sync successfully.

What's Changed

  • Add check for native provider upgrade by @sergenyalcin in #150
  • Add a test-case for config.GetExternalNameFromTemplated where the external-name template variable is not available by @ulucinar in #149
  • Rename migration.Converter.ComposedTemplates as Composition by @ulucinar in #148
  • Filter failed "terraform init" output for sensitive information by @ulucinar in #156
  • Bump Github workflow dependencies by @ulucinar in #157
  • docs: enphasize functions usage image on new resource doc by @Brocaneli in #165
  • Add OWNERS.md and CODEOWNERS by @turkenf in #169
  • Add upjet runtime Prometheus metrics by @ulucinar in #170
  • [scraper]: Add support for deducing resource name from file path by @ulucinar in #172
  • Add terraform.ProviderScheduler by @ulucinar in #178
  • Add an introductory Upjet-based provider monitoring guide by @ulucinar in #181
  • Add a sizing guide for official providers by @sergenyalcin in #182
  • Add config.Resource.Path to support configuring plural names of generated resources by @ulucinar in #184
  • Include full state under status.atProvider by @turkenh in #164
  • Generate CEL validation rules not to enforce required fields when ObserveOnly by @turkenh in #166
  • Observe with Terraform Import when Observe Only by @turkenh in #168
  • Do not add tags to spec when ObserveOnly by @turkenh in #186
  • Guide for adding Observe Only Support by @turkenh in #187
  • Add an example ControllerConfig to sizing-guide by @turkenf in #191
  • Add ToLower & ToUpper template functions for config.TemplatedStringAsIdentifier by @ulucinar in #190
  • Consume ESS plugin changes by @ezgidemirel in #188
  • Generate per-service providers by @ulucinar in #194
  • Deprecate config.BasePackages.Controller in favor of ControllerMap by @ulucinar in #195
  • Add Patch migration step type by @ulucinar in #153
  • Add migration.PlanGeneratorOption to configure a PlanGenerator by @ulucinar in #173
  • Add Smaller Provider Migration guides by @ezgidemirel in #203
  • Fixes and improvements in smaller provider migration docs by @turkenh in #204
  • Move inline bash code to separate files to prevent copy/paste errors by @ezgidemirel in #205
  • Incorporate Feedback on the New Provider Generation Guide by @turkenf in #193
  • Remove the command related to the sample configuration directory by @turkenf in #208
  • Extend SSOP migration script and process with the local source by @ytsarev in #207
  • Merge two SP migration docs into a single file by @ezgidemirel in #206
  • Add an example Smaller Provider migration plan by @ezgidemirel in #210
  • Add support for converting Crossplane Configurations by @ulucinar in #209
  • Switch to discovery client for managed resources by @sergenyalcin in #200
  • Add support for category-based manifest pre-processors by @ulucinar in #212
  • Remove family provider migrator docs by @sergenyalcin in #216
  • Add support for categorical converters by @ulucinar in #217
  • Adding backstage catalog file by @Piotr1215 in #228
  • Fast Follow: Converting upjet to component to include CI visibility by @Piotr1215 in #229
  • Add design-doc-provider-identity-based-auth.md to docs by @dverveiko in #227
  • Bump Go version to v1.20 in go.mod by @ulucinar in #236
  • Support granular managment policies by @lsviben in #224
  • Requeue the reconcile request on async update error by @ulucinar in #231
  • Remove the trailing document separator (---) from examples generated by @turkenf in #221
  • Support initProvider by @lsviben in #237
  • Fix config.Provider.RootGroup field docs by @ulucinar in #242
  • Fix the controller.go.tmpl for MRs reconciled in synchronous mode by @ulucinar in #243
  • feat(doc-generator): fix generation examples for provider-vault by @haarchri in #238
  • Revert "feat(doc-generator): fix generation examples for provider-vault" by @ulucinar in #244
  • Explicitly Queue a Reconcile Request if a Shared Provider has Expired by @ulucinar in #241

New Contributors

Full Changelog: v0.8.0...v0.9.0

v0.8.0

13 Jan 11:45
d381e87
Compare
Choose a tag to compare

This release contains the initial implementation of the base migration tooling for upjet-based providers and improvements to the metadata scraper. It also contains support for map types as sensitive inputs.

What's Changed

New Contributors

Full Changelog: v0.7.0...v0.8.0

v0.7.0

17 Oct 07:19
d102f52
Compare
Choose a tag to compare

This is the first public release of Upjet!

What's Changed

  • types: do not require go binary to be present during configuration by @muvaf in #63
  • crd: remove field transformation struct and its usages by @muvaf in #46
  • Filter sensitive fields in debug level logs by @sergenyalcin in #62
  • Add field&resource descriptions to CRD manifests from scraped metadata by @sergenyalcin in #64
  • Fix field comments by @sergenyalcin in #72
  • types: remove terraform-plugin-sdk v1 support since it reached EOL by @muvaf in #73
  • Fix external-name annotation configuration for examples by @ulucinar in #76
  • types.references: ref and selector fields should tell users which type they use by @muvaf in #85
  • terraform: do not print tf file name in the errors by @muvaf in #83
  • Add marketplace annotation to generated example manifests by @ezgidemirel in #75
  • Do not print skip-list for excluded resources by @ulucinar in #86
  • Add comment-marker for generated reference-type comment lines by @ulucinar in #87
  • Do not override annotations of generated example manifests by @ulucinar in #88
  • terraform: accept extra client metadata for ID calculations by @muvaf in #79
  • Manipulator function for every schema field by @sergenyalcin in #89
  • Make UseAsync=true by default by @ezgidemirel in #90
  • Filter terraform keyword in documentation by @sergenyalcin in #93
  • Generate hierarchical names under argumentDocs in scraped provider metadata by @ulucinar in #92
  • Support parent extraction from header blocks by @ulucinar in #94
  • terraform: make sure that the tfstate is re-produced in case we end up with empty tfstate by @muvaf in #97
  • terraform: do not fill the empty tfstate if the resource is being deleted by @muvaf in #101
  • Bump Go version to 1.19 by @ulucinar in #102
  • add legal notice for MPL usages by @muvaf in #103
  • add Apache 2.0 license by @muvaf in #104
  • Move OP docs by @ezgidemirel in #109
  • docs: add contribution guide and edit outdated parts of README by @muvaf in #111

New Contributors

Full Changelog: v0.6.0...v0.7.0

v0.6.0

05 Aug 10:00
d108c39
Compare
Choose a tag to compare

v0.6.0 contains example generation pipelines that can scrape Terraform registry documentation pages and generate managed resource example manifests together with their dependencies. These pipelines also incorporate automatic reference injection. The scraped metadata currently looks like the following for a resource:

    azurerm_disk_encryption_set:
        subCategory: Compute
        description: Manages a Disk Encryption Set.
        name: azurerm_disk_encryption_set
        title: azurerm_disk_encryption_set
        examples:
            - name: example
              manifest: |-
                {
                  "identity": [
                    {
                      "type": "SystemAssigned"
                    }
                  ],
                  "key_vault_key_id": "${azurerm_key_vault_key.example.id}",
                  "location": "${azurerm_resource_group.example.location}",
                  "name": "des",
                  "resource_group_name": "${azurerm_resource_group.example.name}"
                }
              references:
                key_vault_key_id: azurerm_key_vault_key.example.id
                location: azurerm_resource_group.example.location
                resource_group_name: azurerm_resource_group.example.name
              dependencies:
                azurerm_key_vault.example: |-
                    {
                      "enabled_for_disk_encryption": true,
                      "location": "${azurerm_resource_group.example.location}",
                      "name": "des-example-keyvault",
                      "purge_protection_enabled": true,
                      "resource_group_name": "${azurerm_resource_group.example.name}",
                      "sku_name": "premium",
                      "tenant_id": "${data.azurerm_client_config.current.tenant_id}"
                    }
                azurerm_key_vault_access_policy.example-disk: |-
                    {
                      "key_permissions": [
                        "Get",
                        "WrapKey",
                        "UnwrapKey"
                      ],
                      "key_vault_id": "${azurerm_key_vault.example.id}",
                      "object_id": "${azurerm_disk_encryption_set.example.identity.0.principal_id}",
                      "tenant_id": "${azurerm_disk_encryption_set.example.identity.0.tenant_id}"
                    }
                azurerm_key_vault_access_policy.example-user: |-
                    {
                      "key_permissions": [
                        "get",
                        "create",
                        "delete"
                      ],
                      "key_vault_id": "${azurerm_key_vault.example.id}",
                      "object_id": "${data.azurerm_client_config.current.object_id}",
                      "tenant_id": "${data.azurerm_client_config.current.tenant_id}"
                    }
                azurerm_key_vault_key.example: |-
                    {
                      "depends_on": [
                        "${azurerm_key_vault_access_policy.example-user}"
                      ],
                      "key_opts": [
                        "decrypt",
                        "encrypt",
                        "sign",
                        "unwrapKey",
                        "verify",
                        "wrapKey"
                      ],
                      "key_size": 2048,
                      "key_type": "RSA",
                      "key_vault_id": "${azurerm_key_vault.example.id}",
                      "name": "des-example-key"
                    }
                azurerm_resource_group.example: |-
                    {
                      "location": "West Europe",
                      "name": "example-resources"
                    }
        argumentDocs:
            auto_key_rotation_enabled: '- (Optional) Boolean flag to specify whether Azure Disk Encryption Set automatically rotates encryption Key to latest version. Defaults to false.'
            create: '- (Defaults to 60 minutes) Used when creating the Disk Encryption Set.'
            delete: '- (Defaults to 60 minutes) Used when deleting the Disk Encryption Set.'
            encryption_type: '- (Optional) The type of key used to encrypt the data of the disk. Possible values are EncryptionAtRestWithCustomerKey, EncryptionAtRestWithPlatformAndCustomerKeys and ConfidentialVmEncryptedWithCustomerKey. Defaults to EncryptionAtRestWithCustomerKey.'
            id: '- The ID of the Disk Encryption Set.'
            identity: '- (Required) An identity block as defined below.'
            key_vault_key_id: '- (Required) Specifies the URL to a Key Vault Key (either from a Key Vault Key, or the Key URL for the Key Vault Secret).'
            location: '- (Required) Specifies the Azure Region where the Disk Encryption Set exists. Changing this forces a new resource to be created.'
            name: '- (Required) The name of the Disk Encryption Set. Changing this forces a new resource to be created.'
            principal_id: '- The (Client) ID of the Service Principal.'
            read: '- (Defaults to 5 minutes) Used when retrieving the Disk Encryption Set.'
            resource_group_name: '- (Required) Specifies the name of the Resource Group where the Disk Encryption Set should exist. Changing this forces a new resource to be created.'
            tags: '- (Optional) A mapping of tags to assign to the Disk Encryption Set.'
            tenant_id: '- The ID of the Tenant the Service Principal is assigned in.'
            type: '- (Required) The type of Managed Service Identity that is configured on this Disk Encryption Set. The only possible value is SystemAssigned.'
            update: '- (Defaults to 60 minutes) Used when updating the Disk Encryption Set.'
        importStatements:
            - terraform import azurerm_disk_encryption_set.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group1/providers/Microsoft.Compute/diskEncryptionSets/encryptionSet1

The generated example manifests for this resource then looks like the following:

apiVersion: compute.azure.upbound.io/v1beta1
kind: DiskEncryptionSet
metadata:
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    identity:
    - type: SystemAssigned
    keyVaultKeyIdSelector:
      matchLabels:
        testing.upbound.io/example-name: example
    location: West Europe
    resourceGroupNameSelector:
      matchLabels:
        testing.upbound.io/example-name: example

---

apiVersion: keyvault.azure.upbound.io/v1beta1
kind: Vault
metadata:
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    enabledForDiskEncryption: true
    location: West Europe
    purgeProtectionEnabled: true
    resourceGroupNameSelector:
      matchLabels:
        testing.upbound.io/example-name: example
    skuName: premium
    tenantId: ${data.azurerm_client_config.current.tenant_id}

---

apiVersion: keyvault.azure.upbound.io/v1beta1
kind: AccessPolicy
metadata:
  labels:
    testing.upbound.io/example-name: example-disk
  name: example-disk
spec:
  forProvider:
    keyPermissions:
    - Get
    - WrapKey
    - UnwrapKey
    keyVaultIdSelector:
      matchLabels:
        testing.upbound.io/example-name: example
    objectId: ${azurerm_disk_encryption_set.example.identity.0.principal_id}
    tenantId: ${azurerm_disk_encryption_set.example.identity.0.tenant_id}

---

apiVersion: keyvault.azure.upbound.io/v1beta1
kind: AccessPolicy
metadata:
  labels:
    testing.upbound.io/example-name: example-user
  name: example-user
spec:
  forProvider:
    keyPermissions:
    - get
    - create
    - delete
    keyVaultIdSelector:
      matchLabels:
        testing.upbound.io/example-name: example
    objectId: ${data.azurerm_client_config.current.object_id}
    tenantId: ${data.azurerm_client_config.current.tenant_id}

---

apiVersion: keyvault.azure.upbound.io/v1beta1
kind: Key
metadata:
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    keyOpts:
    - decrypt
    - encrypt
    - sign
    - unwrapKey
    - verify
    - wrapKey
    keySize: 2048
    keyType: RSA
    keyVaultIdSelector:
      matchLabels:
        testing.upbound.io/example-name: example

---

apiVersion: azure.upbound.io/v1beta1
kind: ResourceGroup
metadata:
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    location: West Europe

---

v0.5.0

01 Jun 11:32
39f5696
Compare
Choose a tag to compare

The first release of Upjet that can be used as replacement to Terrajet!