Hetzner: drop deprecated `datacenter` field in favor of `location`

[buraksekili]

Summary

Hetzner Cloud deprecated the datacenter property in the request body and response of Servers and Primary IPs on 16 December 2025, in favor of a top-level location property.
The removal was scheduled for "after 1 July 2026" and, as of the July 2026 changelog, has already been carried out.
The GET /datacenters endpoints are deprecated separately (removal after 1 October 2026).

The machine-controller Hetzner provider still exposes and uses a datacenter field.
Any KKP / KubeOne Hetzner installation whose MachineDeployment sets cloudProviderSpec.datacenter is affected: server creation sends the removed datacenter field in POST /servers, and validation calls the deprecated GET /datacenters/{name} endpoint.

The location field is already fully supported.
The work is to deprecate and remove the datacenter field and migrate users onto location.

References:

• Hetzner Cloud API Changelog: https://docs.hetzner.cloud/changelog
• "We removed the datacenter property in the request body and response of Servers and Primary IPs as announced on 16 December 2025."

Impact

Affected: Hetzner MachineDeployments where cloudProviderSpec.datacenter is set (and location is empty).

Not affected:

• installations using cloudProviderSpec.location (already fully supported)
• installations that set neither field (Hetzner picks a default location)

Failure mode for an affected user, after Hetzner removes the datacenter concept:

1. Create sends datacenter in the POST /servers body.
   Hetzner ignores or rejects it; the server is either not created or lands in an unintended default location.
   Create also calls client.Datacenter.Get against the deprecated GET /datacenters/{name} endpoint and checks dc == nil, so it fails explicitly when the datacenter cannot be resolved.
2. Validate calls client.Datacenter.Get against GET /datacenters/{name}.
   Note the SDK returns (nil, nil, nil) when a datacenter is not found (it swallows NotFound and returns nil on an empty list), and Validate only checks err != nil, not the nil pointer.
   So a removed or invalid datacenter may pass Validate silently and only surface as a failure at Create time.
   Either way the machine does not provision correctly.

Current state in the code

Config schema

sdk/cloudprovider/hetzner/types.go:24-36 - RawConfig exposes both fields:

type RawConfig struct {
    Token                providerconfig.ConfigVarString   `json:"token,omitempty"`
    ServerType           providerconfig.ConfigVarString   `json:"serverType"`
    Datacenter           providerconfig.ConfigVarString   `json:"datacenter"`
    Image                providerconfig.ConfigVarString   `json:"image"`
    Location             providerconfig.ConfigVarString   `json:"location"`
    PlacementGroupPrefix providerconfig.ConfigVarString   `json:"placementGroupPrefix"`
    Networks             []providerconfig.ConfigVarString `json:"networks"`
    Firewalls            []providerconfig.ConfigVarString `json:"firewalls"`
    Labels               map[string]string                `json:"labels,omitempty"`
    AssignPublicIPv4     providerconfig.ConfigVarBool     `json:"assignPublicIPv4,omitempty"`
    AssignPublicIPv6     providerconfig.ConfigVarBool     `json:"assignPublicIPv6,omitempty"`
}

• Datacenter at types.go:27 (json:"datacenter", no omitempty)
• Location at types.go:29 (json:"location", no omitempty)

pkg/cloudprovider/provider/hetzner/provider.go:58-70 - resolved Config struct has both Datacenter (line 61) and Location (line 63) as plain strings.

getConfig resolves both independently with no fallback:

• provider.go:115-118 resolves c.Datacenter
• provider.go:125-128 resolves c.Location

Runtime use of the deprecated Hetzner datacenter API

Validate (provider.go:194-264):

• mutual-exclusion check at provider.go:206-208:

  if c.Location != "" && c.Datacenter != "" {
      return fmt.Errorf("location and datacenter must not be set at the same time")
  }

• datacenter validation at provider.go:216-220:

  if c.Datacenter != "" {
      if _, _, err = client.Datacenter.Get(ctx, c.Datacenter); err != nil {
          return fmt.Errorf("failed to get datacenter: %w", err)
      }
  }

Create (provider.go:266-409):

• datacenter block at provider.go:304-313:

  if c.Datacenter != "" {
      dc, _, err := client.Datacenter.Get(ctx, c.Datacenter)
      if err != nil {
          return nil, hzErrorToTerminalError(err, "failed to get datacenter")
      }
      if dc == nil {
          return nil, fmt.Errorf("datacenter %q does not exist", c.Datacenter)
      }
      serverCreateOpts.Datacenter = dc
  }

Metrics

MachineMetricsLabels (provider.go:529-540) emits both a dc label (from c.Datacenter, line 535) and a location label (from c.Location, line 536).
Both are always set, even when empty.

No migration exists

• AddDefaults (provider.go:461-463) is a no-op: return spec, nil.
• MigrateUID (provider.go:492-527) only migrates the machine-uid label.
• No code path converts a datacenter value into a location value.

How validation / migration is wired

Provider Validate and AddDefaults run inside the admission webhook, not in the CRD schema.

• Webhook entry: cmd/webhook/main.go:128-148, registers /machinedeployments and /machines (pkg/admission/admission.go:110-111).
  It is a mutating webhook producing JSON patches.
• pkg/admission/machines.go:120-201 (defaultAndValidateMachineSpec):
  • :190-194 calls prov.AddDefaults
  • :196-198 calls prov.Validate; a returned error rejects admission and surfaces the message to the user
• MachineDeployment-level provider migrations live in pkg/admission/machinedeployments_validation.go:117-146 (mutationsForMachineDeployment), which currently runs migrateToEquinixMetal and migrateVMwareCloudDirector.
• Migration helpers are in pkg/admission/util.go:
  • migrateToEquinixMetal (:29-52): renames provider packet to equinixmetal and apiKey to token.
  • migrateVMwareCloudDirector (:54-76): moves legacy Network into Networks[0], clears Network, dedupes, re-marshals.

The CRD does not constrain cloudProviderSpec.
The MachineDeployment CRD (examples/machine-controller.yaml:105-156) uses x-kubernetes-preserve-unknown-fields: true with no per-provider OpenAPI schema.
cloudProviderSpec is a runtime.RawExtension (sdk/providerconfig/types.go:161), strict-unmarshalled into the provider RawConfig at runtime.
So removing the datacenter field from the Go struct does not require any CRD regeneration.

Established deprecation pattern in this repo

Other providers deprecate fields by keeping the field in RawConfig with a // Deprecated: comment, rejecting the both-old-and-new case, and translating the old field into the new one with a logged warning:

• anexia: sdk/cloudprovider/anexia/types.go:82-88 (DiskSize, VlanID); sentinel errors at :46-49; migration in pkg/cloudprovider/provider/anexia/resolve_config.go:82-143.
• vsphere: sdk/cloudprovider/vsphere/types.go:27-28 (VMNetName); mutual exclusion at pkg/cloudprovider/provider/vsphere/provider.go:246-248.
• vmwareclouddirector: sdk/cloudprovider/vmwareclouddirector/types.go:49-50 (Network); migrated at the webhook level in pkg/admission/util.go:54-76 (migrateVMwareCloudDirector).

The AddDefaults active-defaulting pattern (parse RawConfig, mutate, re-marshal into spec.ProviderSpec.Value.Raw, return) is demonstrated by OpenStack (pkg/cloudprovider/provider/openstack/provider.go:400-496) and Azure (pkg/cloudprovider/provider/azure/provider.go:516-590).

SDK version

go.mod:32 pins github.com/hetznercloud/hcloud-go/v2 v2.13.1; go list resolves the same.
No vendor directory; consumed from the module cache.

In v2.13.1, hcloud.ServerCreateOpts has a Datacenter *Datacenter field.
Create() serializes it into the POST /servers body as "datacenter" (schema/server.go:106, json:"datacenter,omitempty"), via server.go:428-434.
The field is not marked // Deprecated: in v2.13.1.

Upstream has since caught up to the Hetzner API change:

• The latest hcloud-go release as of July 2026 is v2.44.0 (2026-06-18).
• ServerCreateOpts.Datacenter is marked // Deprecated: as of v2.33.0:

  // Deprecated: [ServerCreateOpts.Datacenter] is deprecated and will be removed after 1 July 2026.
  // Use [ServerCreateOpts.Location] instead.
  Datacenter *Datacenter

  The same deprecation applies to the response field Server.Datacenter.
• In v2.44.0 the entire Datacenters resource (DatacenterClient, Datacenter struct, GET /datacenters endpoints) is also deprecated, with removal after 1 October 2026.

Bumping the SDK to v2.33.0+ brings the deprecation marker and staticcheck warnings, which helps surface the issue, but it does not by itself fix a MachineDeployment that still has datacenter populated: the field still serializes into the request body until the provider stops setting it.
The user-facing fix remains at the provider level.

Proposed changes

1. Deprecate, then remove, the datacenter field

In sdk/cloudprovider/hetzner/types.go:

• Mark Datacenter // Deprecated: Hetzner removed the datacenter concept; use location instead.
  Keep the field temporarily so existing specs still parse.
• Remove the Datacenter field from Config in provider.go:58-70 once migration is in place.

In pkg/cloudprovider/provider/hetzner/provider.go:

• Remove the client.Datacenter.Get call in Validate (provider.go:216-220).
• Remove the client.Datacenter.Get block and serverCreateOpts.Datacenter assignment in Create (provider.go:304-313).
• Update MachineMetricsLabels (provider.go:535): drop the dc label, keep location.
• Adjust the mutual-exclusion check at provider.go:206-208 (it becomes unnecessary once datacenter is gone, or it should reject datacenter with a migration message during the deprecation window).

2. Migrate datacenter to location

A datacenter value like nbg1-dc3 is not a valid location (nbg1).
Migration cannot blindly copy the value; it must map the datacenter to its location.
Options:

• Add a migrateHetznerDatacenter helper in pkg/admission/util.go, following migrateVMwareCloudDirector, invoked from mutationsForMachineDeployment (pkg/admission/machinedeployments_validation.go:117-146).
  The helper reads the typed RawConfig, and if datacenter is set and location is empty, derives the location from the datacenter name (the location is the part before the first -, e.g. nbg1-dc3 -> nbg1) and clears datacenter.
  Reject if both are set, matching the existing mutual-exclusion behavior.
• Alternatively, do the migration in the provider's AddDefaults (currently a no-op at provider.go:461-463), using the active-defaulting pattern from OpenStack / Azure.
  This keeps the logic in the provider but runs via the same webhook path.

Note: deriving location from the datacenter name prefix is a heuristic.
If a more reliable mapping is required, the migration could query the Hetzner /datacenters/{id} endpoint while it still exists, but that adds a cloud API call to admission and ties migration to the deprecated endpoint.
The prefix heuristic is simpler and sufficient for known datacenter names (nbg1-dc3, fsn1-dc14, hel1-dc2, ash-dc1, hil-dc1).

3. Decide on deprecation window vs. hard removal

The Hetzner datacenter request/response property removal (after 1 July 2026) has already happened, so the safe path is urgent:

• Minimal / safe: accept datacenter in specs, migrate it to location (prefix heuristic), log a warning, and stop sending datacenter to the Hetzner API.
  This keeps existing specs working without user action.
• After the deadline (or once the field is unused): remove Datacenter from RawConfig and Config entirely.

jsonutil.StrictUnmarshal is used for RawConfig (types.go:38-42), so removing the field will cause strict unmarshal to reject specs that still set datacenter.
Hard removal must come with a release note and must follow the migration window.

4. Update fixtures, examples, and docs

Files carrying hetzner datacenter:

Non-empty (real datacenter values, most affected by removal):

• sdk/apis/cluster/v1alpha1/conversions/testdata/clusterv1alpha1machineSetWithProviderConfig/hetzner.yaml:42 (nbg1-dc3)
• sdk/apis/cluster/v1alpha1/conversions/testdata/clusterv1alpha1machineDeploymentWithProviderConfig/hetzner.yaml:35 (nbg1-dc3)
• sdk/apis/cluster/v1alpha1/conversions/testdata/migrated_clusterv1alpha1machineDeploymentWithProviderConfig/hetzner.yaml:36 (nbg1-dc3)
• sdk/apis/cluster/v1alpha1/conversions/testdata/migrated_clusterv1alpha1machineSetWithProviderConfig/hetzner.yaml:42 (nbg1-dc3)

Empty (paired with a set location):

• docs/cloud-provider.md:228 (location: fsn1 at :229)
• examples/hetzner-machinedeployment.yaml:48 (location: fsn1 at :49)
• test/e2e/provisioning/testdata/machinedeployment-hetzner.yaml:31 (location: nbg1 at :32)
• test/e2e/provisioning/testdata/machine-invalid.yaml:16 (location: fsn1 at :17)
• sdk/apis/cluster/v1alpha1/conversions/testdata/clusterv1alpha1machineWithProviderConfig/hetzner.yaml:11 (location: fsn1 at :12)
• sdk/apis/cluster/v1alpha1/conversions/testdata/migrated_clusterv1alpha1machineWithProviderConfig/hetzner.yaml:14 (location: fsn1 at :15)
• sdk/apis/cluster/v1alpha1/conversions/testdata/migrated_clusterv1alpha1machine/hetzner.yaml:12 (location: fsn1 at :13)
• sdk/apis/cluster/v1alpha1/conversions/testdata/machinesv1alpha1machine/hetzner.yaml:17 (location: fsn1 at :18)

Action: remove the datacenter key from all of these.
The four non-empty conversion fixtures (nbg1-dc3) should move the value to location: nbg1 if they are meant to keep exercising a populated location, or drop it if the test only checks round-trip serialization.
Verify the conversion tests still pass.

Out of scope (different API surface): seed.yaml carries hetzner datacenter values (hel1-dc2 at line 91, nbg1-dc3 at line 98; also embedded in the last-applied-configuration annotation at line 6) but this is a KKP Seed resource, not the machine-controller cloudProviderSpec.
It is not consumed by pkg/cloudprovider/provider/hetzner/provider.go.
Track separately in KKP.

Suggested task breakdown

1. Add migrateHetznerDatacenter to pkg/admission/util.go and wire it into mutationsForMachineDeployment; cover with unit tests (datacenter set, location empty -> migrated; both set -> rejected; location set -> noop).
2. In the provider, stop calling client.Datacenter.Get in Validate and Create; stop setting serverCreateOpts.Datacenter.
   Route everything through location.
3. Mark RawConfig.Datacenter // Deprecated:; keep parsing for the migration window.
4. Update MachineMetricsLabels to drop the dc label.
5. Update fixtures, examples, and docs/cloud-provider.md.
6. Bump hcloud-go from v2.13.1 to v2.33.0+ (latest is v2.44.0) so the ServerCreateOpts.Datacenter deprecation is surfaced via // Deprecated: and staticcheck.
   This is independent of the user-facing provider fix.
7. Release note for KKP / KubeOne users: Hetzner datacenter is deprecated and auto-migrated to location; switch specs to location before the field is removed.