Skip to content

fix: use local dist key for bundle verification in config update polling - #152

Merged
encodeous merged 2 commits into
encodeous:mainfrom
weisanju:fix/dist-key-zero-value-fallback
Sep 5, 2026
Merged

fix: use local dist key for bundle verification in config update polling#152
encodeous merged 2 commits into
encodeous:mainfrom
weisanju:fix/dist-key-zero-value-fallback

Conversation

@weisanju

@weisanju weisanju commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Problem

When central distribution is configured via central.yaml's dist.repos (without an explicit dist.key), the daemon's config update polling fails repeatedly with:

ERR <node> Error updating config err="failed to unbundle config from <repo>: chacha20poly1305: message authentication failed"

This affects any node whose daemon was freshly started with a central config that has dist.repos. The WireGuard data plane comes up and nodes handshake correctly, but the daemon never applies new config revisions, making automatic distribution non-functional.

Reproduction Steps

  1. Create a central.yaml with dist.repos but no dist.key:

    dist:
      repos:
        - file:central.nybundle
        - https://example.com/central.nybundle
  2. Seal with a dist key pair:

    nylon seal -c central.yaml -k keys/dist.key
  3. Start the daemon on a node with node.yaml containing:

    dist:
      url: https://example.com/central.nybundle
      key: <valid-base64-public-key>
  4. Observe daemon logs — the initial OSS download succeeds (central.yaml is written), but every 10-second poll cycle logs:

    ERR <node> Error updating config err="failed to unbundle config from
    https://example.com/central.nybundle: chacha20poly1305: message authentication failed"
    

The recovered central.yaml on disk shows dist.key: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (zero-value placeholder).

Root Cause

The issue has two contributing factors:

1. BundleConfig serializes a zero-value Dist.Key

In state/distribution.go:68, BundleConfig unmarshals the source central.yaml, updates the timestamp, and re-marshals:

func BundleConfig(config string, rootKey NyPrivateKey) (string, error) {
    cfg := CentralCfg{}
    yaml.Unmarshal([]byte(config), &cfg)   // Dist.Key is zero (no key in source)
    cfg.Timestamp = time.Now().UnixNano()
    plainText, _ := yaml.Marshal(cfg)       // zero-value Key serialized as AAAA...
}

DistributionCfg.Key has type NyPublicKey ([32]byte). Its MarshalText() always produces a non-empty base64 string (AAAAAAAAAAAAAAAAAAAAAA== for zero-value), so it cannot be omitted by yaml:",omitempty". Since the source central.yaml typically only contains dist.repos (the signing key is specified via CLI -k), the re-marshaled config embeds a zero-value key into the bundle.

2. checkForConfigUpdates uses central config's key

In core/nylon_distribution.go:67, the config update loop reads the key from the central config:

key := n.CentralCfg.Dist.Key           // zero-value placeholder

This key was recovered from the bundle via UnbundleConfig, which deserializes the YAML containing AAAAAAAAAAAAAAAAAAAAAA== back into 32 zero bytes. Using this zero-value key to decrypt the ChaCha20-Poly1305 sealed bundle always fails.

The correct key is available from the node-level config (n.LocalCfg.Dist.Key), which was used successfully for the initial OSS fetch in readCentralConfig().

Fix

In checkForConfigUpdates, prefer the local (node-level) dist key from node.yaml, falling back to the central key for backward compatibility, and return an error if no valid key is available.

key := n.CentralCfg.Dist.Key
if n.LocalCfg.Dist != nil && n.LocalCfg.Dist.Key != (state.NyPublicKey{}) {
    key = n.LocalCfg.Dist.Key
}
if key == (state.NyPublicKey{}) {
    return errors.New("no valid dist key configured for bundle verification")
}

This is safe because:

  • The local key was already used successfully for the initial OSS fetch
  • n.LocalCfg is always populated from node.yaml in NewNylon
  • Fallback preserves backward compatibility for deployments that set dist.key in central.yaml
  • The zero-value guard catches misconfiguration early

Testing

  • go build ./... — compiles cleanly
  • go test ./... — all tests pass
  • Deployed to 3-node mesh — daemon logs show no auth errors; config updates from OSS succeed
  • WireGuard handshakes and Babel routes remain intact

checkForConfigUpdates was reading n.CentralCfg.Dist.Key, which may be a
zero-value placeholder (AAAAAAAAAAAAAAAAAAAAAA==). This happens because
BundleConfig re-marshals the CentralCfg after unmarshalling the source
central.yaml, serializing the zero-value DistributionCfg.Key field when
it was not explicitly set. Using this zero-value key for bundle decryption
always fails with chacha20poly1305: message authentication failed.

Fix: prefer the local (node-level) dist key from node.yaml, falling back
to the central key for backward compatibility, and returning an error if
neither key is valid.

Fixes #NNN
@encodeous

Copy link
Copy Markdown
Owner

I think the real issue here is that we don't warn the user when the config's pubkey is not the same as the key used to seal the config. (this is not an error however, since we want the user to be able to rotate keys)

I'm thinking of reworking the local dist config in the near future. It feels a little inconsistent to me.

@encodeous
encodeous force-pushed the fix/dist-key-zero-value-fallback branch from 9302d1c to df81f2c Compare September 5, 2026 13:49
@encodeous
encodeous merged commit cec40e4 into encodeous:main Sep 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants