maintenance: version cluster message envelopes

[zqr10159]

Summary

This standalone update defines a versioned cluster-message envelope, a safe
fresh-install path, and an explicit mixed-version rollout path.

• bases the complete change directly on current master as one neutral commit;
  it no longer depends on [fix](security): stop sending AES secret over unauthenticated Netty #4263
• binds signed messages to a per-connection server challenge
• ships required enforcement as the Java and YAML default
• keeps optional available only as an explicit mixed-version rollout setting
• supports active and previous key IDs for non-disruptive rotation
• derives a dedicated MAC key with HKDF domain separation, or accepts an
  independently configured authentication secret
• validates configuration before the Netty endpoint starts
• makes clock skew and handshake timeout configurable
• records bounded rejection, legacy-acceptance, and handshake-timeout metrics

Compatibility matrix

Manager	Collector	Result
old	old	compatible
old	new explicit optional	compatible during rollout
new explicit optional	old	compatible during rollout
new optional	new optional	compatible during rollout
new required	new required	compatible steady state
old	new required	rejected
new required	old	rejected

Installation and upgrade

Manager and every standalone Collector now receive two independent values:

• COMMON_SECRET: the identical AES key on both sides, exactly 16, 24, or 32
  bytes
• CLUSTER_AUTH_ACTIVE_SECRET: a separate cluster-message authentication
  secret of at least 32 bytes

Fresh Docker and Compose instructions use:

umask 077
printf 'COMMON_SECRET=%s\n' "$(openssl rand -hex 16)" > .env
printf 'CLUSTER_AUTH_ACTIVE_SECRET=%s\n' "$(openssl rand -hex 32)" >> .env

rand -hex 16 produces 32 ASCII bytes and is therefore a valid AES-256
COMMON_SECRET. rand -hex 32 produces 64 ASCII characters for cluster
authentication and must not be reused directly as the AES key.

All five shipped Compose definitions fail interpolation with an actionable
message when either value is absent. Their packaged application configuration
maps COMMON_SECRET to common.secret, and the same private .env is supplied
to Manager and standalone Collector.

Existing installations must preserve their current AES secret. Replacing it
during an ordinary upgrade can make encrypted data unreadable. The upgrade
guide covers recovering the existing Manager value through restricted
administrator access, provisioning both values on every node, using explicit
optional mode only during the mixed-version window, and returning to required
mode after convergence.

Regression proof

• The default-mode contract failed on an earlier implementation with
  expected REQUIRED but was OPTIONAL; it now passes.
• The first packaging contract failed on the reviewed head because Compose did
  not require an install-specific authentication secret; it now passes for all
  five distributions.
• The reopened packaging contract failed because Compose, package
  configuration, and installation documentation omitted COMMON_SECRET; it
  now requires and documents both independent values.
• CollectorDocumentedEnvironmentIntegrationTest starts a separate real
  Collector JVM with the documented environment, waits for Spring startup, and
  verifies that it stays alive without missing- or invalid-AES-secret errors.
• A separate extracted release-package proof starts
  apache-hertzbeat-collector-1.9.0.jar with those same two variables and
  observes stable startup.

Validation on the standalone master-based commit

• Focused authentication, compatibility, Manager/Collector, packaging, and
  documented-process reactor: all selected tests passed.
• Real Collector process integration: passed and leaves generated logs under
  ignored target output.
• Docker Compose v5.0.2: all five source definitions render successfully with
  independent valid fixtures.
• Standalone Collector release package: 13-module cluster-profile assembly
  passed; the extracted packaged JAR reached stable startup with the documented
  environment.
• Startup source package proof: all 24 reactor modules passed.
• Release-profile assembly: all 24 reactor modules passed and produced server,
  Docker, and Docker Compose archives.
• The generated Compose archive retains both required interpolation
  expressions, the common.secret mapping, and the corrected AES generator in
  all five packaged distributions.
• Maven Checkstyle, added-code CJK scan, git diff --check, and
  git diff --cached --check passed.
• Branch shape: current master plus one commit (e533b4e106), with no
  dependency branch.
• GitHub CI passed on the amended head: Backend Build (9m51s), Backend Maven
  E2E (12m57s), Backend Image E2E (2m46s), DOC CI (9m25s), License Checker,
  and labeler.

Operational notes

Blank, known-default, weak authentication secrets and invalid-length AES
secrets fail before the standalone Collector endpoint starts. Required mode
rejects unsigned peers, so existing clusters must follow the documented
explicit-optional rollout sequence rather than upgrading Manager first. Clock
synchronization, secret backup, and controlled key distribution remain
deployment responsibilities.

AI assistance: used for draft implementation and test iteration.
Human validation: reproduced the documented startup failure, ran focused
authentication/compatibility and bootstrap tests, rendered all Compose
variants, started a real packaged Collector, and completed source and release
package proofs.
Risk notes: optional mode deliberately accepts unsigned legacy traffic and must
be enabled only for a bounded rollout window; required remains the shipped and
intended steady-state default.

[zqr10159]

Author remediation update and remaining blocker:

The branch now has versioned, challenge-bound envelopes; configurable optional-to-required rollout modes; HKDF-separated authentication keys; key IDs and dual-key acceptance for rotation; startup validation; configurable clock skew; rejection metrics; replay handling; and an old and new manager and collector test matrix. Focused and reactor validation passed locally.

This PR is intentionally not author-approved for direct merge yet. It still targets the unmerged #4263 branch, so it must be rebased onto the final integrated master state after #4263 lands and then receive the full repository CI. GitHub currently shows only the label check. Until that dependency and CI condition are satisfied, the merge result remains blocked.

[zqr10159]

The remaining review blockers are addressed in the latest head (7768405ced). The PR now targets current master directly and contains exactly one commit, so it no longer depends on #4263. Java plus all shipped YAML defaults are required; optional remains available only through an explicit rollout override. The new default regression failed on the prior head (expected REQUIRED but was OPTIONAL) and now passes. Latest local validation: 21 remoting authentication/compatibility tests, 16 targeted manager/collector tests, and the 24-module startup source package proof all passed. Full GitHub CI has now been triggered by the master retarget/push.

[zqr10159]

Standalone verification is now complete on head 183cf6bb21.

• The PR targets master and is exactly one commit above it; the former [fix](security): stop sending AES secret over unauthenticated Netty #4263 dependency is included in that standalone commit.
• required is the shipped Java and YAML default. optional remains available only as an explicit mixed-version rollout override, with removal after convergence documented in the PR.
• GitHub checks are green: Backend Build, Backend Maven E2E, Backend Image E2E, DOC CI, License Checker, and labeler.

For transparency, the first Maven E2E attempt stopped after 2m42s in the unrelated CommonHttpClientVirtualThreadTest.dispatchConnectionPoolCleanupClosesExpiredAndIdleConnections timing test, where a cleanup call was observed twice. This PR does not modify that client or test, and the complete class passed locally. GitHub attempt 2 passed the full Maven E2E job without any code change in 12m45s: https://github.com/apache/hertzbeat/actions/runs/30592359568

The dependency, CI coverage, and secure steady-state default blockers from the review are therefore resolved.

[zqr10159]

Addressed the remaining default-install and upgrade blocker on head
0d850535b1.

What changed:

• all five shipped Docker Compose definitions now require
  CLUSTER_AUTH_ACTIVE_SECRET during interpolation and return an actionable
  error before container startup when it is absent;
• every corresponding conf/application.yml exposes the complete
  authentication block;
• each English and Chinese distribution guide creates a private .env with
  openssl rand -hex 32, requires the value to be preserved across upgrades,
  and requires standalone Collectors to use the same value;
• root/localized quick starts plus Docker, Compose, package, native-Collector,
  Collector-reference, and upgrade documentation now describe generation,
  sharing, backup, and optional-to-required rollout;
• a Manager regression proves the all-in-one default replaces and persists its
  known-default AES root before Netty authentication validation, then reuses
  the same root after restart.

Regression proof:

• on the reviewed head, the packaging contract failed because the Compose
  distributions did not require/document an install secret and the package
  upgrade path was incomplete;
• after the change, all five definitions fail as expected when the variable is
  absent and all five render with a valid fixture under Docker Compose v5.0.2.

Validation:

• remoting authentication/compatibility: 18 tests passed, including the
  seven-combination old/new loopback matrix;
• Manager bootstrap and packaging contracts: 3 tests passed;
• startup source package: 24-module reactor passed;
• release profile: 24-module reactor passed and generated server, Docker, and
  Docker Compose archives;
• the generated Compose archive preserves the required-secret expression in
  all five definitions;
• Checkstyle, added-test CJK scan, and Git whitespace checks passed.

The branch remains exactly one commit above master. GitHub CI is running on
the amended head.

[zqr10159]

Final current-head CI update for 0d850535b1: all checks passed.

• Backend Build: 9m35s
• Backend Maven E2E: 13m01s
• Backend Image E2E: 1m52s
• DOC CI: 7m26s
• License Checker: passed
• labeler: passed

The PR remains based on master, mergeable, and exactly one commit above the
base. No rerun or code change was needed.

[zqr10159]

Addressed the remaining standalone Collector bootstrap blocker on head
e533b4e106.

The installation contract now provisions two independent values to Manager and
every standalone Collector:

• COMMON_SECRET, generated with openssl rand -hex 16, yielding a valid
  32-byte AES key;
• CLUSTER_AUTH_ACTIVE_SECRET, independently generated with
  openssl rand -hex 32.

The 64-character authentication output is explicitly not reused as
COMMON_SECRET. All five Compose definitions require both variables, all five
packaged application configurations map COMMON_SECRET to common.secret,
and the root/localized quick starts plus Docker, Compose, package, native
Collector, Collector-reference, and upgrade guides describe the same contract.
The upgrade path preserves an existing AES key rather than replacing it and
risking unreadable encrypted data.

Regression and runtime proof:

• the new packaging contract failed on the reviewed head because Compose and
  documentation omitted COMMON_SECRET;
• the updated 22-module packaging/authentication reactor passed;
• CollectorDocumentedEnvironmentIntegrationTest launches a separate real
  Collector JVM with both documented variables and verifies stable startup;
• a rebuilt and extracted apache-hertzbeat-collector-1.9.0-bin.tar.gz was
  started with the same environment, reached Started Collector, and remained
  alive without a common-secret validation error;
• all five Compose definitions rendered successfully;
• the 24-module startup package and 24-module release-profile assembly passed;
• the generated Compose archive retains both requirements in all five
  distributions.

The temporary package extraction and source-tree test logs were removed after
verification. GitHub CI is now running on the amended head.

[zqr10159]

Final current-head CI update for e533b4e106: all checks passed.

• Backend Build: 9m51s
• Backend Maven E2E: 12m57s
• Backend Image E2E: 2m46s
• DOC CI: 9m25s
• License Checker: passed
• labeler: passed

The PR remains based directly on master, mergeable, and exactly one commit
above the base. Together with the real packaged-Collector startup proof, the
standalone configuration and documentation blocker is resolved.

[Duansg]

I think there's a gap in the rollout story that blocks merging as-is.

The compatibility matrix misses AES key distribution

optional mode makes a new Manager accept unsigned messages from an old Collector, so the signing side of a mixed-version rollout is covered. But on master, the GO_ONLINE response is the only place an old Collector ever obtains the AES key (GoOnlineProcessor.java:49-54):

ServerInfo serverInfo = JsonUtil.fromJson(message.getMsg().toStringUtf8(), ServerInfo.class);
if (serverInfo == null || serverInfo.getAesSecret() == null) {
log.warn("The message that server response to collector has not secret empty, please check");
} else {
AesUtil.setDefaultSecretKey(serverInfo.getAesSecret());
}

This PR deletes ServerInfo entirely and strips that branch. So with new Manager + old Collector, the collector logs one warning, keeps AesUtil.secretKey at DEFAULT_ENCODE_RULES, comes online successfully — and then every job carrying a PARAM_TYPE_PASSWORD configmap fails at WheelTimerTask.java:70-73:

if (item.getType() == CommonConstants.PARAM_TYPE_PASSWORD && item.getValue() != null) {
String decodeValue = AesUtil.aesDecode(String.valueOf(item.getValue()));
if (decodeValue == null) {
log.error("Aes Decode value {} error.", item.getValue());

The credential is set to null and collection silently breaks for every monitor with a password parameter. optional mode does not help here, because the failure is in key distribution, not message signing.

The added compatibility tests exercise heartbeat/handshake paths; as far as I can tell none dispatch a job containing a real encrypted parameter, which is why this doesn't show up. Worth adding a test at that level.

Suggestion: either keep a documented upgrade order (upgrade all collectors first, with common.secret pre-provisioned, before upgrading the manager) enforced/asserted somewhere, or have the manager detect a pre-auth collector and refuse to dispatch credentialed jobs to it rather than dispatching jobs it cannot decrypt.

Breaking change that needs a PMC decision

CollectServer.initializeAesSecret() now throws on a blank common.secret. Since the key used to arrive over Netty, every existing standalone collector will fail to start after upgrade until an operator pulls the manager's randomized AES secret out of SystemSecret and provisions it on each collector.

I think that's the correct end state, and I can see the docs were updated across the deploy guides. But it's a hard compatibility break for anyone running remote collectors, so it needs to be an explicit release decision with a prominent upgrade note — not something carried under a maintenance: title. Worth raising on dev@ before merge.

AUTH_CHALLENGE is processed before authentication and can re-pin the nonce

In processReceiveMsg, the AUTH_CHALLENGE branch returns before the verification block, and setChannelNonce has no guard against being called again. A peer that can write to the socket can therefore send a well-formed challenge after the handshake and overwrite the stored channel nonce. It can't forge a signature, so this isn't a bypass, but it does let an on-path party break an established session at will, and it makes the channel-binding invariant weaker than it reads.

Suggestion: accept a challenge only while the channel nonce is unset (mirror the APPLICATION_ACTIVE_NOTIFIED compare-and-set pattern you already use).

Replay cache can evict inside the acceptance window

maximumSize(65_536) with expireAfterWrite(2 × maxClockSkew): on a busy cluster — every heartbeat and every metrics message is signed and inserted — entries can be evicted well before the 10-minute window closes, at which point a captured message becomes replayable again within its skew window. The timestamp bound still limits the exposure, so this is a degradation rather than a hole, but it should be stated in the docs rather than implied to be absolute.

Performance — needs numbers before I'd call it a regression

Every message is signed via message.toByteArray() (a full copy), and verification does toBuilder().clearAuthSignature().build().toByteArray() — two more full serializations — on the Netty event loop. For metrics uploads, which are the large messages here, that's three extra full serializations per message plus the HMAC. I have no benchmark, so I'm flagging it as something to measure rather than asserting a regression, but given this sits on the data path it seems worth a throughput comparison before/after.

Minor

.gitignore and the three README files are unrelated to cluster message authentication. Splitting those out would make the PR easier to review and easier to revert if needed.