fix(pd): validate REST credentials and return 401 on refusal (#3189)

PD's REST interceptor decodes the Basic credential, keeps only the part before the colon, and checks it against the fixed set hg, store, hubble, vermeer. Any of those names with any password, including an empty one, is treated as an internal component; the password is never read. Separately, RestAuthentication.preHandle writes an error body without calling setStatus, so success, refusal and missing credential all return HTTP 200 and nothing keyed on a status code (monitors, curl -f, the shipped healthchecks) can see a refusal. The endpoints behind the interceptor mutate the cluster: POST /v1/members/change, DELETE /v1/store/{storeId}, graph and graphspace writes, and the balance and patrol tasks. The issue has the measured 27-request matrix.

---------

Co-authored-by: imbajin <jin@apache.org>

Author: bitflicker64

.github/workflows/pd-store-ci.yml

@@ -134,6 +134,14 @@ jobs:
           done
           echo "can_run=true" >> "$GITHUB_OUTPUT"
 
+      - name: Run PD docker entrypoint secret override tests
+        run: |
+          $TRAVIS_DIR/test-pd-docker-entrypoint.sh
+
+      - name: Check every shipped PD config carries the REST hardening
+        run: |
+          $TRAVIS_DIR/test-pd-shipped-config.sh
+
       - name: Run start-hugegraph-pd.sh foreground mode tests
         if: steps.pd-preflight.outputs.can_run == 'true'
         run: |

.gitignore

@@ -44,6 +44,8 @@ build/
 .env.test.local
 .env.production.local
 docker/.env
+# generated by docker/set-hubble-pd-password.sh, carries the PD REST secret
+docker/conf/hubble/*.local.properties*
 
 *.orig
 *.rej

docker/README.md

@@ -39,33 +39,42 @@ contains a single quote or newline.
     echo ".env already exists; edit it instead of overwriting it" >&2
     exit 1
   }
-  printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" \
-    'replace-with-your-password' "${jwt_secret}" > .env
+  pd_secret="$(openssl rand -hex 24)"
+  printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\nHG_PD_AUTH_SECRET_KEY='%s'\n" \
+    'replace-with-your-password' "${jwt_secret}" "${pd_secret}" > .env
+  # Hubble reads the PD secret from a file, not from .env: generate the untracked properties files the HStore topologies mount.
+  HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore
+  HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore-ha
 )
 ```
 
-Do not commit `.env`. Keeping the same JWT secret preserves authentication
-tokens when containers are recreated. For authenticated topologies with
-multiple Server replicas, all replicas receive this same secret. The HA
-topology fails fast if authentication is enabled without this shared secret.
+Do not commit `.env` or `conf/hubble/*.local.properties`; both are in `.gitignore`. Keeping the same JWT secret preserves authentication tokens when containers are recreated. For authenticated topologies with multiple Server replicas, all replicas receive this same secret. The HA topology fails fast if authentication is enabled without this shared secret.
 
-A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and
-Hubble detects that mode automatically. Omitting the variable or setting it to
-an empty value disables authentication. Auth-off is only suitable for a
-trusted local environment; never expose it to a public or untrusted network.
-Hubble listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only
-behind an HTTPS reverse proxy and trusted network controls.
+A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and Hubble detects that mode automatically. Omitting the variable or setting it to an empty value disables authentication. Auth-off is only suitable for a trusted local environment; never expose it to a public or untrusted network. Hubble listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only behind an HTTPS reverse proxy and trusted network controls.
 
-`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its
-first authenticated startup. Changing `.env` does not rotate an existing
-administrator password; use the HugeGraph user API for credential changes.
+`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its first authenticated startup. Changing `.env` does not rotate an existing administrator password; use the HugeGraph user API for credential changes.
 
-For the verification commands below, set the password in your current shell:
+For the verification commands below, load `.env` into your current shell and set the password:
 
 ```bash
+set -a; . ./.env; set +a
 ADMIN_PASSWORD='the-same-password-used-in-.env'
 ```
 
+The PD REST API (port 8620, HStore topologies only) requires HTTP Basic auth (`hg:${HG_PD_AUTH_SECRET_KEY}`) for all endpoints except health/readiness probes (`/v1/health`, `/v1/ready`). `HG_PD_AUTH_SECRET_KEY` is shared across PD, Server (`bin/wait-storage.sh`), and Hubble (`conf/hubble/*.local.properties` generated by `./set-hubble-pd-password.sh`).
+
+Verify registered stores:
+
+```bash
+curl -u "hg:${HG_PD_AUTH_SECRET_KEY}" http://localhost:8620/v1/stores
+```
+
+To regenerate Hubble configuration after modifying `.env`:
+
+```bash
+./set-hubble-pd-password.sh hstore      # or hstore-ha
+```
+
 ### Standalone
 
 This is the recommended quickstart.
@@ -181,8 +190,7 @@ Status:
 docker compose -f docker-compose-3pd-3store-3server.yml ps
 ```
 
-Verify all published PD, Store, and Server endpoints, Server authentication,
-and Hubble:
+Verify all published PD, Store, and Server endpoints, Server authentication, and Hubble:
 
 ```bash
 for port in 8620 8621 8622; do
@@ -202,34 +210,11 @@ done
 curl -fsS http://localhost:8088/about
 ```
 
-PD answers two unauthenticated probe endpoints. `/v1/health` is liveness only:
-it returns `200` as soon as the REST listener is up, even when the PD has no
-raft leader. `/v1/ready` returns `200` only while the PD sees a raft leader,
-and `503` otherwise. Each PD answers for itself: a single PD elects itself, and
-in a three-PD group the two that can reach each other elect a leader and turn
-ready, while a partitioned third keeps answering `503` until it sees that
-leader.
-
-The healthchecks in these files still gate on `/v1/health`, because
-`/v1/ready` ships from the next release onwards while the files run published
-images. Two things to know before pointing them at readiness:
-
-- Match on the body, not the status code. As of 1.7.0 PD answers `200` with
-  `{"status":-1,"error":"Unauthorized!"}` on every path its auth interceptor
-  does not exclude, a path that does not exist included, so a status-only
-  probe reads a PD too old to have `/v1/ready` as ready. The body match holds
-  whichever status a refusal carries. Gate with
-  `curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'` instead.
-- Pin `HUGEGRAPH_VERSION` to a release that carries the endpoint, or build the
-  images from source with `docker-compose.dev.yml`.
-
-The `HEALTHCHECK` baked into `hugegraph-pd/Dockerfile` is `/v1/health` as well.
-Both compose files override it, so it governs `docker run` and anything else
-inheriting the image probe, and those keep reading a PD without a quorum as
-healthy.
+PD answers two unauthenticated probe endpoints: `/v1/health` for liveness (returns 200 once the REST listener is up, regardless of raft state), and `/v1/ready` for readiness (returns 200 only when PD sees a raft leader, 503 otherwise).
 
-Open `http://localhost:8088` and sign in as `admin` with the password from
-`.env`.
+Compose healthchecks currently gate on `/v1/health` for compatibility with published images. When targeting readiness on newer releases or source builds (`docker-compose.dev.yml`), match on the response body (`curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'`).
+
+Open `http://localhost:8088` and sign in as `admin` with the password from `.env`.
 
 Stop containers while keeping them:
 
@@ -265,9 +250,7 @@ HUBBLE_IMAGE=hugegraph/hubble:latest \
 docker compose -f docker-compose.yml up -d
 ```
 
-The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and
-Server `latest`; compatibility with versions older than 1.7 is not promised.
-Pin immutable image references when reproducibility is required.
+The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and Server `latest`; compatibility with versions older than 1.7 is not promised. Pin immutable image references when reproducibility is required.
 
 ### Server startup timeout
 
@@ -348,16 +331,15 @@ docker compose -f docker-compose-hstore.yml up -d --wait
 
 ### Hubble configuration
 
-The three small files under `conf/hubble/` contain only topology-specific
-discovery settings and container paths:
+The three small files under `conf/hubble/` contain only topology-specific discovery settings, the PD REST credential (`operations.pd.username` and `operations.pd.password`, which must match PD's `auth.secret-key`), and container paths:
 
 - `conf/hubble/standalone.properties` uses direct Server mode.
-- `conf/hubble/hstore.properties` uses one PD and one Store REST target.
-- `conf/hubble/hstore-ha.properties` uses all three PD peers and all three
-  allowed Store REST targets.
+- `conf/hubble/hstore.properties.example` uses one PD and one Store REST target.
+- `conf/hubble/hstore-ha.properties.example` uses all three PD peers and all three allowed Store REST targets.
+
+The two HStore topologies mount the generated `*.local.properties` next to these examples (see `set-hubble-pd-password.sh`), never the examples themselves, so the PD secret stays out of tracked files.
 
-Hubble detects Server authentication through the Server API. Do not add an
-`auth.enabled` property or duplicate auth-on/auth-off configurations.
+Hubble detects Server authentication through the Server API. Do not add an `auth.enabled` property or duplicate auth-on/auth-off configurations.
 
 ### Render and smoke checks
 
@@ -367,12 +349,9 @@ Render every topology with auth-on inputs before submitting a change:
 bash test-compose.sh render
 ```
 
-The HA render is mandatory even when local resources are insufficient to start
-its ten containers.
+The HA render is mandatory even when local resources are insufficient to start its ten containers.
 
-Run focused auth-on smoke checks for standalone and minimal HStore with the
-corresponding `up -d --wait`, status, authentication, Hubble `/about`, and
-`down -v` commands from the Users section:
+Run focused auth-on smoke checks for standalone and minimal HStore with the corresponding `up -d --wait`, status, authentication, Hubble `/about`, and `down -v` commands from the Users section:
 
 ```bash
 bash test-compose.sh smoke
@@ -384,6 +363,4 @@ Run the required local auth-off checks separately:
 bash test-compose.sh smoke-auth-off
 ```
 
-The auth-off mode is intentionally excluded from the default CI matrix and must
-remain on a trusted local machine. Both smoke modes remove only the isolated
-Compose projects and volumes that they create.
+The auth-off mode is intentionally excluded from the default CI matrix and must remain on a trusted local machine. Both smoke modes remove only the isolated Compose projects and volumes that they create.

docker/conf/hubble/hstore-ha.properties …conf/hubble/hstore-ha.properties.exampledocker/conf/hubble/hstore-ha.properties renamed to docker/conf/hubble/hstore-ha.properties.example docker/conf/hubble/hstore-ha.properties renamed to docker/conf/hubble/hstore-ha.properties.example

@@ -20,6 +20,12 @@ pd.enabled=true
 server.direct_url=http://server0:8080
 pd.peers=pd0:8686,pd1:8686,pd2:8686
 pd.server=pd0:8620
+# PD REST credential. The password must equal PD's auth.secret-key, which has
+# no default. Do not edit this tracked example: docker/set-hubble-pd-password.sh
+# generates the untracked .local.properties that Compose mounts, with the value
+# from HG_PD_AUTH_SECRET_KEY in .env.
+operations.pd.username=hubble
+operations.pd.password=
 operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520]
 upload_file.location=/hubble/data/upload-files
 dashboard.address=

docker/conf/hubble/hstore.properties …er/conf/hubble/hstore.properties.exampledocker/conf/hubble/hstore.properties renamed to docker/conf/hubble/hstore.properties.example docker/conf/hubble/hstore.properties renamed to docker/conf/hubble/hstore.properties.example

@@ -20,6 +20,12 @@ pd.enabled=true
 server.direct_url=http://server:8080
 pd.peers=pd:8686
 pd.server=pd:8620
+# PD REST credential. The password must equal PD's auth.secret-key, which has
+# no default. Do not edit this tracked example: docker/set-hubble-pd-password.sh
+# generates the untracked .local.properties that Compose mounts, with the value
+# from HG_PD_AUTH_SECRET_KEY in .env.
+operations.pd.username=hubble
+operations.pd.password=
 operations.store.allowed_targets=[http://store:8520]
 upload_file.location=/hubble/data/upload-files
 dashboard.address=

docker/docker-compose-3pd-3store-3server.yml

@@ -73,6 +73,8 @@ x-server-environment: &server-environment
   HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120}
   HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-}
   PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-}
+  # bin/wait-storage.sh polls the PD REST API, so it needs the same secret
+  PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
 
 x-server-common: &server-common
   image: hugegraph/server:${HUGEGRAPH_VERSION:-latest}
@@ -110,6 +112,7 @@ services:
       HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
       HG_PD_DATA_PATH: /hugegraph-pd/pd_data
       HG_PD_INITIAL_STORE_COUNT: 3
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
     ports: ["8620:8620", "8686:8686"]
     volumes:
       - hg-pd0-data:/hugegraph-pd/pd_data
@@ -128,6 +131,7 @@ services:
       HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
       HG_PD_DATA_PATH: /hugegraph-pd/pd_data
       HG_PD_INITIAL_STORE_COUNT: 3
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
     ports: ["8621:8620", "8687:8686"]
     volumes:
       - hg-pd1-data:/hugegraph-pd/pd_data
@@ -146,6 +150,7 @@ services:
       HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
       HG_PD_DATA_PATH: /hugegraph-pd/pd_data
       HG_PD_INITIAL_STORE_COUNT: 3
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
     ports: ["8622:8620", "8688:8686"]
     volumes:
       - hg-pd2-data:/hugegraph-pd/pd_data
@@ -239,7 +244,15 @@ services:
       - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088"
     volumes:
       - hubble-data:/hubble/data
-      - ./conf/hubble/hstore-ha.properties:/hubble/conf/hugegraph-hubble.properties:ro
+      - type: bind
+        source: ./conf/hubble/hstore-ha.local.properties
+        target: /hubble/conf/hugegraph-hubble.properties
+        read_only: true
+        bind:
+          # The file is generated by set-hubble-pd-password.sh and is
+          # gitignored. Without this, Docker would create an empty
+          # directory at that path and Hubble would boot unconfigured.
+          create_host_path: false
     healthcheck:
       test:
         - CMD-SHELL

docker/docker-compose-hstore.yml

@@ -39,6 +39,7 @@ services:
       HG_PD_RAFT_PEERS_LIST: pd:8610
       HG_PD_INITIAL_STORE_LIST: store:8500
       HG_PD_DATA_PATH: /hugegraph-pd/pd_data
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
     ports:
       - "8620:8620"
     volumes:
@@ -97,6 +98,8 @@ services:
       HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120}
       HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-}
       PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-}
+      # bin/wait-storage.sh polls the PD REST API, so it needs the same secret
+      PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
     ports:
       - "8080:8080"
     healthcheck:
@@ -120,7 +123,15 @@ services:
       - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088"
     volumes:
       - hubble-data:/hubble/data
-      - ./conf/hubble/hstore.properties:/hubble/conf/hugegraph-hubble.properties:ro
+      - type: bind
+        source: ./conf/hubble/hstore.local.properties
+        target: /hubble/conf/hugegraph-hubble.properties
+        read_only: true
+        bind:
+          # The file is generated by set-hubble-pd-password.sh and is
+          # gitignored. Without this, Docker would create an empty
+          # directory at that path and Hubble would boot unconfigured.
+          create_host_path: false
     healthcheck:
       test:
         - CMD-SHELL

docker/set-hubble-pd-password.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Generate the Hubble properties file a Compose topology mounts, with PD's
+# REST secret written in as operations.pd.password.
+#
+#   usage: set-hubble-pd-password.sh <hstore|hstore-ha> [secret]
+#
+# Reads conf/hubble/<name>.properties.example (tracked) and writes
+# conf/hubble/<name>.local.properties (ignored by git), so the secret never
+# lands in a tracked file. The secret defaults to $HG_PD_AUTH_SECRET_KEY. The
+# value never goes through a sed replacement, where & # and backslash are
+# special, and backslashes are doubled for the .properties format. Run this
+# before `docker compose up`: the bind pins create_host_path: false, so a
+# missing target makes Compose refuse to start.
+#
+# The secret must be printable ASCII. PD compares it as UTF-8 bytes
+# (Authentication.verifySecret), while Hubble reads this file through
+# commons-configuration2, whose DEFAULT_ENCODING is ISO-8859-1, so a non-ASCII
+# secret decodes to different bytes on the two sides and gives a permanent 401
+# with no diagnostic anywhere. The README recipe generates hex, which is safe.
+set -euo pipefail
+
+name=${1:?usage: $0 <hstore|hstore-ha> [secret]}
+secret=${2:-${HG_PD_AUTH_SECRET_KEY:-}}
+dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/conf/hubble"
+example="${dir}/${name}.properties.example"
+out="${dir}/${name}.local.properties"
+
+[[ -f "$example" ]] || { echo "no such topology: ${name} (expected ${example})" >&2; exit 1; }
+[[ -n "$secret" ]] || { echo "secret is empty; load .env first (set -a; . ./.env; set +a)" >&2; exit 1; }
+case "$secret" in
+    *$'\n'*|*$'\r'*) echo "secret contains a line break, which a .properties value cannot hold" >&2; exit 1 ;;
+esac
+# LC_ALL=C so the range is ordinal and the walk byte-wise: under the caller's
+# collation a non-ASCII character can sort inside \x20-\x7e and slip through.
+is_printable_ascii() {
+    local LC_ALL=C
+    case "$1" in
+        *[!$'\x20'-$'\x7e']*) return 1 ;;
+    esac
+}
+is_printable_ascii "$secret" || {
+    echo "secret must be printable ASCII: Hubble reads .properties as ISO-8859-1, PD compares as UTF-8" >&2
+    exit 1
+}
+
+escaped=${secret//\\/\\\\}
+# java.util.Properties skips whitespace between the separator and the value, so
+# a secret that starts with a space would reach Hubble shortened while PD and
+# the Server kept the original. A backslash before it keeps that first byte.
+case "$escaped" in
+    [$' \t']*) escaped="\\${escaped}" ;;
+esac
+tmp=$(mktemp "${out}.XXXXXX")
+trap 'rm -f "$tmp"' EXIT
+{
+    printf '# Generated from %s by set-hubble-pd-password.sh; not tracked by git.\n' "$(basename "$example")"
+    grep -v '^operations\.pd\.password=' "$example" || true
+    printf 'operations.pd.password=%s\n' "$escaped"
+} > "$tmp"
+# Hubble runs unprivileged and the mount is read-only, so the file must be world-readable
+chmod 644 "$tmp"
+mv "$tmp" "$out"
+trap - EXIT
+echo "wrote ${out}"