[Experimental] Prototype Rust CPEX secrets detection integration

[gandhipratik203]

Summary

Adds the Rust-native secrets detection CPEX plugin as a first-class dataplane workspace crate and wires it behind the existing plugins feature.

This supersedes the earlier local-path spike by keeping the plugin source inside the dataplane repo, so CI and reviewers do not need a sibling cpex-plugins checkout.

What Changed

• Added cpex-secrets-detection as a workspace crate.
• Registered the secrets detection factory from the gateway binary when plugins is enabled.
• Updated the Docker image build to compile the gateway with contextforge-gateway-rs/plugins.
• Added CI workspace test/build coverage with all features enabled; the ignored binary E2E remains a manual/local check.
• Updated gateway plugin tests to use the local workspace crate.
• Added CPEX PluginManager coverage for redaction, blocking, clean no-op, field filters, and supported CMF hook stages.
• Documented experimental two-stage activation: compile with plugins, start with --runtime-plugins-enabled true, and provide Redis config.
• Documented runtime Redis config and known CPEX 0.2.2 propagation gaps in the crate README.

Explanatory Diagrams

Request/response flow:

MCP client
   |
   | tools/call
   v
contextforge-gateway-rs
   |
   | before backend call
   v
CPEX PluginManager
   |
   | cmf.tool_pre_invoke
   v
secrets detection plugin
   |
   | allow / redact / block
   v
backend MCP server
   |
   | tool result
   v
CPEX PluginManager
   |
   | cmf.tool_post_invoke
   v
secrets detection plugin
   |
   | allow / redact / block
   v
MCP client

Integration summary:

Gateway binary
   registers
SecretsDetectionFactory
   with
CPEX PluginManager
   loads config from
Redis
   invokes
cmf.tool_pre_invoke / cmf.tool_post_invoke
   on
Rust secrets detection plugin

Component Layout

                          Runtime Config
                              Redis
                               |
                               |  ContextForgeGatewayRuntimePluginConfig
                               v
+---------------------------------------------------------------+
|                    contextforge-data-plane                    |
|                                                               |
|  +--------------------------+                                 |
|  | contextforge-gateway-rs  |                                 |
|  | binary                   |                                 |
|  |                          |                                 |
|  |  --features plugins      |                                 |
|  +------------+-------------+                                 |
|               |                                               |
|               | registers factory at startup                  |
|               v                                               |
|  +---------------------------------------------------------+  |
|  | contextforge-gateway-rs-cpex                           |  |
|  | Gateway CPEX runtime adapter                           |  |
|  |                                                         |  |
|  |  +-------------------+                                  |  |
|  |  | CPEX PluginManager|                                  |  |
|  |  |                   |                                  |  |
|  |  | - loads Redis cfg |                                  |  |
|  |  | - creates plugin  |                                  |  |
|  |  | - invokes hooks   |                                  |  |
|  |  +---------+---------+                                  |  |
|  +------------|--------------------------------------------+  |
|               |                                               |
|               | create(kind = validator/secrets-detection)    |
|               v                                               |
|  +---------------------------------------------------------+  |
|  | cpex-secrets-detection                                |  |
|  | Rust CPEX plugin crate                                 |  |
|  |                                                         |  |
|  |  SecretsDetectionFactory                               |  |
|  |      creates                                           |  |
|  |  SecretsDetectionCore                                  |  |
|  |      holds config + scanner logic                      |  |
|  |  StageHandler                                          |  |
|  |      handles cmf.tool_pre_invoke                       |  |
|  |      handles cmf.tool_post_invoke                      |  |
|  +---------------------------------------------------------+  |
+---------------------------------------------------------------+

Project Layout And Test Boundaries

contextforge-data-plane/
|
|-- Cargo.toml
|   |-- workspace members
|   |-- workspace dependencies
|   `-- includes crates/plugins/cpex-secrets-detection
|
`-- crates/
    |
    |-- contextforge-gateway-rs/
    |   |
    |   |-- src/main.rs
    |   |   `-- gateway binary entry point
    |   |      registers SecretsDetectionFactory
    |   |
    |   |-- Cargo.toml
    |   |   `-- defines plugins feature
    |   |
    |   `-- tests/
    |       `-- secrets_detection_e2e.rs
    |          `-- E2E test
    |             starts redis-server, backend MCP server,
    |             real gateway binary, then makes MCP tool calls
    |
    |-- contextforge-gateway-rs-cpex/
    |   |
    |   `-- src/
    |      `-- CPEX runtime adapter
    |         maps gateway lifecycle to CPEX hooks:
    |         before_tool_call -> cmf.tool_pre_invoke
    |         after_tool_call  -> cmf.tool_post_invoke
    |
    |-- contextforge-gateway-rs-lib/
    |   |
    |   `-- tests/
    |      `-- gateway_plugins.rs
    |         `-- gateway integration tests
    |            test gateway behavior with plugin runtime wired in
    |            without starting the real binary
    |
    `-- plugins/
        |
        `-- cpex-secrets-detection/
            |
            |-- Cargo.toml
            |   `-- new plugin crate manifest
            |
            |-- README.md
            |   `-- plugin usage/config notes
            |
            |-- src/
            |   |
            |   |-- lib.rs
            |   |   `-- CPEX plugin entry point
            |   |      `-- SecretsDetectionFactory
            |   |      `-- SecretsDetectionCore
            |   |      `-- StageHandler
            |   |
            |   |-- config.rs
            |   |   `-- parses plugin config:
            |   |      `-- redact
            |   |      `-- block_on_detection
            |   |      `-- field_allowlist / field_denylist
            |   |
            |   |-- scanner.rs
            |   |   `-- scans JSON/text payloads
            |   |      `-- applies redaction
            |   |
            |   `-- patterns.rs
            |       `-- secret regex definitions
            |
            `-- tests/
                `-- plugin_manager.rs
                   `-- crate-level CPEX PluginManager tests
                      register factory -> load config -> invoke hook

Validation

CI runs:

cargo nextest run --locked --workspace --all-features
cargo build --locked --workspace --all-features

Local validation run here:

cargo +1.96 test --locked --workspace --all-features
cargo +1.96 build --locked --workspace --all-features

cargo-nextest was not installed locally, so cargo test was used as the local fallback. The binary E2E remains ignored/manual.

Manual Reviewer Verification

The script below drives the real gateway over MCP and uses synthetic secret-shaped values generated at runtime. The expected output is sanitized here and does not include the generated values.

Detailed Manual Steps

Terminal 1: start Redis and one backend MCP server.

cd /Users/pratik/Desktop/work/new_mcf/contextforge-data-plane

docker compose -f docker/docker-compose-local.yaml up -d redis gateway-one
docker compose -f docker/docker-compose-local.yaml ps redis gateway-one

Terminal 3: create the local reviewer script from the collapsible below, then start it.

cd /Users/pratik/Desktop/work/new_mcf/contextforge-data-plane

chmod +x ./secrets_detection_demo.sh
./secrets_detection_demo.sh

The script writes the Redis runtime plugin config and then pauses.

Terminal 2: when the script asks, start or restart the real gateway binary.

cd /Users/pratik/Desktop/work/new_mcf/contextforge-data-plane

cargo +1.96 run -p contextforge-gateway-rs \
  --features "plugins contextforge-gateway-rs-lib/with_tools" \
  --bin contextforge-gateway-rs -- \
  --address 127.0.0.1:8001 \
  --redis-address 127.0.0.1 \
  --redis-port 6379 \
  --redis-mode plain-text \
  --token-verification-public-key assets/jwt.key.pub \
  --token-verification-private-key assets/jwt.key \
  --upstream-connection-mode plain-text-or-tls \
  --runtime-plugins-enabled true \
  --number-of-cpus 1

Return to Terminal 3 and press Enter. The script should run four checks:

• Clean payload no-op.
• cmf.tool_pre_invoke redaction.
• cmf.tool_pre_invoke blocking.
• cmf.tool_post_invoke redaction.

Reviewer Script Contents

Create this as secrets_detection_demo.sh at the repo root. It is a local reviewer helper and is not committed in this PR.

#!/usr/bin/env bash
set -euo pipefail

cd /Users/pratik/Desktop/work/new_mcf/contextforge-data-plane

USER_ID=11111111-1111-1111-1111-111111111111
USER_EMAIL=admin@example.com
VH_ID=c0ffee00f001f00lf00ldeadbeefdead
GATEWAY_URL=${GATEWAY_URL:-http://127.0.0.1:8001/contextforge-rs}
BACKEND_URL=${BACKEND_URL:-http://127.0.0.1:5555/mcp}
REDIS_COMPOSE=(docker compose -f docker/docker-compose-local.yaml exec -T redis redis-cli)

SESSION_ID=""
INIT_HEADERS=""
TOKEN=""

cleanup() {
  if [[ -n "${INIT_HEADERS}" ]]; then
    rm -f "${INIT_HEADERS}"
  fi
}
trap cleanup EXIT

write_runtime_plugin_config() {
  local hooks_json=$1
  local plugin_config_json=$2

  "${REDIS_COMPOSE[@]}" SET \
    ContextForgeGatewayRuntimePluginConfig \
    "{
      \"version\": 1,
      \"cpex\": {
        \"plugins\": [
          {
            \"name\": \"secrets-detection\",
            \"kind\": \"validator/secrets-detection\",
            \"hooks\": ${hooks_json},
            \"config\": ${plugin_config_json}
          }
        ]
      }
    }" >/dev/null
}

wait_for_gateway_start() {
  echo
  echo "Redis plugin config is set for all reviewer smoke cases."
  echo "In the gateway terminal, start or restart the gateway with the cargo run command."
  read -r -p "Press Enter here once the gateway is running..."
  echo
}

configure_route() {
  local token=$1

  curl --silent --show-error --request POST \
    --url "${GATEWAY_URL}/admin/userconfigs/${USER_ID}" \
    --header 'content-type: application/json' \
    --data "{
      \"virtual_hosts\": {
        \"${VH_ID}\": {
          \"backends\": {
            \"gateway-one\": {
              \"name\": \"gateway-one\",
              \"url\": \"${BACKEND_URL}\",
              \"transport\": \"STREAMABLEHTTP\",
              \"passthrough_headers\": [],
              \"allowed_tool_names\": [],
              \"allowed_resource_names\": [],
              \"allowed_prompt_names\": []
            }
          }
        }
      }
    }" >/dev/null
}

open_mcp_session() {
  local token=$1

  if [[ -n "${INIT_HEADERS}" ]]; then
    rm -f "${INIT_HEADERS}"
  fi
  INIT_HEADERS=$(mktemp)

  curl --silent --show-error \
    --dump-header "${INIT_HEADERS}" \
    --url "${GATEWAY_URL}/servers/${VH_ID}/mcp" \
    --header "authorization: Bearer ${token}" \
    --header 'content-type: application/json' \
    --header 'accept: application/json, text/event-stream' \
    --data '{
      "jsonrpc": "2.0",
      "id": 0,
      "method": "initialize",
      "params": {
        "protocolVersion": "2025-11-25",
        "capabilities": {},
        "clientInfo": { "name": "curl", "version": "0.1.0" }
      }
    }' >/dev/null

  SESSION_ID=$(awk 'tolower($1) == "mcp-session-id:" { gsub("\r", "", $2); print $2 }' "${INIT_HEADERS}")

  if [[ -z "${SESSION_ID}" ]]; then
    echo "Failed to read mcp-session-id from initialize response headers" >&2
    exit 1
  fi

  curl --silent --show-error \
    --url "${GATEWAY_URL}/servers/${VH_ID}/mcp" \
    --header "authorization: Bearer ${token}" \
    --header "mcp-session-id: ${SESSION_ID}" \
    --header 'mcp-protocol-version: 2025-11-25' \
    --header 'content-type: application/json' \
    --header 'accept: application/json, text/event-stream' \
    --data '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null
}

prepare_client() {
  TOKEN=$(curl --silent --show-error \
    "${GATEWAY_URL}/admin/tokens/${USER_ID}?email=${USER_EMAIL}")

  configure_route "${TOKEN}"
  open_mcp_session "${TOKEN}"
}

call_mcp() {
  local token=$1
  local request_body=$2

  curl --silent --show-error \
    --url "${GATEWAY_URL}/servers/${VH_ID}/mcp" \
    --header "authorization: Bearer ${token}" \
    --header "mcp-session-id: ${SESSION_ID}" \
    --header 'mcp-protocol-version: 2025-11-25' \
    --header 'content-type: application/json' \
    --header 'accept: application/json, text/event-stream' \
    --data "${request_body}"
}

print_json_data_lines() {
  sed -n '/^data: {/{s/^data: //; p;}'
}

require_contains() {
  local response=$1
  local pattern=$2
  local failure_message=$3

  if ! printf '%s\n' "${response}" | grep -Fq "${pattern}"; then
    echo "      ${failure_message}"
    echo "      Raw response:"
    printf '%s\n' "${response}"
    echo "      Parsed JSON response:"
    printf '%s\n' "${response}" | print_json_data_lines
    exit 1
  fi
}

require_absent() {
  local response=$1
  local pattern=$2
  local failure_message=$3

  if printf '%s\n' "${response}" | grep -Fq "${pattern}"; then
    echo "      ${failure_message}"
    echo "      Raw response:"
    printf '%s\n' "${response}"
    echo "      Parsed JSON response:"
    printf '%s\n' "${response}" | print_json_data_lines
    exit 1
  fi
}

echo_request() {
  local id=$1
  local args_json=$2

  printf '{"jsonrpc":"2.0","id":%s,"method":"tools/call","params":{"name":"echo","arguments":%s}}' \
    "${id}" \
    "${args_json}"
}

run_clean_case() {
  local token=$1
  local request_body response

  request_body=$(echo_request 1 '{"message":"hello-reviewer"}')

  echo "[1/4] Clean payload no-op"
  echo "      Hook coverage: cmf.tool_pre_invoke + cmf.tool_post_invoke"
  echo "      Expected: request succeeds unchanged, no redaction marker"
  echo
  echo "Full request:"
  printf '%s\n' "${request_body}"
  echo

  response=$(call_mcp "${token}" "${request_body}")

  require_contains "${response}" "hello-reviewer" "FAIL: clean response did not contain original message"
  require_absent "${response}" "[redacted]" "FAIL: clean response was unexpectedly redacted"

  echo "      Clean response contains original message: yes"
  echo "      Response contains [redacted]: no"
  echo "      Result: PASS"
  echo
}

run_pre_redaction_case() {
  local token=$1
  local fake_secret request_body response

  fake_secret="AKIA$(printf '1%.0s' {1..16})"
  request_body=$(echo_request 2 "{\"token\":\"${fake_secret}\",\"message\":\"hello\"}")

  echo "[2/4] Tool pre-invoke redaction"
  echo "      Hook: cmf.tool_pre_invoke"
  echo "      Input token: ${fake_secret}"
  echo "      Expected: one fake secret-shaped argument is redacted and not blocked"
  echo
  echo "Full request:"
  printf '%s\n' "${request_body}"
  echo

  response=$(call_mcp "${token}" "${request_body}")

  require_contains "${response}" "[redacted]" "FAIL: redacted marker was missing"
  require_absent "${response}" "${fake_secret}" "FAIL: raw fake secret leaked"

  echo "      Response contains [redacted]: yes"
  echo "      Raw fake secret leaked: no"
  echo "      Redacted response:"
  printf '%s\n' "${response}" | print_json_data_lines
  echo "      Result: PASS"
  echo
}

run_block_case() {
  local token=$1
  local fake_secret request_body response

  fake_secret="AKIA$(printf '2%.0s' {1..16})"
  request_body=$(echo_request 3 "{\"token_one\":\"${fake_secret}\",\"token_two\":\"${fake_secret}\",\"message\":\"please block\"}")

  echo "[3/4] Tool pre-invoke blocking"
  echo "      Hook: cmf.tool_pre_invoke"
  echo "      Input tokens: two fake secret-shaped values"
  echo "      Expected: request is denied before backend echo succeeds"
  echo
  echo "Full request:"
  printf '%s\n' "${request_body}"
  echo

  response=$(call_mcp "${token}" "${request_body}")

  require_contains "${response}" "\"error\"" "FAIL: blocking response did not contain an MCP error"
  require_absent "${response}" "please block" "FAIL: backend echo output appeared in blocked response"

  echo "      Response contains MCP error: yes"
  echo "      Backend echo output present: no"
  echo "      Block response:"
  printf '%s\n' "${response}" | print_json_data_lines
  echo "      Result: PASS"
  echo
}

run_post_redaction_case() {
  local token=$1
  local fake_secret request_body response

  fake_secret="AKIA$(printf '3%.0s' {1..16})"
  request_body=$(echo_request 4 "{\"credential\":\"${fake_secret}\",\"message\":\"post hook only\"}")

  echo "[4/4] Tool post-invoke redaction"
  echo "      Hook: cmf.tool_post_invoke"
  echo "      Input credential: ${fake_secret}"
  echo "      Expected: pre-hook skips credential via field_denylist, post-hook redacts backend response"
  echo
  echo "Full request:"
  printf '%s\n' "${request_body}"
  echo

  response=$(call_mcp "${token}" "${request_body}")

  require_contains "${response}" "[redacted]" "FAIL: post-hook redacted marker was missing"
  require_absent "${response}" "${fake_secret}" "FAIL: raw fake secret leaked from post-hook response"

  echo "      Response contains [redacted]: yes"
  echo "      Raw fake secret leaked: no"
  echo "      Redacted response:"
  printf '%s\n' "${response}" | print_json_data_lines
  echo "      Result: PASS"
  echo
}

echo "== Reviewer smoke test: Rust CPEX secrets detection in dataplane =="
echo
echo "Prerequisites:"
echo "  Redis container: docker/docker-compose-local.yaml redis"
echo "  Backend MCP server: ${BACKEND_URL}"
echo "  Dataplane gateway: ${GATEWAY_URL}"
echo "  Plugin kind: validator/secrets-detection"
echo
echo "This script writes one Redis runtime plugin config that covers all cases:"
echo "  redact=true, block_on_detection=true, min_findings_to_block=2"
echo "  field_denylist=[credential] to isolate the post-hook case"
echo

write_runtime_plugin_config \
  '["cmf.tool_pre_invoke","cmf.tool_post_invoke"]' \
  '{"redact":true,"redaction_text":"[redacted]","block_on_detection":true,"min_findings_to_block":2,"field_denylist":["credential"]}'
wait_for_gateway_start
prepare_client
run_clean_case "${TOKEN}"
run_pre_redaction_case "${TOKEN}"
run_block_case "${TOKEN}"
run_post_redaction_case "${TOKEN}"

echo "Reviewer smoke test result: PASS"

Expected Manual Result

Expected high-level result:

[1/4] Clean payload no-op
      Clean response contains original message: yes
      Response contains [redacted]: no
      Result: PASS

[2/4] Tool pre-invoke redaction
      Response contains [redacted]: yes
      Raw fake secret leaked: no
      Result: PASS

[3/4] Tool pre-invoke blocking
      Response contains MCP error: yes
      Backend echo output present: no
      Result: PASS

[4/4] Tool post-invoke redaction
      Response contains [redacted]: yes
      Raw fake secret leaked: no
      Result: PASS

Reviewer smoke test result: PASS

Sanitized representative responses:

{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"{\"message\":\"hello\",\"token\":\"[redacted]\"}"}],"isError":false}}
{"jsonrpc":"2.0","id":3,"error":{"code":-32600,"message":"Plugin denied tool call"}}
{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"{\"credential\":\"[redacted]\",\"message\":\"post hook only\"}"}],"isError":false}}

Notes

This PR keeps the crate unpublished and feature-gated. Redis runtime config can activate the plugin only after the gateway binary is built with plugins.

The secrets detection binary E2E is intentionally ignored and not run by CI.

[lucarlig]

I would name the crate cpex-secrets-detection because is not part of the data-plane, not sure if we put in /crates/plugins folder as well

[lucarlig]

@gandhipratik203, we need a follow-up PR to remove the two existing test plugins. They are now obsolete and have been completely superseded by this plugin.

[lucarlig]

Please complete the dataplane integration by shipping the feature, covering it in CI, and documenting its activation.

[dawid-nowak]

a bit late here, but yeah, keeping plugins in a different folder under crates sounds like a valid approach.

[lucarlig]

LGTM

[dawid-nowak]

This smells wrong: binary_e2e_redacts_tool_arguments_and_results

1. it requires a binary version of Redis
2. it requires a binary version of contextforge-dataplane ( a bit of chicken and egg problem there)
3. It runs them as separate commands etc.

In the long run as part of the CI this is going to be very slow and error prone. The code is good but it really should be used as part of the integration tests.

For contextforge-dataplane it is more advisable to use tokio and not rely on Command, Command will fork the process which is very expensive.

[lucarlig]

LGTM

[gandhipratik203]

Agreed, binary_e2e_redacts_tool_arguments_and_results is not the right long-term CI shape for the dataplane. I removed it from CI. The plugin behavior is covered through the existing in-process gateway integration tests, which avoid shelling out to Redis or forking the gateway binary.