evaluator` exec_module operator-supplied code in-process; `visualizer` serves checkpoint data with no auth

[EvolveAegis]

Repo: algorithmicsuperintelligence/openevolve at commit 411fb59c886c18704caaffb611e17cf9e7d824d2.

These are two separate issues. (1) is operator/task-pack code-execution hardening (by design, with no sandbox/opt-out); (2) is loopback-only unauth read of evolution output plus a debug=True server. Neither is remotely exploitable. I'm reporting them together only because they live in the same repo.

1. Evaluator runs the evaluator file (and the example evaluators run the candidate) in-process, inheriting the caller's env

What

openevolve/evaluator.py:85 runs spec.loader.exec_module(module) on the file passed as the positional evaluation_file argument (declared at openevolve/cli.py:24-26, wired into Evaluator(evaluation_file=...) at openevolve/evaluator.py:43) during Evaluator.__init__:

openevolve/evaluator.py:85        spec.loader.exec_module(module)

The same call sits on the cascade path at openevolve/evaluator.py:385. The shipped example evaluators exec the candidate program the same way, e.g. examples/function_minimization/evaluator.py:65:

examples/function_minimization/evaluator.py:65    spec.loader.exec_module(program)

Both run in-process, so the loaded code inherits the caller's environment (OPENAI_API_KEY, network, filesystem). In the shipped example evaluator the candidate program is exec'd the same way, so its import-time side effects run unsandboxed alongside scoring.

Reproduce

# Against openevolve @ 411fb59c886c18704caaffb611e17cf9e7d824d2
git checkout 411fb59c886c18704caaffb611e17cf9e7d824d2
pip install -e .
python - <<'PY'
import secrets, os
from openevolve.config import EvaluatorConfig
from openevolve.evaluator import Evaluator
nonce = secrets.token_hex(6)
canary_py = f"/tmp/eval_canary_{nonce}.py"
fired = f"/tmp/eval_fired_{nonce}"
open(canary_py, "w").write(
    "open('%s','w').write('ran')\n"
    "def evaluate(path):\n    return {'score': 1.0}\n" % fired)
Evaluator(EvaluatorConfig(), evaluation_file=canary_py)
print("hit:", os.path.exists(fired))
PY

Observed: hit: True. The module's top-level code ran during Evaluator.__init__, before any candidate is scored.

Impact / scope

This is the design — openevolve exec's the evaluator file you point it at, and the shipped example exec's the machine-generated candidate. This is operator/task-pack hardening, not a gate bypass or an attacker-steered self-evolution path: the demonstrated sink is operator-supplied code (the evaluator file you select, and via the example pattern the candidate). The concrete risk is that a third-party task pack (an evaluator.py from an untrusted repo) runs in-process with your keys, and the example pattern exec's the candidate unsandboxed. Running your own evaluators on your own box is expected; treat third-party task packs as untrusted code. Operator-scoped, no remote ingress.

Suggested change

Consider adding an opt-in subprocess/container mode so third-party task packs don't inherit API keys. Beyond that, document that evaluator files (and the example evaluators) run in-process with the caller's env.

2. Visualizer serves the full checkpoint (every evolved program's source) with no auth, and boots debug=True

What

scripts/visualizer.py:112 serves /api/data with no auth and returns the full checkpoint:

scripts/visualizer.py:112    @app.route("/api/data")

scripts/visualizer.py:127 dumps one program (source + prompts) at /program/<program_id>:

scripts/visualizer.py:127    @app.route("/program/<program_id>")

scripts/visualizer.py:234 starts the server with debug=True:

scripts/visualizer.py:234    app.run(host=args.host, port=args.port, debug=True)

Reproduce

# Against openevolve @ 411fb59c886c18704caaffb611e17cf9e7d824d2
git checkout 411fb59c886c18704caaffb611e17cf9e7d824d2
pip install -e .   # Flask + the visualizer deps
CKPT=/tmp/evolve_ckpt/checkpoint_demo
UUID=$(python3 -c "import uuid; print(uuid.uuid4())")
TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(6))")
mkdir -p "$CKPT/programs"
python3 -c "import json; json.dump({'islands': [['$UUID']], 'archive': ['$UUID'], 'best_program_id': '$UUID'}, open('$CKPT/metadata.json','w'))"
python3 -c "import json; json.dump({'id':'$UUID','code':'SECRET = \"$TOKEN\"\ndef run_search():\n    return 1','metrics':{'score':1.0},'metadata':{},'parent_id':None}, open('$CKPT/programs/$UUID.json','w'))"
python scripts/visualizer.py --path "$CKPT" --port 8765 --host 127.0.0.1 &
sleep 2
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8765/api/data
curl -s http://127.0.0.1:8765/api/data | grep -o "$TOKEN"
curl -s -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:8765/program/$UUID"
curl -s "http://127.0.0.1:8765/program/$UUID" | grep -o "$TOKEN"

Pass --path without a trailing slash — find_latest_checkpoint (scripts/visualizer.py:22) does os.path.basename(base_folder).startswith("checkpoint_"), and basename(".../checkpoint_100/") is "", so a trailing slash makes /api/data return empty nodes.

Observed (openevolve @ 411fb59): /api/data → 200, body contains the planted SECRET token; /program/<id> → 200, HTML body contains the planted token. No auth header, cookie, or token sent. A populated checkpoint is required — if find_latest_checkpoint finds no checkpoint_* folder, /api/data returns {"archive": [], "nodes": [], "edges": [], "checkpoint_dir": ""}. debug=True also surfaces the PIN-gated Werkzeug debugger.

Impact / scope

The data is self-owned evolution output and the default bind is loopback, so this is local-only exposure. Network reach needs an explicit --host 0.0.0.0; the debugger is PIN-gated. This is unauth data exposure plus debug-server hygiene, not a remote RCE.

Suggested change

Bind 127.0.0.1 by default, refuse --host 0.0.0.0 without an explicit flag, drop debug=True behind a local-only flag, and require a token on /api/data and /program/<program_id> when remote-bound. Also strip a trailing slash on --path so find_latest_checkpoint detects the checkpoint. Happy to open a PR.