LFX 2026 Term 3 Simulation Sandbox: analysis by Prachi Agrawal #863
Replies: 3 comments
Task 1: Working Environment ValidationEnvironment
Screen recording - full container run to ranked leaderboard (11m32s): task1_ianvs_pcb_aoi_run.mp4Final leaderboard, second run (four test cases across two independent runs): task1_run3_2026-08-23.1.mp41. Setup - commands executed, in order# Clone
git clone https://github.com/kubeedge/ianvs.git ianvs-pretest
cd ianvs-pretest
git log -1 --format="%H %ad"
# 37a9c60a9747af0cfe3170f84249bc349c56e8d5 Fri Aug 21 17:57:44 2026 +0800
# Virtual environment (--clear guarantees nothing is inherited)
python3 -m venv --clear venv
source venv/bin/activate
pip list # empty - see raw transcript
pip install --upgrade pip
pip install -r requirements.txt
# Successfully installed contourpy-1.3.3 cycler-0.12.1 fonttools-4.63.0 joblib-1.5.3
# kiwisolver-1.5.0 matplotlib-3.11.1 ml_dtypes-0.6.0 narwhals-2.25.0 numpy-2.5.2
# onnx-1.22.0 packaging-26.3 pandas-3.0.5 pillow-12.3.0 prettytable-2.5.0
# protobuf-7.36.0 pyparsing-3.3.2 python-dateutil-2.9.0.post0 scikit-learn-1.9.0
# scipy-1.18.1 six-1.17.0 threadpoolctl-3.6.0 tqdm-4.70.0 typing_extensions-4.16.0
# wcwidth-0.8.2
# (clean install, no conflicts - requirements.txt resolves fine in isolation)
python setup.py installIssue 1 -
|
| # | Issue | Severity | Fix |
|---|---|---|---|
| 1 | setuptools absent from a Python 3.12 venv |
Blocker | pip install setuptools |
| 2 | colorlog imported but not in requirements.txt |
Blocker | pip install colorlog; add to requirements |
| 3 | pyyaml imported but not in requirements.txt |
Blocker | pip install pyyaml; add to requirements |
| 4 | pip install sedna fetches an unrelated PyPI package |
Blocker, silent | Use the vendored wheel |
| 5 | Documented wheel path examples/resources/third_party/* does not exist |
Blocker, ~20 docs + 3 CI files | Correct path is resources/third_party/* |
| 6 | Dead cross-reference for the PCB-AoI dataset doc | Docs gap | Restore or remove the link |
| 7 | pcb-aoi README points at a guide about a different example | Docs gap | Write a pcb-aoi-specific guide |
| 8 | Config paths resolve against CWD, not the YAML's own directory | Blocker if run from the example dir | Rebase on dirname(config_file) |
| 9 | cv2 missing; per-example requirements never installed |
Blocker | pip install opencv-python~=3.4 |
| 10 | onnx==1.9.0 has no cp312 wheel and fails to build |
Blocker on Python 3.12 | Works on cp37 |
| 11 | tensorflow.contrib.slim removed in TF2; no TF1 wheels for Python 3.8+ |
Hard blocker | Python 3.7 container; port to tf-slim upstream |
| 12 | FPN_TensorFlow wheel declares the wrong PyPI libs dependency |
Blocker, silent | Install with --no-deps |
| 13 | onnx==1.9.0 + protobuf ≥ 4.21 crashes every ianvs invocation via eager multiedge_inference import |
Hard blocker, global | Pin protobuf==3.20.3; lazy-import upstream |
| 14 | initial_model_url file ships nowhere; only reachable via a dead orphaned link |
Blocker | Download from the still-live OBS URL |
| 15 | setup.py never passes install_requires; requirements.txt is dead at install time |
Structural | Pass _infos.basic_dependencies |
Task 2: Simulation Codebase Bug IdentificationFile under analysis: §0. Provenance and prior artThis analysis extends work I filed before this pre-test was published:
It also has to be read against the now-merged design proposal Findings below are classified into three tiers so that priority is unambiguous:
Tier A1 additionally corrects the published analysis in #819-B1, PR #656, One correction I made against my own prior work while preparing this §1. Understanding1.1 Structural summaryThe module is 186 lines, imports only Control flow is a two-layer hierarchy: Three design characteristics govern every defect below:
1.2 Role of each function
Note on naming: 1.3 Difference between the proposal and the implementationThe simulation proposal in #348 specifies
The same proposal specifies a Simulation Job Administrator with four duties - Cross-referencing PR #526 - the Note for the record, since it bears directly on how duty 4 should be read: Two further divergences worth recording:
§2. Tier A - new findingsEvery reproduction below was run on my own machine; commands and raw output A1 -
|
| # | Line | Defect | Fix |
|---|---|---|---|
| A9.1 | L177 | Docstring of destory_simulation_enviroment() reads "build the simulation enviroment" - copy-paste from L151 |
"""destroy a simulation environment""" |
| A9.2 | L133 | Log message formats a CPU count with kB units: "Number of Cpus: %s kB, Cpus Require: %s kB" |
Drop kB |
| A9.3 | L135 | Typo in the raised message: "The number os cpus is insufficient." |
"of" |
| A9.4 | L132 | Uses LOGGER.info to report a failure that immediately raises; L101 in the equivalent memory path uses LOGGER.exception |
Use LOGGER.error in both |
| A9.5 | L86, L116–117 | Parsing relies on str(bytes) producing a Python repr: str(b'MemFree: 123 kB\n') yields "b'MemFree: 123 kB\\n'", and L117 then splits on a literal backslash to strip the repr's \n. Correct only by accident |
.decode("utf-8") and split on real whitespace |
The enviroment/destory misspelling across the public API (L23–186) is a real
defect, but I am not counting it here as a new finding - PR #526's design doc
already calls for exactly this rename (see §4), predating both #819 and this
pre-test, so it is listed there instead of here.
§3. Tier B - first reported by me in #819 (2026-08-17)
Reproduce with the verification script from that issue:
git clone https://github.com/kubeedge/ianvs.git && cd ianvs
curl -LO https://raw.githubusercontent.com/Prachi194agrawal/ianvs/b68b5b1/verify_legacy_simulation.py
python3 verify_legacy_simulation.py| ID | Finding | Relation to this file |
|---|---|---|
| B12 | Ianvs accepts cloud_number/edge_number above the Sedna backend limits (MAX_CLOUD_WORKER_NODES=2, MAX_EDGE_WORKER_NODES=3). edge_number: 10 passes validation and fails during provisioning with Only support NUM_EDGE_NODES at most 3 |
Reaches the shell at L160. Should be rejected at parse time, before host checks or provisioning. Checked specifically against PR #526's design doc: no mention of a node-count ceiling anywhere in it |
| B11 | The Sedna installer does not translate aarch64 to the KubeEdge release asset name |
Triggered from L165. Not mentioned in PR #526 either |
| B9 | No shipped example contains a simulation: block |
Means L149–L186 has no end-to-end test coverage in-repo. Not mentioned in PR #526 |
Adjacent modules, recorded in #819 and out of scope for this file:
B13 (no peak-memory / CPU-utilization / wall-time leaderboard metrics - this is the
"System Metrics Profiling" half of the project title, and is covered at a design
level by PR #526's §G2/§4, so I am not claiming it as unclaimed design space, only
noting it is out of scope for simulation_system_admin.py itself) and B14 (one
failing test case aborts the loop before completed results reach the ranking layer).
§4. Tier C - previously reported by others
Included for completeness of the file analysis. Not claimed as novel.
| ID | Lines | Finding | Prior art | Does my solution differ? |
|---|---|---|---|---|
| B1 | L58, L60–74 | check=True makes the kind fallback unreachable |
PR #656; independently listed in PR #526 §2 | Yes, materially - see A1. The stated mechanism holds for check_host_kind() only. For check_host_docker() the cause is pipeline exit-status masking and check=True never fires at all - a point none of PR #656, #819-B1, or PR #526 makes. The same dead-branch pattern also exists at L168–172, which none of the three cover (A5) |
| B4 | L163, L182 (Simulation-level check in simulation.py) |
Empty required values, including cluster_name, pass type validation |
PR #526 design doc (§2, opened 2026-06-09 - two months before my own #819 report, which I am correcting here) | Not materially - I additionally show the injection-bearing variant (non-empty, well-typed, metacharacter-laden values) reaching a shell=True sink; that is A3, not B4, and is not covered by PR #526 or any open PR |
| B5 | L113–117 | lscpu | grep CPU: fails on current util-linux output |
PR #656; PR #526 (fix: os.cpu_count()) |
Partly - both #656 and PR #526 fix the parse with os.cpu_count(). I would additionally use os.sched_getaffinity(0), which is cgroup- and affinity-aware and therefore correct inside a container, where both lscpu and plain os.cpu_count() report the host's cores rather than the container's quota |
| B6 | L157 vs L180 | Sedna branch master vs main |
PR #656; independently noted in PR #526 §2 | Yes - see A8. Making both main resolves the inconsistency but leaves both URLs unpinned and unverified; neither prior source raises that |
| B7 | L175 | Cleanup function exported but never called | PR #745 | Yes - see A4 (return contract) and A5 (partial-failure path). Both survive #745 |
| B8 | L64–66 | kind installer pinned to an old amd64-only binary | PR #777 (in part); independently noted in PR #526 §2 (fix: update to v0.32.0) | No material difference |
| A9.6 | L23–186 | enviroment/destory misspelled throughout the public API |
PR #526 design doc, Stage-1 restoration table ("Rename to build_simulation_environment() … same pattern as destory typo") |
No material difference beyond the deprecating-alias detail in my proposed fix |
| B2, B3 | - | YAML booleans pass node-count type checks; unknown keys silently ignored | PR #698; independently noted in PR #526 §2 | Not in this file (parser). Listed for completeness |
5. Summary
| Tier | Count | Items |
|---|---|---|
| A - new in this submission | 8 + 5 minor | A1 (critical, corrects prior art), A2, A3 (critical), A4, A5, A6, A7, A8, A9.1–A9.5 |
| B - mine, from #819 (2026-08-17) | 3 in-file | B12, B11, B9 |
| C - others' prior art | 8 (7 in-file + B2/B3 out of file) | B1, B4, B5, B6, B7, B8, A9.6, (B2/B3 out of file) |
B4 moved from Tier B to Tier C in this submission after checking it against PR #526,
which predates #819 by two months - see §0.
The two findings I would most want acted on:
A1, because it is the difference between a prerequisite check that fails loudly and
one that reports success on a host that cannot run a simulation - and because three
independent published analyses of it (my own #819, PR #656, and PR #526) are each
incorrect in the same way, in a way that a fix based on any of them will not repair.
A3, because a benchmarking framework whose configuration files reach a shell=True
sink unvalidated cannot safely accept community-contributed benchmarks, which is
precisely what Ianvs exists to do.
Environment for all reproductions in this comment
- OS: Ubuntu 24.04.3 LTS (Noble), kernel 7.0.0-29-generic, x86_64
- Python: 3.12.3
- Docker: 29.1.3, daemon running (used for the A1 control case, showing the masking
bug is present even when Docker is installed and healthy) - CPU: 12 logical cores, 19.23 GiB RAM (
MemTotal), 8.30 GiBMemFree/
12.73 GiBMemAvailableat time of the A2 run - Ianvs commit analyzed:
37a9c60a9747af0cfe3170f84249bc349c56e8d5
(upstream/main, fetched 2026-08-23 - matches the commit named at the top of
this comment)
Task 3: Container Isolation Capability DemonstrationThis is a standalone prototype of the Simulation Job Administrator duty 1. LaunchThe script accepts its inputs either as CLI flags or as a config dict passed
It launches the container via Screenshot - success run (case 1 below): Video - full session: Screencast.from.2026-08-24.10-49-43.webm2. Handle OOMIf the exit code is exactly Screenshots - OOM and timeout runs (cases 2 and 4 below): (Same recording as the Launch section covers this - no separate video 3. READMEPurposeGive the Ianvs Simulation Job Administrator a caller that can run one Design decisions
Usagepython3 container_sandbox.py --image <image> --command "<cmd>" \
[--cpu-limit 1.0] [--memory-limit 512m] [--timeout 60]Pre-pull the image once ( Success: python3 container_sandbox.py --image python:3.7-slim \
--command "python3 -c \"print('hello from sandbox')\"" \
--cpu-limit 1.0 --memory-limit 512m --timeout 60OOM (allocates 400 MiB under a 100 MiB cap): python3 container_sandbox.py --image python:3.7-slim \
--command "python3 -c \"x = bytearray(400*1024*1024); print('should not get here')\"" \
--cpu-limit 1.0 --memory-limit 100m --timeout 60Non-OOM failure: python3 container_sandbox.py --image python:3.7-slim \
--command "python3 -c \"import sys; print('about to fail', file=sys.stderr); sys.exit(1)\"" \
--cpu-limit 1.0 --memory-limit 512m --timeout 60Timeout: python3 container_sandbox.py --image python:3.7-slim \
--command "sleep 30" --cpu-limit 1.0 --memory-limit 128m --timeout 3As a library call with a config dict: from container_sandbox import run_in_sandbox
result = run_in_sandbox(config={
"image": "python:3.7-slim",
"command": "python3 -c 'print(1)'",
"cpu_limit": "1.0",
"memory_limit": "512m",
"timeout": 60,
})
run_in_sandbox(command="echo hi")
# ValueError: image is required, either as an argument or in config['image']Structured output format# status == "success"
{"status": "success", "exit_code": 0, "wall_time_s": <float>,
"stdout": <str>, "stderr": <str>}
# status == "oom_killed"
{"status": "oom_killed", "exit_code": 137, "wall_time_s": <float>,
"stdout": <str>,
"error": "Container killed due to OOM (exit code 137). Consider increasing memory_limit."}
# status == "failed" (any other non-zero exit)
{"status": "failed", "exit_code": <int>, "wall_time_s": <float>,
"stdout": <str>, "stderr": <str>,
"error": "Container exited with non-zero code <int>."}
# status == "timeout"
{"status": "timeout", "exit_code": None, "wall_time_s": <float>,
"stdout": "", "error": "Container exceeded timeout of <n>s and was killed."}Complete source code#!/usr/bin/env python3
# Copyright 2026 The KubeEdge Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
container_sandbox.py - isolated Docker execution prototype for Ianvs'
Simulation Job Administrator.
Launches one algorithm command inside a resource-bounded, disposable Docker
container; measures wall-clock time; captures stdout/stderr and the exit
code; and returns a structured result dict so the caller can distinguish a
clean run from an OOM kill from any other failure, instead of raising.
Usage::
python3 container_sandbox.py --image python:3.7-slim \\
--command "python3 -c 'print(1)'" \\
--cpu-limit 1.0 --memory-limit 512m --timeout 60
Can also be imported and called as a library::
from container_sandbox import run_in_sandbox
result = run_in_sandbox(config={
"image": "python:3.7-slim",
"command": "python3 -c 'print(1)'",
"cpu_limit": "1.0",
"memory_limit": "512m",
"timeout": 60,
})
"""
import argparse
import json
import subprocess
import sys
import time
import uuid
OOM_EXIT_CODE = 137
def run_in_sandbox(image=None, command=None, cpu_limit=None, memory_limit=None,
timeout=None, config=None):
"""Run ``command`` inside a throwaway container of ``image``.
Accepts either explicit keyword arguments or a single ``config`` dict
(per the task spec's "CLI arguments or a config dict" requirement) - a
dict takes precedence over any keyword arguments passed alongside it.
``image`` has no default of its own for either calling style - a config
dict without one is a caller error, not a silent no-op.
Returns a structured dict; never raises for a container-side failure.
Docker itself not being reachable is the one case treated as a hard
error, since no container ever started.
"""
if config is not None:
image = config.get("image", image)
command = config.get("command", command)
cpu_limit = config.get("cpu_limit", cpu_limit)
memory_limit = config.get("memory_limit", memory_limit)
timeout = config.get("timeout", timeout)
if not image:
raise ValueError("image is required, either as an argument or in config['image']")
container_name = f"ianvs-sandbox-{uuid.uuid4().hex[:8]}"
docker_cmd = ["docker", "run", "--rm", "--name", container_name]
if cpu_limit:
docker_cmd += ["--cpus", str(cpu_limit)]
if memory_limit:
# --memory-swap defaults to 2x --memory when left unset, which lets a
# workload swap-thrash up to double the declared cap instead of being
# killed at it. Pinning swap equal to the memory limit makes the cap
# behave the way a caller reading "memory_limit" would expect.
docker_cmd += ["--memory", str(memory_limit), "--memory-swap", str(memory_limit)]
docker_cmd.append(image)
if command:
docker_cmd += ["sh", "-c", command]
start = time.monotonic()
try:
proc = subprocess.run(
docker_cmd,
capture_output=True,
text=True,
timeout=timeout,
check=False,
)
wall_time_s = round(time.monotonic() - start, 3)
exit_code = proc.returncode
stdout, stderr = proc.stdout, proc.stderr
except subprocess.TimeoutExpired:
wall_time_s = round(time.monotonic() - start, 3)
# subprocess's timeout only kills the local `docker` CLI process; the
# daemon-side container keeps running (--rm's AutoRemove fires once
# the container itself stops, but nothing has told it to stop yet),
# so it must be killed explicitly or it runs to completion unbounded.
subprocess.run(["docker", "kill", container_name],
capture_output=True, text=True, check=False)
return {
"status": "timeout",
"exit_code": None,
"wall_time_s": wall_time_s,
"stdout": "",
"error": f"Container exceeded timeout of {timeout}s and was killed.",
}
except FileNotFoundError as err:
return {
"status": "error",
"exit_code": None,
"wall_time_s": round(time.monotonic() - start, 3),
"stdout": "",
"error": f"docker executable not found: {err}",
}
if exit_code == OOM_EXIT_CODE:
return {
"status": "oom_killed",
"exit_code": OOM_EXIT_CODE,
"wall_time_s": wall_time_s,
"stdout": stdout,
"error": (
"Container killed due to OOM (exit code 137). "
"Consider increasing memory_limit."
),
}
if exit_code != 0:
return {
"status": "failed",
"exit_code": exit_code,
"wall_time_s": wall_time_s,
"stdout": stdout,
"stderr": stderr,
"error": f"Container exited with non-zero code {exit_code}.",
}
return {
"status": "success",
"exit_code": 0,
"wall_time_s": wall_time_s,
"stdout": stdout,
"stderr": stderr,
}
def main():
parser = argparse.ArgumentParser(
description="Run a command inside an isolated, resource-bounded Docker container."
)
parser.add_argument("--image", required=True, help="Docker image name/tag")
parser.add_argument("--command", default=None, help="algorithm command to run inside the container")
parser.add_argument("--cpu-limit", default=None, help='e.g. "1.0"')
parser.add_argument("--memory-limit", default=None, help='e.g. "512m"')
parser.add_argument("--timeout", type=float, default=None, help="wall-clock seconds")
args = parser.parse_args()
result = run_in_sandbox(
image=args.image,
command=args.command,
cpu_limit=args.cpu_limit,
memory_limit=args.memory_limit,
timeout=args.timeout,
)
print(json.dumps(result, indent=2))
sys.exit(0 if result["status"] == "success" else 1)
if __name__ == "__main__":
main()Sample outputAll six runs below are real, captured on my own machine (Ubuntu 24.04.3 1) Success 2) OOM kill (allocate 400 MiB under a 100 MiB cap; Note 3) Non-OOM failure 4) Timeout 5) Config-dict library call 6) Missing Environment: Ubuntu 24.04.3 LTS, kernel 7.0.0-29-generic, x86_64, Docker |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Pre-test submission for LFX Mentorship 2026 Term 3 - CNCF / KubeEdge (Ianvs)
Project: Simulation Sandbox: Environment-Isolated Execution and System Metrics Profiling
Scope
This discussion contains my complete pre-test submission. Each task is posted as a
separate comment in this thread, per the submission requirements:
Comment 1 - Task 1: Working Environment Validation
Ianvs installation, dependency resolution, and end-to-end execution of the
pcb-aoi/singletask_learning_bench/fault_detectionbenchmark.Comment 2 - Task 2: Simulation Codebase Bug Identification
Analysis of
core/testcasecontroller/simulation_system_admin/simulation_system_admin.py,function-by-function understanding, proposal-vs-implementation gaps, and identified
bugs with line numbers, impact analysis, and proposed fixes.
Comment 3 - Task 3: Container Isolation Capability Demonstration
A runnable prototype that launches an algorithm in an isolated Docker container with
CPU/memory quotas, captures wall-clock time, stdout/stderr and exit code, and handles
OOM (exit code 137) by returning a structured error dictionary.
Prior work on the simulation subsystem
I have been working on the Ianvs simulation code since before this pre-test was
published. The following are filed under my account and predate this discussion:
core/testcasecontroller/simulation): 13 verified breakages in the 2022 implementationkind/design,kind/feature,size/XXL)Where my Task 2 analysis overlaps with #819, I state this explicitly and cite the
original filing date, so that priority is unambiguous. Findings new to this pre-test
are marked as such.
Other contributions to Ianvs
Cloud_Robotics/Semantic_Segmentationis missing its vendoredRFNet/source tree, unlike 4 sibling examplesrequirement.txthas invalid pip syntax (dashscope=1.24.5) and aborts the entire installperception_reasoning.yamluse the wrong directory namebenchmarkingjob.yamlin perception-reasoning references a non-existent directory (hyphen vs underscore)cloud-edge-collaborative-inference-for-llmbenchmarkAll reactions