Conversation
ycycse
changed the title
There was a problem hiding this comment.
Pull request overview
This PR extends SimpleTES to better support non-Python tasks (notably qubit-routing) by making EVOLVE-BLOCK marker handling and prompt code-fencing language-aware, and by propagating the init program’s file extension through evaluation/checkpoint artifacts.
Changes:
- Preserve and reuse the exact EVOLVE-BLOCK marker lines from the init program, and extract the evolved block preferentially from fenced code.
- Make generation prompts/inspirations use a language-specific code fence tag derived from
init_program’s extension. - Use the init program’s extension for evaluator temp filenames and checkpoint
best_program.*, and add qubit-routing dataset requirements/docs updates.
Reviewed changes
Copilot reviewed 9 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
simpletes/utils/code_extract.py |
Tracks exact marker lines and updates extraction to prefer fenced code + exact marker matching. |
simpletes/templates/generation.py |
Parameterizes code fence tags and language name in prompt templates. |
simpletes/generator.py |
Resolves language/code-fence tag from init_program suffix and feeds it into prompts/inspirations. |
simpletes/evaluator.py |
Allows configurable target filename for evaluated program and registers evaluator module in sys.modules. |
simpletes/engine/core.py |
Adds qubit-routing slot namespace env/cleanup and passes suffix-derived target filename to evaluator worker. |
simpletes/engine/checkpoint.py |
Writes best_program using the init program’s extension. |
datasets/qubit_routing/requirements.txt |
Adds task-local requirements for qubit routing. |
datasets/qubit_routing/README.md |
Updates scoring description. |
pyproject.toml |
Adds Qiskit deps to core dependencies and sets pytest testpaths. |
uv.lock |
Lockfile updates reflecting dependency changes. |
.gitignore |
Adds standard ignores and task output/cache paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for i, line in enumerate(lines): | ||
| if _EVOLVE_BLOCK_START in line: | ||
| line_clean = line.rstrip("\r\n") | ||
| if "EVOLVE-BLOCK-START" in line: | ||
| start_idx = i | ||
| elif _EVOLVE_BLOCK_END in line and start_idx >= 0: | ||
| start_marker_line = line_clean | ||
| elif "EVOLVE-BLOCK-END" in line: | ||
| end_idx = i | ||
| end_marker_line = line_clean | ||
| break |
There was a problem hiding this comment.
from_program() now treats any line containing "EVOLVE-BLOCK-END" as the end marker even if no start marker has been seen yet, and breaks immediately. This can cause has_markers=False (and later extraction failures) when the string appears earlier in the file (e.g., in a docstring or comment) before the real start marker. Consider only accepting the end marker after a start marker has been found (e.g., guard with start_idx >= 0), mirroring the previous behavior.
| dependencies = [ | ||
| "cython>=3.0.0", | ||
| "litellm>=1.80.0", | ||
| "matplotlib>=3.10.8", | ||
| "numpy>=2.2.6", | ||
| "pandas>=3.0.0", | ||
| "protobuf>=6.33.5", | ||
| "psutil>=7.2.2", | ||
| "qiskit==1.2.0", | ||
| "qiskit-qasm3-import==0.6.0", | ||
| "questionary>=2.0.0", |
There was a problem hiding this comment.
qiskit and qiskit-qasm3-import were added to the project’s core dependencies, but other datasets appear to rely on task-local datasets/**/requirements.txt instead of expanding the base install (e.g., tasks requiring cvxpy, scanpy, etc.). Making Qiskit a mandatory dependency will significantly increase install size/time for users who aren’t running the qubit-routing dataset. Consider moving these into an optional extra (e.g. qubit_routing = [...]) and/or relying solely on datasets/qubit_routing/requirements.txt for the evaluator venv.
| "qiskit==1.2.0", | ||
| "qiskit-qasm3-import==0.6.0", |
There was a problem hiding this comment.
It should not be put in a global denpendecny requirement.
|
|
||
| [project.optional-dependencies] | ||
| vllm = ["vllm>=0.5.0"] | ||
|
|
There was a problem hiding this comment.
you can add your denpency here:
| qubit_routing = ["qiskit==1.2.0", "qiskit-qasm3-import==0.6.0"] |
| if self._qubit_routing_slot_mode: | ||
| # Task-local hint consumed by qubit-routing evaluator for slot count fallback. | ||
| os.environ.setdefault("QUBIT_ROUTING_SLOT_COUNT", str(config.eval_concurrency)) | ||
| # Total per-evaluation timeout from SimpleEvolve evaluator worker. |
There was a problem hiding this comment.
| # Total per-evaluation timeout from SimpleEvolve evaluator worker. | |
| # Total per-evaluation timeout from SimpleTES evaluator worker. |
No description provided.