feat(scene-engine): add semantic scene generation and editing - #532
Open
skywhite1024 wants to merge 1 commit into
Open
feat(scene-engine): add semantic scene generation and editing#532skywhite1024 wants to merge 1 commit into
skywhite1024 wants to merge 1 commit into
Conversation
skywhite1024
changed the title
4 tasks
skywhite1024
added
assets
13 tasks
Greptile SummaryThe PR refactors Scene Engine generation into auditable analysis/materialization stages and introduces text-driven scene editing, new service clients, scene graph persistence, and graph-constrained layout processing.
Confidence Score: 1/5The PR is not safe to merge until unsafe edit IDs, unnecessary service dependencies, and repeated export-frame rotation are corrected. LLM-derived categories can escape the edit workspace through generated paths, move/delete edits fail when unrelated generation services are unavailable, and repeated edit/export cycles transform unchanged scene coordinates again. Files Needing Attention: embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_understanding.py, embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_asset_preparation.py, embodichain/gen_sim/scene_engine/pipeline/api.py, embodichain/gen_sim/scene_engine/pipeline/utils/scene_exporter.py, embodichain/gen_sim/scene_engine/pipeline/utils/scene_importer.py
|
| Filename | Overview |
|---|---|
| embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_understanding.py | Introduces strict edit-plan parsing, but leaves LLM-generated categories unrestricted before deriving filesystem-facing object IDs. |
| embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_asset_preparation.py | Implements added-asset generation but uses generated object IDs directly in image and mask paths. |
| embodichain/gen_sim/scene_engine/pipeline/api.py | Adds public analysis/materialization boundaries; edit materialization unnecessarily gates no-add edits on all generation services and reuses the rotating exporter. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_exporter.py | Persists scene and graph artifacts with a default 180-degree transform that is not inverted by the new importer. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_importer.py | Reconstructs exported scenes but retains the additional export-frame rotation, making repeated edit/export cycles inconsistent. |
| embodichain/gen_sim/scene_engine/clients/image_generation.py | Adds prompt-based PNG generation with retries and seed acknowledgement, but omits the required explicit public export list. |
Sequence Diagram
sequenceDiagram
participant CLI
participant Analyze as analyze_edit
participant Importer as SceneExportImporter
participant VLM
participant Materialize as materialize_edit
participant Services as Generation Services
participant Layout
participant Exporter as SceneExporter
CLI->>Analyze: output_root + edit_prompt
Analyze->>Importer: import existing scene and graph
Analyze->>VLM: derive add/move/delete plan
VLM-->>Analyze: validated SceneEditPlan
CLI->>Materialize: edit blueprint
Materialize->>Services: initialize and health-check
Materialize->>Services: generate assets for add operations
Materialize->>Layout: apply graph-constrained layout
Materialize->>Exporter: export revised scene and graph
Prompt To Fix All With AI
### Issue 1
embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_understanding.py:409-410
**Unsafe edit IDs escape workspace**
When an add operation returns a category containing path separators or traversal components, the parser embeds it directly in `object_id`, which is then resolved as generated image and mask paths, causing writes outside the scene-edit workspace with the process's filesystem permissions.
**How this was verified:** The category is accepted as any non-empty string, embedded in `object_id`, and passed to resolved write paths without a containment check.
### Issue 2
embodichain/gen_sim/scene_engine/pipeline/api.py:255-266
**Unused services gate all edits**
When a move-only or delete-only edit runs while any asset-generation service is unconfigured or unavailable, `materialize_edit` still constructs and health-checks all three clients before reaching the no-add early return, causing a valid edit to fail before layout.
### Issue 3
embodichain/gen_sim/scene_engine/pipeline/api.py:304-308
**Edit exports rotate coordinates again**
When an existing export is edited, the importer retains coordinates in the additional 180-degree export frame and this call constructs `SceneExporter` with the same rotation enabled again, causing unchanged positions and XY metadata to flip between frames across edit cycles.
### Issue 4
embodichain/gen_sim/scene_engine/clients/image_generation.py:26-31
**Public module lacks export declaration**
The new public client module defines `ImageGenerationClient` without `__all__`, leaving its intended API implicit and exposing helper symbols inconsistently with the repository's public-module convention.
```suggestion
from embodichain.gen_sim.scene_engine.configs.environment import (
read_scene_engine_env_values,
)
__all__ = ["ImageGenerationClient"]
class ImageGenerationClient:
```
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(scene-engine): add semantic scene g..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+409
to
+410
| object_id = _optional_string(value.get("object_id"), field_name="object_id") | ||
| category = _optional_string(value.get("category"), field_name="category") |
There was a problem hiding this comment.
When an add operation returns a category containing path separators or traversal components, the parser embeds it directly in object_id, which is then resolved as generated image and mask paths, causing writes outside the scene-edit workspace with the process's filesystem permissions.
How this was verified: The category is accepted as any non-empty string, embedded in object_id, and passed to resolved write paths without a containment check.
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_understanding.py
Line: 409-410
Comment:
**Unsafe edit IDs escape workspace**
When an add operation returns a category containing path separators or traversal components, the parser embeds it directly in `object_id`, which is then resolved as generated image and mask paths, causing writes outside the scene-edit workspace with the process's filesystem permissions.
**How this was verified:** The category is accepted as any non-empty string, embedded in `object_id`, and passed to resolved write paths without a containment check.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+255
to
+266
| image_generation = image_generation_client or ImageGenerationClient.from_dotenv() | ||
| geometry = geometry_generation_client or GeometryGenerationClient.from_dotenv() | ||
| segmentation = image_segmentation_client or ImageSegmentationClient.from_dotenv() | ||
| owned_clients = ( | ||
| (image_generation, image_generation_client is None), | ||
| (geometry, geometry_generation_client is None), | ||
| (segmentation, image_segmentation_client is None), | ||
| ) | ||
| log_info("Starting Objects Preparation") | ||
| try: | ||
| for client, _ in owned_clients: | ||
| client.check_health() |
There was a problem hiding this comment.
When a move-only or delete-only edit runs while any asset-generation service is unconfigured or unavailable, materialize_edit still constructs and health-checks all three clients before reaching the no-add early return, causing a valid edit to fail before layout.
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/scene_engine/pipeline/api.py
Line: 255-266
Comment:
**Unused services gate all edits**
When a move-only or delete-only edit runs while any asset-generation service is unconfigured or unavailable, `materialize_edit` still constructs and health-checks all three clients before reaching the no-add early return, causing a valid edit to fail before layout.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+304
to
+308
| scene_config_path = SceneExporter( | ||
| scene=scene, | ||
| scene_graph=scene_graph, | ||
| output_root=output_root, | ||
| ).export() |
There was a problem hiding this comment.
When an existing export is edited, the importer retains coordinates in the additional 180-degree export frame and this call constructs SceneExporter with the same rotation enabled again, causing unchanged positions and XY metadata to flip between frames across edit cycles.
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/scene_engine/pipeline/api.py
Line: 304-308
Comment:
**Edit exports rotate coordinates again**
When an existing export is edited, the importer retains coordinates in the additional 180-degree export frame and this call constructs `SceneExporter` with the same rotation enabled again, causing unchanged positions and XY metadata to flip between frames across edit cycles.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+26
to
+31
| from embodichain.gen_sim.scene_engine.configs.environment import ( | ||
| read_scene_engine_env_values, | ||
| ) | ||
|
|
||
|
|
||
| class ImageGenerationClient: |
There was a problem hiding this comment.
The new public client module defines ImageGenerationClient without __all__, leaving its intended API implicit and exposing helper symbols inconsistently with the repository's public-module convention.
Suggested change
| from embodichain.gen_sim.scene_engine.configs.environment import ( | |
| read_scene_engine_env_values, | |
| ) | |
| class ImageGenerationClient: | |
| from embodichain.gen_sim.scene_engine.configs.environment import ( | |
| read_scene_engine_env_values, | |
| ) | |
| __all__ = ["ImageGenerationClient"] | |
| class ImageGenerationClient: |
Context Used: CLAUDE.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/scene_engine/clients/image_generation.py
Line: 26-31
Comment:
**Public module lacks export declaration**
The new public client module defines `ImageGenerationClient` without `__all__`, leaving its intended API implicit and exposing helper symbols inconsistently with the repository's public-module convention.
```suggestion
from embodichain.gen_sim.scene_engine.configs.environment import (
read_scene_engine_env_values,
)
__all__ = ["ImageGenerationClient"]
class ImageGenerationClient:
```
**Context Used:** CLAUDE.md ([source](https://github.com/dexforce/embodichain/blob/main/CLAUDE.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the GenSim Scene Engine pipeline into auditable “blueprint” stage boundaries, adds first-class scene-graph + edit flows, and expands export/import + layout utilities to support deterministic generation and iterative editing.
Changes:
- Introduces
pipeline.apiwithanalyze_*(blueprint capture) andmaterialize_*(generation/materialization) entry points, and updates CLI/generate/edit entry points to use them. - Adds/extends core data structures and utilities:
SceneGraph,SceneEditPlan, export/import of scene+graph, table support-surface metadata, and new layout construction helpers. - Adds new service clients (image generation) and updates existing clients (segmentation endpoint naming, geometry seed handling), with broad unit test coverage.
Reviewed changes
Copilot reviewed 43 out of 43 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gen_sim/scene_engine/test_simready_processor_utils.py | Adds unit coverage for VLM-driven SimReady scaling behavior. |
| tests/gen_sim/scene_engine/test_scene_understanding.py | Updates/extends scene understanding tests (mask overlays, orientation states, parsing rules). |
| tests/gen_sim/scene_engine/test_scene_layout_optimizer.py | Adds tests for table-region bounds and stacked placement behavior. |
| tests/gen_sim/scene_engine/test_scene_graph.py | Adds validation/serialization/constraint derivation tests for scene graphs. |
| tests/gen_sim/scene_engine/test_scene_generation.py | Adds calibration tests for scene-graph-conditioned orientation handling. |
| tests/gen_sim/scene_engine/test_scene_engine_config.py | Extends CLI behavior tests (generate vs edit vs both). |
| tests/gen_sim/scene_engine/test_scene_edit.py | Adds import/edit validation tests for existing exports. |
| tests/gen_sim/scene_engine/test_scene_edit_plan.py | Adds comprehensive tests for edit-plan parsing, validation, asset prep, and graph updates. |
| tests/gen_sim/scene_engine/test_scene_core_and_export.py | Expands export/import tests (graph export, global z-rotation, support metadata). |
| tests/gen_sim/scene_engine/test_pipeline_api.py | Adds end-to-end tests for blueprint persistence and immutability during materialization. |
| tests/gen_sim/scene_engine/test_clients.py | Updates client tests (segmentation endpoint rename, new image generation client, seed propagation). |
| embodichain/gen_sim/scene_engine/pipeline/utils/table_support_surface.py | Adds optimization rectangle computation + debug visualization for support regions. |
| embodichain/gen_sim/scene_engine/pipeline/utils/simready_processor.py | Refactors SimReady processing to support SceneGraph-conditioned transforms and table support-surface detection. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_layout_utils.py | Adds shared pose/mesh utilities for y-up↔z-up placement and AABB measurement. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_layout_constructor.py | Introduces graph-driven layout construction over table + stacked parents. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_importer.py | Adds importer for exported scene+graph back into editable in-memory state. |
| embodichain/gen_sim/scene_engine/pipeline/utils/scene_exporter.py | Extends export to include graph + rotated export frame + metadata transforms + stale asset cleanup. |
| embodichain/gen_sim/scene_engine/pipeline/utils/image_segmentation_utils.py | Adds mask inversion heuristic and asset-ID overlay rendering utilities. |
| embodichain/gen_sim/scene_engine/pipeline/utils/gravity_settler.py | Adds generalized gravity settling for dynamic/static participants using Lab simulation. |
| embodichain/gen_sim/scene_engine/pipeline/utils/assets_group_layout_optimizer.py | Adjusts initial constraint handling by projecting AABBs into a rectangular support region. |
| embodichain/gen_sim/scene_engine/pipeline/utils/assets_gravity_settler.py | Removes legacy asset gravity settling implementation (superseded by GravitySettler). |
| embodichain/gen_sim/scene_engine/pipeline/generation/scene_understanding.py | Refactors understanding to return (Scene, SceneGraph) and adds orientation-state querying. |
| embodichain/gen_sim/scene_engine/pipeline/generation/init.py | Adds package init for generation pipeline module. |
| embodichain/gen_sim/scene_engine/pipeline/generate.py | Switches generation entrypoint to blueprint API (analyze_image → materialize_blueprint). |
| embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_layout_generation.py | Adds edit-time layout dispatch driven by updated goal scene graph. |
| embodichain/gen_sim/scene_engine/pipeline/editing/scene_edit_asset_preparation.py | Adds image→mask→geometry pipeline for newly added assets during editing. |
| embodichain/gen_sim/scene_engine/pipeline/editing/init.py | Adds package init for editing pipeline module. |
| embodichain/gen_sim/scene_engine/pipeline/edit.py | Adds edit entrypoint using blueprint API (analyze_edit → materialize_edit). |
| embodichain/gen_sim/scene_engine/pipeline/api.py | Introduces auditable blueprint/materialization API + manifest hashing + artifact recording. |
| embodichain/gen_sim/scene_engine/pipeline/init.py | Exposes new blueprint/materialization API from the pipeline package. |
| embodichain/gen_sim/scene_engine/errors.py | Adds SceneServiceError for consistent transient/remote failure signaling. |
| embodichain/gen_sim/scene_engine/core/scene_object.py | Extends scene object schema with center/support metadata used by layout/export/import. |
| embodichain/gen_sim/scene_engine/core/scene_edit_plan.py | Adds edit operation + plan model with validation and serialization. |
| embodichain/gen_sim/scene_engine/clients/image_segmentation.py | Renames segmentation endpoint configuration key/field to “by prompt”. |
| embodichain/gen_sim/scene_engine/clients/image_generation.py | Adds image generation client with seed acknowledgment and PNG validation. |
| embodichain/gen_sim/scene_engine/clients/geometry_generation.py | Adds seed propagation + unified SceneServiceError and response seed verification. |
| embodichain/gen_sim/scene_engine/cli/start.py | Extends CLI to support edit-only, generate-only, or generate+edit flows. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+114
to
+117
| outside_pixel_count = width * height - center_mask.width * center_mask.height | ||
| outside_foreground_ratio = outside_foreground_pixels / outside_pixel_count | ||
| if center_foreground_ratio >= outside_foreground_ratio: | ||
| return candidate |
Comment on lines
+326
to
+339
| image = Image.open(image_path).convert("RGBA") | ||
| overlay = Image.new("RGBA", image.size, (0, 0, 0, 0)) | ||
| colors = ( | ||
| (239, 83, 80, 255), | ||
| (66, 165, 245, 255), | ||
| (102, 187, 106, 255), | ||
| (255, 202, 40, 255), | ||
| (171, 71, 188, 255), | ||
| (38, 198, 218, 255), | ||
| ) | ||
| decoded_masks: list[tuple[str, Image.Image]] = [] | ||
| for index, (asset_id, mask_path) in enumerate(asset_masks): | ||
| mask = Image.open(mask_path).convert("L") | ||
| _require_image_size(mask, image.size) |
Comment on lines
+387
to
+406
| for x_index, minimum_x in enumerate(x_values[:-1]): | ||
| for maximum_x in x_values[x_index + 1 :]: | ||
| if maximum_x <= minimum_x: | ||
| continue | ||
| for y_index, minimum_y in enumerate(y_values[:-1]): | ||
| for maximum_y in y_values[y_index + 1 :]: | ||
| if maximum_y <= minimum_y: | ||
| continue | ||
| rectangle = Polygon( | ||
| [ | ||
| (minimum_x, minimum_y), | ||
| (maximum_x, minimum_y), | ||
| (maximum_x, maximum_y), | ||
| (minimum_x, maximum_y), | ||
| ] | ||
| ) | ||
| area = rectangle.area | ||
| if area > best_area and polygon.covers(rectangle): | ||
| best_rectangle = rectangle | ||
| best_area = area |
| def y_up_to_z_up_matrix() -> np.ndarray: | ||
| """Return the coordinate transform used by layout and export stages.""" | ||
| matrix = np.eye(4) | ||
| matrix[:3, :3] = np.array([[1.0, 0.0, 0.0], [0.0, 0.0, -1.0], [0.0, 1.0, 0.0]]) |
| def _y_up_to_z_up_matrix() -> np.ndarray: | ||
| """Return the coordinate conversion used by Scene Engine layouts.""" | ||
| matrix = np.eye(4) | ||
| matrix[:3, :3] = np.array([[1.0, 0.0, 0.0], [0.0, 0.0, -1.0], [0.0, 1.0, 0.0]]) |
| rot: list[float] | None = None # Final y-up Euler XYZ rotation in degrees. | ||
| pos: list[float] | None = None # Final y-up world position in metres. | ||
| scale: list[float] | None = None # Final y-up object scale. | ||
| center_xy: list[float] | None = None # Z-up table-frame XY AABB center. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stack
mainAdd the Scene Engine foundation required by the task-first GenSim workflow. This layer introduces semantic scene graphs, image and geometry service boundaries, scene generation/editing stages, deterministic layout and gravity handling, and scene import/export with preserved semantic metadata.
This layer is independently reviewable and does not depend on Action Engine or Task Engine.
Refs #531
Type of change
Validation
pytest -q tests/gen_sim/scene_engine- 86 passed, 9 warningsgit diff --check- passedChecklist