3.14 Automated Verification Loop Upgrade

The goal of this task is to upgrade "verify in game" from "launch the game + manually check" to "launch the game + automatically verify whether items appear in the creative mode inventory," and provide a clear verification result.
After completing this task, the ModSmith Web UI can do everything in one click: generate → compile → package → launch game → auto-verify → return result. Users no longer need to manually enter the game to check.

---

Prerequisites

• Completed Tasks 1–13: All core modules, CLI, and Web UI are ready.
• JDK 25 is installed and ./gradlew runClient runs normally.
• runClient has succeeded at least once before (Loom cache is ready, avoiding download time).

---

Step 1: Understand the two verification strategies

Phase 4 offers two complementary verification approaches; it is recommended to implement both, but they can be phased in:

Strategy	Principle	Pros	Cons
Log parsing	Look for mod registration, item registration, and creative tab registration logs in runClient's stdout/logs	Simple, stable, no GUI dependency	Only verifies "registration succeeded," cannot verify "actually displayed in-game"
RPA (Robotic Process Automation)	Use Java's Robot class to simulate keyboard and mouse: automatically open the creative inventory, search for items, take screenshots for comparison	Closer to real verification, can verify textures and names	Complex to implement, platform-dependent, sensitive to resolution

Recommended order: Implement log parsing first; once it works, consider RPA.

---

Step 2: Create modsmith/verifier/runtime.py

This module is responsible for:

• Launching ./gradlew runClient and capturing logs.
• Searching logs for key information (mod loading, item registration).
• Returning structured verification results.

"""Runtime verification: launch runClient and parse logs to verify mod loading and item registration."""

import os
import re
import subprocess
import time
from dataclasses import dataclass, field
from pathlib import Path


@dataclass
class RuntimeResult:
    """Runtime verification result."""
    success: bool
    mod_loaded: bool
    item_registered: bool
    raw_log: str
    matched_lines: list[str] = field(default_factory=list)
    message: str = ""


def run_client_with_log(
    project_dir: Path,
    timeout: int = 180,
) -> RuntimeResult:
    """Launch ./gradlew runClient, capture logs, and determine whether the mod and item loaded successfully.

    Args:
        project_dir: The generated project directory.
        timeout: Maximum wait time (seconds).

    Returns:
        A RuntimeResult object.
    """
    project_dir = project_dir.resolve()
    if os.name == "nt":
        gradlew = project_dir / "gradlew.bat"
    else:
        gradlew = project_dir / "gradlew"

    if not gradlew.exists():
        return RuntimeResult(
            success=False,
            mod_loaded=False,
            item_registered=False,
            raw_log="",
            message=f"Could not find {gradlew}",
        )

    try:
        proc = subprocess.Popen(
            [str(gradlew), "runClient"],
            cwd=str(project_dir),
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            bufsize=1,
        )
    except Exception as e:
        return RuntimeResult(
            success=False,
            mod_loaded=False,
            item_registered=False,
            raw_log="",
            message=f"Launch failed: {e}",
        )

    log_lines: list[str] = []
    mod_loaded = False
    item_registered = False
    start = time.time()

    try:
        while True:
            if proc.poll() is not None:
                # Process has exited; read any remaining output
                if proc.stdout:
                    log_lines.extend(proc.stdout.readlines())
                break

            if time.time() - start > timeout:
                proc.terminate()
                break

            if proc.stdout:
                line = proc.stdout.readline()
                if not line:
                    time.sleep(0.2)
                    continue
                log_lines.append(line)

                # Check if the mod loaded successfully
                if "Loading" in line and "mods:" in line:
                    # The following lines will list the loaded mods
                    pass
                if "Hello Fabric world from" in line:
                    mod_loaded = True

                # Check if the item registered successfully (based on the log in ModItemsGenerated)
                if "Registered item" in line or "item.example-mod" in line:
                    item_registered = True

                # Early exit: both conditions met
                if mod_loaded and item_registered:
                    # Wait a few more seconds to collect remaining logs
                    time.sleep(2)
                    proc.terminate()
                    break
    except KeyboardInterrupt:
        proc.terminate()

    raw_log = "".join(log_lines)

    # Fallback: run a regex scan once more
    if not mod_loaded:
        mod_loaded = bool(re.search(r"Hello Fabric world from", raw_log))
    if not item_registered:
        item_registered = bool(re.search(r"item\.example-mod\.", raw_log))

    matched = [
        line.strip() for line in log_lines
        if "Hello Fabric world" in line or "item.example-mod" in line
    ]

    success = mod_loaded and item_registered
    if success:
        message = "✅ Mod loaded, item registered."
    elif mod_loaded:
        message = "⚠️ Mod loaded, but no item registration log detected."
    else:
        message = "❌ Mod failed to load. Please check the logs."

    return RuntimeResult(
        success=success,
        mod_loaded=mod_loaded,
        item_registered=item_registered,
        raw_log=raw_log,
        matched_lines=matched,
        message=message,
    )

Notes:

• subprocess.Popen reads output line by line and detects key log entries in real time.
• mod_loaded is determined by Hello Fabric world from.
• item_registered is determined by item.example-mod. (requires Task 7 to add logging in ModItemsGenerated).
• timeout prevents waiting indefinitely if the game fails to launch.

---

Step 3: Add registration logs to ModItemsGenerated.java.jinja

To let log parsing detect item registration, the generator template needs to emit logs.

Open modsmith/templates/java/ModItemsGenerated.java.jinja and add the following inside the initialize() method:

package {{ package_name }};

import net.fabricmc.fabric.api.creativetab.v1.CreativeModeTabEvents;
import net.minecraft.world.item.CreativeModeTabs;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class ModItemsGenerated {
    private static final Logger LOGGER = LoggerFactory.getLogger("{{ mod_id }}");

    public static void initialize() {
        ModItems.initialize();

{% if items %}
        CreativeModeTabEvents.modifyOutputEvent(CreativeModeTabs.INGREDIENTS)
            .register(entries -> {
{% for item in items %}
                entries.accept(ModItems.{{ item.id | upper }});
                LOGGER.info("item.{{ mod_id }}.{{ item.id }} registered");
{% endfor %}
            });
{% endif %}
    }
}

Key points:

• Each item emits a log line item.<mod_id>.<item_id> registered after registration.
• The log parsing module uses this to determine whether items registered successfully.

---

Step 4: Integrate auto-verification into the Web UI

4.1 Add a new endpoint in modsmith/web/app.py

from modsmith.verifier.runtime import run_client_with_log


class AutoVerifyRequest(BaseModel):
    task_id: str


@app.post("/api/auto_verify")
async def auto_verify(req: AutoVerifyRequest) -> dict:
    """Launch the game and automatically verify mod loading and item registration."""
    if req.task_id not in TASKS:
        raise HTTPException(status_code=404, detail="Task not found")
    task = TASKS[req.task_id]
    project_dir = task.get("project_dir")
    if not project_dir:
        raise HTTPException(status_code=400, detail="Project directory does not exist")

    project_path = Path(project_dir).resolve()
    if not project_path.exists():
        raise HTTPException(status_code=400, detail=f"Project directory does not exist: {project_path}")

    # Run synchronously (long-running; acceptable during MVP)
    result = run_client_with_log(project_path, timeout=180)
    return {
        "success": result.success,
        "mod_loaded": result.mod_loaded,
        "item_registered": result.item_registered,
        "message": result.message,
        "matched_lines": result.matched_lines,
    }

Note: During MVP, this runs synchronously and may block for several minutes. For a better experience, use a background thread + SSE, but the implementation is more complex.

4.2 Add a button in index.html

Next to the "🎮 Verify in Game" button, add a "🔍 Auto Verify" button:

<div id="verify-section">
  <button id="verify-btn" onclick="runClient()">🎮 Verify in Game</button>
  <button id="auto-verify-btn" onclick="autoVerify()" style="margin-left: 8px;">🔍 Auto Verify</button>
  <div id="verify-msg"></div>
</div>

4.3 Add the autoVerify() function

async function autoVerify() {
  if (!currentTaskId) return;
  const msg = document.getElementById('verify-msg');
  msg.textContent = 'Launching the game and running automatic verification. Please wait (may take 1–3 minutes)...';

  try {
    const resp = await fetch('/api/auto_verify', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ task_id: currentTaskId }),
    });
    const result = await resp.json();
    if (resp.ok) {
      msg.innerHTML =
        (result.success ? '✅ ' : '❌ ') + result.message +
        '<br>Mod loaded: ' + (result.mod_loaded ? '✅' : '❌') +
        '<br>Item registered: ' + (result.item_registered ? '✅' : '❌') +
        (result.matched_lines && result.matched_lines.length
          ? '<br><br>Matched log lines:<br>' + result.matched_lines.join('<br>')
          : '');
    } else {
      msg.textContent = '❌ ' + (result.detail || 'Verification failed');
    }
  } catch (e) {
    msg.textContent = '❌ Request failed: ' + e;
  }
}

---

Step 5: Create the test script test_runtime.py

"""Test the runtime auto-verification module."""

from pathlib import Path

from modsmith.verifier.runtime import run_client_with_log

# Assume generated_project already exists and compiled successfully
project_dir = Path("./generated_project")

result = run_client_with_log(project_dir, timeout=180)
print(f"Success: {result.success}")
print(f"Mod loaded: {result.mod_loaded}")
print(f"Item registered: {result.item_registered}")
print(f"Message: {result.message}")
print("Matched log lines:")
for line in result.matched_lines:
    print(f"  {line}")

Run:

python test_runtime.py

Expected result:

• If both the mod and item are fine, output Success: True.
• If only the mod loaded but the item did not register, output Item registered: False.

---

Step 6: (Optional) RPA verification

If you want to further verify "the item is actually displayed in-game," you can implement Java Robot-based RPA. The idea is:

1. Launch runClient.
2. Wait for the game window to appear.
3. Use Robot to simulate keys: enter a single-player creative world, open the inventory, search for the item.
4. Take a screenshot and compare with expectations.

Example implementation (conceptual code):

def run_rpa_verification(project_dir: Path) -> bool:
    """Use Java Robot for RPA verification.

    Requires writing a Java test class in the project, invoked via a Gradle task.
    """
    # 1. Call gradlew runRpaTest
    # 2. Java side uses Robot to operate, take screenshots, and compare
    # 3. Determine success by exit code
    ...

The implementation details of this approach depend on the specific environment (resolution, window position, game language) and are recommended as a later iteration.

---

Step 7: Common issues

Problem	Cause	Solution
Hello Fabric world not in log	ExampleMod did not execute correctly	Check entrypoints in fabric.mod.json and ExampleMod.java
item.xxx registered not in log	ModItemsGenerated did not add logging	Modify the template per Step 3 and regenerate
Auto-verification timed out	Game startup slow or downloading dependencies	Increase timeout, or run runClient manually once to warm up the cache
Game exits before the window opens	Missing graphical environment	Confirm you're running in an environment with a display; server/CI environments are unsuitable for RPA
Web UI blocks	Synchronous execution takes too long	Switch to a background task + polling, or push progress via SSE

---

Step 8: Task 14 Acceptance Criteria

• modsmith/verifier/runtime.py is created, containing run_client_with_log().
• ModItemsGenerated.java.jinja outputs item registration logs.
• The /api/auto_verify endpoint is added in modsmith/web/app.py.
• A "🔍 Auto Verify" button and the autoVerify() function are added in index.html.
• test_runtime.py runs successfully and outputs the verification results for mod loading and item registration.
• After clicking "Auto Verify" in the Web UI, structured verification results are returned.
• (Optional) An evaluation record of the RPA verification approach.

After completing this task, ModSmith achieves a complete verification loop: from a natural language description, to project generation, compilation, packaging, game launch, and automatic verification of item registration—all done automatically without user intervention. This is also the core goal of Phase 4.

---

Future extension directions

• Change /api/auto_verify to an asynchronous task with SSE progress push.
• Support automatic verification of multiple items (search each item ID).
• Introduce screenshot comparison to verify whether textures display correctly.
• Run headlessly in CI environments to enable automated regression testing.