Skip to content

add CI tests for openhands-getting-started playbook - #615

Open
harkgill-amd wants to merge 11 commits into
mainfrom
users/harkgill/openhands-ci-tests
Open

add CI tests for openhands-getting-started playbook#615
harkgill-amd wants to merge 11 commits into
mainfrom
users/harkgill/openhands-ci-tests

Conversation

@harkgill-amd

@harkgill-amd harkgill-amd commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Addresses #601.

Added tests for openhands CI following test patterns from existing playbooks (most notable n8ns and other lemonade based playbooks).

Temporarily forcing uv install through CI tests to ensure consistency across all runners.

  ┌──────────────────────────┬───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Test ID (×linux/windows) │           Step            │                                                                  What it validates                                                                   │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ prereq-clis-*            │ Prerequisites             │ lemonade, node, npm, uv all present                                                                                                                  │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ lemonade-chat-*          │    Verify the Local Model │ Lemonade /api/v1/models responds; model is present + downloaded; deterministic "Reply with exactly: OK" chat (temperature:0, max_tokens:32) succeeds │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-install-*   │    Install                │ npm install -g @openhands/agent-canvas succeeds (user-owned npm prefix on Linux, no root needed)                                                     │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-version-*   │    Install                │ agent-canvas CLI is on PATH (--version, falling back to --help)                                                                                      │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-server-*    │    Launch                 │ Server starts, http://127.0.0.1:8000/ returns HTTP 200, then the process is killed cleanly (frees the shared runner)                                 │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ lemonade-unload-*        │ Cleanup                   │ lemonade unload frees the GPU pool for the next job                                                                                                  │
  └──────────────────────────┴───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

@harkgill-amd
harkgill-amd requested a review from sreeram-11 July 21, 2026 20:44
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
Comment thread playbooks/supplemental/openhands-getting-started/README.md Outdated
@sreeram-11

sreeram-11 commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

@adamlam2-amd,

Can you please verify whether Agent Canvas is supported on Windows using Docker, and add the instructions to the playbook if it is supported?

Please also address any formatting issues in the playbook. One thing I noticed is that the README.md content in this playbook is not formatted consistently with the other playbooks. In the other playbooks, sentences are generally split onto separate lines for readability and cleaner diffs. Could you please update this playbook to follow the same formatting style?

@sreeram-11
sreeram-11 marked this pull request as ready for review July 27, 2026 06:36
@harkgill-amd

Copy link
Copy Markdown
Collaborator Author

Hey @neubig, our Windows CI runners are failing to execute the playbook commands with the below logs from the agent-canvas step,

  [1/2] Checking prerequisites... 
  [2/2] Starting services...
  [agent-server] Starting on port 18000...
  [agent-server] Using PyPI (1.36.1, default)
  [agent-server] The system cannot find the file specified.
  [agent-server] Exited with code 1                                  
  [agent-server] Timeout waiting for http://localhost:18000/server_info after 60s
  [agent-server] Last error: fetch failed
  [secrets] Skipping secret seeding - agent-server not ready
  [automation] Building litellm==1.93.0
  [automation] × Failed to build `litellm==1.93.0`
  [automation]   ╰─▶ Call to `maturin.build_wheel` failed (exit code: 1)
  [automation]       Rust not found, installing into a temporary directory

The frontend comes up but the agent-server backend never starts due to a missing litellm dependency. Noticing the documentation at https://github.com/OpenHands/openhands#quickstart points to a docker based setup for Windows systems. Do we want to go this route for the playbook as well or is there a recommended solution for the environment setup issues seen on a native Windows install?

@adamlam2-amd

Copy link
Copy Markdown
Collaborator

Let's resolve the Windows issue and make sure everything looks good before we merge this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants