Skip to content

Step 6: Headless navigation pipeline (vertical slice) #9

Closed
Closed
Step 6: Headless navigation pipeline (vertical slice)#9
@thomasnemer

Description

@thomasnemer
thomasnemer
opened on Mar 24, 2026

Parent: #2

Goal

Wire ie-net into the headless execution path so that --headless --dump-source --url <URL> actually fetches the page and prints it. Also implement the interactive headless JSON protocol that e2e tests will use.

This is the Phase 1a milestone: the first time the browser does something useful end-to-end.

Prerequisites

  • Step 2 (ie-net — working HTTP client)
  • Step 3 (headless mode and CLI)
  • Step 4 (navigation service, tab model, bookmarks)

File Changes

  • src/headless.rs — replace placeholder implementations with real logic
  • Add serde/serde_json to ie-shell deps if not already present

Implementation

One-shot headless commands (headless.rs)

  • DumpSource: 
    let navigator = InProcessNavigator::new()?;
let result = navigator.navigate(&url).await?;
let text = String::from_utf8_lossy(&result.body);
print!("{text}");
    • Exit code 0 on success
    • On navigation error: print error to stderr, exit code 1
  • DumpStatus: 
    let navigator = InProcessNavigator::new()?;
let result = navigator.navigate(&url).await?;
println!("{}", result.status);
    • Same error handling as DumpSource

Interactive headless mode — JSON protocol (headless.rs)

  • Internal state: 
    struct HeadlessSession {
    tab_manager: TabManager,
    bookmark_store: BookmarkStore,
    navigator: InProcessNavigator,
}
  • Command format (newline-delimited JSON, one command per line on stdin): 
    #[derive(Deserialize)]
#[serde(tag = "cmd")]
enum Command {
    #[serde(rename = "navigate")]
    Navigate { url: String },
    #[serde(rename = "get_source")]
    GetSource,
    #[serde(rename = "get_tabs")]
    GetTabs,
    #[serde(rename = "new_tab")]
    NewTab,
    #[serde(rename = "close_tab")]
    CloseTab { id: u64 },
    #[serde(rename = "switch_tab")]
    SwitchTab { id: u64 },
    #[serde(rename = "go_back")]
    GoBack,
    #[serde(rename = "go_forward")]
    GoForward,
    #[serde(rename = "bookmark_add")]
    BookmarkAdd { url: String, title: String },
    #[serde(rename = "bookmark_list")]
    BookmarkList,
    #[serde(rename = "quit")]
    Quit,
}
  • Response format (newline-delimited JSON, one response per line on stdout): 
    #[derive(Serialize)]
struct Response {
    ok: bool,
    #[serde(skip_serializing_if = "Option::is_none")]
    data: Option<serde_json::Value>,
    #[serde(skip_serializing_if = "Option::is_none")]
    error: Option<String>,
}

Command handlers

  • navigate:
    1. Parse URL (prepend https:// if no scheme)
    2. Set active tab state to Loading
    3. Call navigator.navigate(&url).await
    4. On success: check content-type — if not text/html (and content-type is set), respond with {"ok": false, "error": "unsupported content type: ..."} and set tab state to Error
    5. On success with valid content-type: update tab (url, state=Loaded, source=body as UTF-8), respond {"ok": true, "data": {"status": 200, "url": "..."}}
    6. On failure: update tab (state=Error), respond {"ok": false, "error": "..."}
  • get_source:
    • Return active tab's source or error if no source/no active tab
    • {"ok": true, "data": "<html>..."}
  • get_tabs:
    • Return list of tabs: {"ok": true, "data": [{"id": 1, "url": "...", "title": "...", "state": "loaded"}]}
    • State serialized as lowercase string: "blank", "loading", "loaded", "error"
  • new_tab:
    • tab_manager.new_tab()
    • {"ok": true, "data": {"id": 2}}
  • close_tab:
    • tab_manager.close_tab(id)
    • {"ok": true} or {"ok": false, "error": "tab not found"}
  • switch_tab:
    • tab_manager.switch_to(TabId(id))
    • {"ok": true} or {"ok": false, "error": "tab not found"}
  • go_back:
    • tab_manager.go_back()
    • {"ok": true, "data": {"url": "..."}} if navigated back, {"ok": false, "error": "no back history"} if at beginning
  • go_forward:
    • tab_manager.go_forward()
    • {"ok": true, "data": {"url": "..."}} if navigated forward, {"ok": false, "error": "no forward history"} if at end
  • bookmark_add:
    • bookmark_store.add(url, title)
    • {"ok": true}
  • bookmark_list:
    • {"ok": true, "data": [{"url": "...", "title": "...", "created": "..."}]}
  • quit:
    • Respond {"ok": true}, then break the read loop and exit

Main read loop

  • run_interactive_headless(): 
    let stdin = BufReader::new(tokio::io::stdin());
let mut stdout = tokio::io::stdout();
let mut session = HeadlessSession::new()?;

let mut lines = stdin.lines();
while let Some(line) = lines.next_line().await? {
    let cmd: Command = serde_json::from_str(&line)?;
    let response = session.handle_command(cmd).await;
    let json = serde_json::to_string(&response)?;
    stdout.write_all(json.as_bytes()).await?;
    stdout.write_all(b"\n").await?;
    stdout.flush().await?;
    if matches!(cmd, Command::Quit) { break; }
}
  • On malformed JSON input: respond with {"ok": false, "error": "invalid command: ..."} and continue (don't crash)
  • On EOF (stdin closed): exit cleanly

Sequential command processing

Commands are processed one at a time. navigate blocks until the HTTP request completes before the next command is read. This is by design for Phase 1 simplicity — it makes the protocol deterministic and easy to test. Document this constraint in code comments.

Future optimization: allow concurrent navigations with request IDs for correlation.

--allow-http flag support

  • The --allow-http flag from CLI is passed through to the InProcessNavigator: 
    let navigator = InProcessNavigator::new()?.with_https_only(!cli.allow_http);
  • This affects both one-shot commands (DumpSource, DumpStatus) and interactive mode

Bookmark store path for headless

  • Interactive headless mode: use a unique temp directory for bookmark storage by default (so tests don't pollute real bookmarks)
  • Add --data-dir <path> CLI flag to override data directory (useful for e2e tests that need persistence across restarts)

Verification

# Fetch a page and dump its source:
cargo run -p ie-shell -- --headless --dump-source --url https://example.com
# Should print the HTML of example.com

# Get just the status code:
cargo run -p ie-shell -- --headless --dump-status --url https://example.com
# Should print: 200

# Interactive mode:
echo '{"cmd":"navigate","url":"https://example.com"}
{"cmd":"get_source"}
{"cmd":"quit"}' | cargo run -p ie-shell -- --headless
# Should print JSON responses for each command

# Error case:
cargo run -p ie-shell -- --headless --dump-status --url https://nonexistent.invalid
# Should print error to stderr, exit code 1

Tests

Integration tests (in crates/ie-shell/tests/)

These spawn the binary as a subprocess and test the full pipeline:

  • dump_status_200: start local test server returning 200. Run --headless --dump-status --url http://127.0.0.1:PORT --allow-http. Assert stdout is "200\n" and exit code 0.
  • dump_status_404: server returns 404. Assert stdout is "404\n".
  • dump_source_matches: server returns <html>test</html>. Run --headless --dump-source --allow-http. Assert stdout matches.
  • dump_source_error: URL points to nothing. Assert exit code 1, stderr non-empty.
  • interactive_navigate: spawn --headless --allow-http, send navigate command, send get_source, send quit. Verify navigate response has "ok": true, get_source returns the page body.
  • interactive_tabs: spawn, send new_tab, get_tabs (expect 2), close_tab, get_tabs (expect 1).
  • interactive_switch_tab: spawn, new_tab, switch_tab to first tab, get_tabs to verify active tab changed.
  • interactive_go_back_forward: spawn, navigate to A, navigate to B, go_back (expect A), go_forward (expect B).
  • interactive_bookmarks: spawn, navigate, bookmark_add, bookmark_list (expect 1 entry).
  • interactive_invalid_json: send garbage line, verify error response (not crash), then send valid quit command.
  • interactive_quit: send quit, verify process exits cleanly.

Test helper

  • Reusable test server helper: fn start_test_server() -> (SocketAddr, impl Drop) — HTTP server returning configurable responses, automatically stopped on drop

Acceptance Criteria

  • cargo run -p ie-shell -- --headless --dump-source --url https://example.com prints HTML
  • cargo run -p ie-shell -- --headless --dump-status --url https://example.com prints 200
  • Interactive headless mode handles all command types including switch_tab, go_back, go_forward
  • Malformed input doesn't crash the interactive session
  • All integration tests pass
  • cargo clippy -p ie-shell -- -D warnings — no warnings

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions