Skip to content

Hook cwd should default to workspaceFileDir when workspace is a file reference #1080

@christso

Description

@christso

Problem

When a workspace is defined via a file reference (e.g., workspace: .templates/eval-workspace-setup.yaml), hook commands execute with cwd defaulting to evalDir (the eval file's directory). This means the same template produces different behavior depending on which eval file references it, breaking the expectation that file references are transparent.

Repro

Given this directory structure:

repo/
  scripts/eval-config/copy-local-skills.mjs
  evals/
    mygroup/
      .templates/
        eval-workspace-setup.yaml
      root-eval.eval.yaml          # depth 2
      subdir/
        nested-eval.eval.yaml      # depth 3

Template (evals/mygroup/.templates/eval-workspace-setup.yaml):

mode: static
hooks:
  before_all:
    command:
      - node
      - ../../scripts/eval-config/copy-local-skills.mjs

Root eval (evals/mygroup/root-eval.eval.yaml):

workspace: .templates/eval-workspace-setup.yaml
tests:
  - id: test1
    input: hello

Nested eval (evals/mygroup/subdir/nested-eval.eval.yaml):

workspace: ../.templates/eval-workspace-setup.yaml
tests:
  - id: test1
    input: hello

Result

  • root-eval.eval.yamlpasses: evalDir = evals/mygroup/, command resolves ../../scripts/...scripts/... from repo root ✓
  • nested-eval.eval.yamlfails: evalDir = evals/mygroup/subdir/, command resolves ../../scripts/...evals/scripts/...
Error: Cannot find module '/repo/evals/scripts/eval-config/copy-local-skills.mjs'

Expected

Both evals should behave identically since they reference the same template. The hook command's relative paths should resolve from the template file's directory, not the eval file's directory.

Root cause

In executeWorkspaceScript:

const cwd = config.cwd ?? context2.evalDir;

When config.cwd is not set, execution falls back to evalDir. Since evalDir varies by eval file depth, the same template command resolves differently.

Note: when cwd IS explicitly set in the hook config, it's correctly resolved relative to workspaceFileDir during parsing (parseWorkspaceScriptConfig). The issue is only with the default when cwd is omitted.

Proposed fix

Pass workspaceFileDir through the execution context and use it as the default:

const cwd = config.cwd ?? context2.workspaceFileDir ?? context2.evalDir;

This makes file-referenced templates self-contained — their hook commands always resolve relative to the template's own directory, matching how cwd is already resolved during parsing.

Workaround

Add explicit cwd to the hook config:

hooks:
  before_all:
    cwd: ../../..
    command:
      - node
      - scripts/eval-config/copy-local-skills.mjs

This works because cwd is resolved relative to workspaceFileDir during parsing. But it's fragile and couples the template to its filesystem depth.

Metadata

Metadata

Assignees

No one assigned

    Labels

    in-progressClaimed by an agent — do not duplicate work

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions