Problem
When a workspace is defined via a file reference (e.g., workspace: .templates/eval-workspace-setup.yaml), hook commands execute with cwd defaulting to evalDir (the eval file's directory). This means the same template produces different behavior depending on which eval file references it, breaking the expectation that file references are transparent.
Repro
Given this directory structure:
repo/
scripts/eval-config/copy-local-skills.mjs
evals/
mygroup/
.templates/
eval-workspace-setup.yaml
root-eval.eval.yaml # depth 2
subdir/
nested-eval.eval.yaml # depth 3
Template (evals/mygroup/.templates/eval-workspace-setup.yaml):
mode: static
hooks:
before_all:
command:
- node
- ../../scripts/eval-config/copy-local-skills.mjs
Root eval (evals/mygroup/root-eval.eval.yaml):
workspace: .templates/eval-workspace-setup.yaml
tests:
- id: test1
input: hello
Nested eval (evals/mygroup/subdir/nested-eval.eval.yaml):
workspace: ../.templates/eval-workspace-setup.yaml
tests:
- id: test1
input: hello
Result
root-eval.eval.yaml — passes: evalDir = evals/mygroup/, command resolves ../../scripts/... → scripts/... from repo root ✓
nested-eval.eval.yaml — fails: evalDir = evals/mygroup/subdir/, command resolves ../../scripts/... → evals/scripts/... ✗
Error: Cannot find module '/repo/evals/scripts/eval-config/copy-local-skills.mjs'
Expected
Both evals should behave identically since they reference the same template. The hook command's relative paths should resolve from the template file's directory, not the eval file's directory.
Root cause
In executeWorkspaceScript:
const cwd = config.cwd ?? context2.evalDir;
When config.cwd is not set, execution falls back to evalDir. Since evalDir varies by eval file depth, the same template command resolves differently.
Note: when cwd IS explicitly set in the hook config, it's correctly resolved relative to workspaceFileDir during parsing (parseWorkspaceScriptConfig). The issue is only with the default when cwd is omitted.
Proposed fix
Pass workspaceFileDir through the execution context and use it as the default:
const cwd = config.cwd ?? context2.workspaceFileDir ?? context2.evalDir;
This makes file-referenced templates self-contained — their hook commands always resolve relative to the template's own directory, matching how cwd is already resolved during parsing.
Workaround
Add explicit cwd to the hook config:
hooks:
before_all:
cwd: ../../..
command:
- node
- scripts/eval-config/copy-local-skills.mjs
This works because cwd is resolved relative to workspaceFileDir during parsing. But it's fragile and couples the template to its filesystem depth.
Problem
When a workspace is defined via a file reference (e.g.,
workspace: .templates/eval-workspace-setup.yaml), hook commands execute withcwddefaulting toevalDir(the eval file's directory). This means the same template produces different behavior depending on which eval file references it, breaking the expectation that file references are transparent.Repro
Given this directory structure:
Template (
evals/mygroup/.templates/eval-workspace-setup.yaml):Root eval (
evals/mygroup/root-eval.eval.yaml):Nested eval (
evals/mygroup/subdir/nested-eval.eval.yaml):Result
root-eval.eval.yaml— passes: evalDir =evals/mygroup/, command resolves../../scripts/...→scripts/...from repo root ✓nested-eval.eval.yaml— fails: evalDir =evals/mygroup/subdir/, command resolves../../scripts/...→evals/scripts/...✗Expected
Both evals should behave identically since they reference the same template. The hook command's relative paths should resolve from the template file's directory, not the eval file's directory.
Root cause
In
executeWorkspaceScript:When
config.cwdis not set, execution falls back toevalDir. SinceevalDirvaries by eval file depth, the same template command resolves differently.Note: when
cwdIS explicitly set in the hook config, it's correctly resolved relative toworkspaceFileDirduring parsing (parseWorkspaceScriptConfig). The issue is only with the default whencwdis omitted.Proposed fix
Pass
workspaceFileDirthrough the execution context and use it as the default:This makes file-referenced templates self-contained — their hook commands always resolve relative to the template's own directory, matching how
cwdis already resolved during parsing.Workaround
Add explicit
cwdto the hook config:This works because
cwdis resolved relative toworkspaceFileDirduring parsing. But it's fragile and couples the template to its filesystem depth.