Conductor Design — Polymorphic conventions: extend `--workspace-instructions` to recognize `.github/instructions/*.instructions.md` with applyTo-aware filtering

## TL;DR

PR #141 added `--workspace-instructions` to conductor. Today it discovers exactly 3 hard-coded file conventions (`AGENTS.md`, `.github/copilot-instructions.md`, `CLAUDE.md`) — verified literally at `src/conductor/config/instructions.py:21-25`'s `CONVENTION_FILES` list.

GitHub Copilot itself already discovers more — its supported set includes `.github/instructions/**/*.instructions.md` files (per the [GitHub Copilot Chat docs](https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses) and the [VS Code Copilot docs](https://code.visualstudio.com/docs/copilot/customization/custom-instructions)). This PR closes that gap by:

1. **Refactoring** `CONVENTION_FILES: list[str]` into a polymorphic `CONVENTIONS: list[Convention]` using a `Convention` type union (`ConventionFile | ConventionDirectory`).
2. **Adding** `.github/instructions/` as a `ConventionDirectory` with `applyTo` frontmatter filtering — so only files marked always-on (`applyTo: "**"`) are loaded.

Single existing flag (`--workspace-instructions`); no new flag. The polymorphic shape pays the small refactor cost ONCE so future conventions (Cursor's `.cursor/rules/`, Cline's `.clinerules/`, etc.) can be added via "one filter function + one list entry" without further walker changes.

---

## Why polymorphic (and why now)

**Today's `CONVENTION_FILES: list[str]`** assumes every entry is a single relative file path checked with `Path.is_file()`. Adding `.github/instructions/` requires fundamentally different mechanics:
- Recursive directory walk (not single-file existence)
- Glob pattern matching (`*.instructions.md`)
- Per-file frontmatter filter (`applyTo` semantics)
- Closest-wins per relative-path-within-directory (not per-filename)

Either:
- **Option (a) — fork the walker** with a dedicated `_discover_github_instructions_dir()` function. Future conventions each duplicate that walker logic. Smaller initial diff but compounds maintenance burden.
- **Option (b) — refactor to polymorphic** entries. Walker dispatches on type. Future conventions add one filter function + one list entry. Larger initial diff but extension cost converges to ~10 LoC per future convention.

This design picks **(b)**. Truth-grounded reasons:

1. **Bounded refactor scope**: `CONVENTION_FILES` is referenced in exactly 2 files — `instructions.py` (impl) + `test_instructions.py` (tests). Verified via grep. Not exported in `config/__init__.py`'s `__all__`, so it's an internal implementation detail.
2. **Future PRs become uniform**: each next convention (Cursor, Cline, Claude rules-dir, Continue, Windsurf) is a 10-line PR (filter function + list entry + tests).
3. **Walker logic centralizes**: changes to symlink policy, error handling, logging, etc. happen in one place instead of N forks.
4. **Filters become testable in isolation** (pure functions; no walker setup).

---

## Building on PR #141 (verified line-by-line)

12 architectural pieces from PR #141 are reused:

| Capability | Source | This PR |
|---|---|---|
| `--workspace-instructions` flag | `cli/app.py:309-320` | Same; help-text update for new convention |
| Git-root walk (`_find_git_root` + walker loop) | `instructions.py:_find_git_root` + `instructions.py:78-90` | Walker generalized to dispatch on Convention type; `_find_git_root` reused as-is |
| Closest-wins per filename | `instructions.py:80-86` | Generalized: closest-wins-per-key, where key is filename for files, relative-path-within-dir for directories |
| `load_instruction_files()` content reader | `instructions.py:96-130` | Reused as-is |
| `<workspace_instructions>` wrapping | `instructions.py:_wrap_preamble` | Reused as-is |
| 3-source merge (auto + YAML + CLI) | `instructions.py:build_inner_instructions` | Reused as-is |
| Engine preamble injection | `executor/agent.py:157-166` | Untouched |
| Sub-workflow merging | `engine/workflow.py:794-797` | Untouched |
| Checkpoint persistence | `engine/checkpoint.py:319` | Untouched |
| Bg-runner subprocess flag forwarding | `cli/bg_runner.py:199-204` | Untouched |
| Resume command | `cli/app.py:resume()` | Untouched |
| `WorkflowDef.instructions: list[str]` | `config/schema.py:964-984` | Untouched |

PR is small precisely because PR #141 did the heavy lifting.

---

## The Convention (truth-grounded)

GitHub Copilot's documented `.github/instructions/` convention — supported across the entire Copilot product family:

- **GitHub Copilot Chat on github.com** — https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses (refers to `.github/instructions/**/*.instructions.md` as "Path-specific custom instructions")
- **GitHub Copilot in VS Code** — https://code.visualstudio.com/docs/copilot/customization/custom-instructions
- **GitHub Copilot CLI** — supports the `.github/instructions/**/*.instructions.md` convention (per `copilot help` topic-specific output and the `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` environment variable; see also the documented behavior shared with VS Code Copilot since they use the same agentic harness)

Convention details:

- Location: `.github/instructions/` directory at workspace root (or subdirectory)
- File pattern: `*.instructions.md` (note the double extension; this is GitHub's specific naming)
- Recursive: yes — Copilot searches the folder recursively
- Frontmatter dialect:
  ```yaml
  ---
  description: 'Coding conventions'    # optional description
  applyTo: '**/*.py'                   # optional glob
  ---
  ```
- Activation semantics (verbatim from docs):
  - `applyTo: "**"` → auto-applies to all chats (always-on)
  - `applyTo: "<other glob>"` → auto-applies only when the glob matches files in the chat (scoped)
  - `applyTo` absent → "the instructions are **not applied automatically**, but you can still add them manually to a chat request"

Conductor's preamble is always-on for every agent prompt. To honor the convention semantically:

- `applyTo: "**"` → INCLUDE in preamble (the convention says always-on)
- `applyTo: "<other-glob>"` → SKIP (scoped per convention; conductor has no per-agent scoping)
- `applyTo` absent → SKIP (per docs: "not applied automatically"; manual-attach only)

This is the **conservative, correct** interpretation. It prevents conductor from injecting files that the convention says are scoped or manual-only.

> **UX risk acknowledgment**: Some users may misunderstand the convention and create `.github/instructions/foo.instructions.md` without `applyTo` expecting it to be always-on. The conductor design correctly reflects the convention's documented semantic; if user feedback shows widespread misunderstanding, a future PR can add an explicit `--include-unscoped-instructions` flag. This PR does NOT compensate for the misunderstanding by reinterpreting the convention.

> **Cross-tool note**: GitHub Copilot CLI also exposes a `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` env-var for additional directories. Conductor does NOT honor this env-var (env-var override is explicitly out of scope). Teams who want non-standard paths use conductor's `--instructions <path>` escape hatch.

---

## Design

### Hard standards (user-imposed)

1. **Principled** — extends PR #141 architecture; refactors single-purpose constant into a polymorphic abstraction that scales
2. **Truth-grounded** — every claim verifiable from source code or vendor docs
3. **Generic** — applies to any team using GitHub's documented convention (Copilot Chat, VS Code, CLI); polymorphic shape primes for future conventions
4. **No hacks/workarounds** — every corner case addressed; deferrals have explicit rationale
5. **Backward-compatible** — repos without `.github/instructions/` see no behavior change

### Architectural choices

#### A. Polymorphic `Convention` types

Replace `CONVENTION_FILES: list[str]` with:

```python
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Union


@dataclass(frozen=True)
class ConventionFile:
    """A single-file convention (e.g., AGENTS.md, .github/copilot-instructions.md, CLAUDE.md).
    
    Discovery: walk CWD → git root, closest-wins per `path` relative-path key
    (within this convention's own state, not shared across conventions).
    """
    path: str  # e.g., "AGENTS.md", ".github/copilot-instructions.md"


@dataclass(frozen=True)
class ConventionDirectory:
    """A directory-style convention (e.g., .github/instructions/*.instructions.md).
    
    Discovery: walk CWD → git root, closest-wins per relative-path-within-directory
    key (within this convention's own state, not shared across conventions).
    Glob pattern matched recursively within the directory (when recursive=True).
    Optional include_file predicate applied to each candidate file.
    """
    path: str                                   # e.g., ".github/instructions"
    pattern: str                                # e.g., "*.instructions.md"
    include_file: Callable[[Path], bool] | None = None  # True = INCLUDE; None = include all
    recursive: bool = True


Convention = Union[ConventionFile, ConventionDirectory]
# Alternative on Python 3.12+: `type Convention = ConventionFile | ConventionDirectory`


# The single source of truth for what conductor auto-discovers.
CONVENTIONS: list[Convention] = [
    ConventionFile("AGENTS.md"),
    ConventionFile(".github/copilot-instructions.md"),
    ConventionFile("CLAUDE.md"),
    ConventionDirectory(
        path=".github/instructions",
        pattern="*.instructions.md",
        include_file=_is_always_on_instructions_file,
    ),
]


# BACKWARD-COMPAT ALIAS: keep CONVENTION_FILES importable for any external
# code that imported it. The symbol is module-public (no leading underscore)
# and could be referenced by downstream consumers via direct import even
# though it is not in __all__.
CONVENTION_FILES: list[str] = [
    c.path for c in CONVENTIONS if isinstance(c, ConventionFile)
]
```

Walker (in `discover_workspace_instructions`) dispatches on type using **per-convention discovery state** (per rubber-duck non-blocking #5: prevents cross-convention key shadowing):

```python
result: list[Path] = []
for convention in CONVENTIONS:
    # Per-convention state — no shared dict; closest-wins is local to each convention.
    discovered: dict[str, Path] = {}
    
    if isinstance(convention, ConventionFile):
        # Existing file-convention semantic, unchanged.
        # Walk CWD → git root, set `discovered[convention.path] = candidate`
        # only if the path key is not yet seen.
        ...
    elif isinstance(convention, ConventionDirectory):
        # Walk CWD → git root, at each level glob within `convention.path`,
        # apply `convention.include_file` if set, key by relative-path-within-dir.
        # Closest-wins per relative-path key.
        ...
    
    # Append this convention's discoveries in deterministic order.
    result.extend(sorted(discovered.values(), key=lambda p: p.name))
return result
```

**Why per-convention state**: prevents `.github/instructions/style.instructions.md` from shadowing a hypothetical `.cursor/rules/style.instructions.md` (different conventions, same basename). Each convention has its own `discovered` dict.

**Backward-compat for `CONVENTION_FILES`**: kept unconditionally as a module-level alias. The symbol is not in `__all__` but is module-public (importable directly); the alias is the principled compatibility layer, not a hack.

#### A.1 Refactor scope (truth-grounded, verified via grep)

Source references to `CONVENTION_FILES`:
- `src/conductor/config/instructions.py:21` (definition)
- `src/conductor/config/instructions.py:79` (loop)
- `src/conductor/config/instructions.py:92-93` (deterministic return)

Test references:
- `tests/test_config/test_instructions.py:64` — `def test_discovers_all_convention_files`
- `tests/test_config/test_instructions.py:73` — comment: "# Deterministic order matches CONVENTION_FILES"

**No actual test code references the constant**; the test name and comment refer to the *concept*, not the symbol. Tests exercise discovery output, not the constant.

So existing 38 tests should pass unchanged; the test comment may be updated to reference `CONVENTIONS` or the comment kept as-is (since `CONVENTION_FILES` remains as a backward-compat alias).

#### B. Existing file-convention semantics preserved

For `ConventionFile` entries, walker behavior is **byte-identical** to PR #141's:
- Walks CWD → git root
- Closest-wins per `path` value
- Returns paths in `CONVENTIONS` declaration order (same as today)

The 38 existing tests pass with no logic change beyond the constant rename (or with a backward-compat alias).

#### C. Directory-convention semantics

For `ConventionDirectory` entries, walker behavior:

- At each walked level, check if `current_level / convention.path` is a directory
- If yes, glob within (recursively if `recursive=True`) for files matching `convention.pattern`
- For each matched file, if `convention.include_file` is set, apply it; skip files where the predicate returns False
- Track candidates by relative-path-within-the-convention-directory; closest-wins per relative-path
- After walk completes, return the closest set of candidates

**Closest-wins per relative path** (NOT per directory). Concrete example:

```
repo/
├── .github/instructions/style.instructions.md          ← root has style + lang/csharp
│                       /lang/csharp.instructions.md
└── subproject/
    └── .github/instructions/lang/csharp.instructions.md ← subproject has lang/csharp ONLY
```

Discovery from `subproject/`:
- `lang/csharp.instructions.md` — closest is subproject's; root's is shadowed
- `style.instructions.md` — only at root; loaded
- Result: 2 files (subproject's `lang/csharp` + root's `style`)

This matches the existing per-file convention spirit: same logical unit shadowed; non-overlapping units coexist.

#### D. `applyTo` filter (truth-grounded, robust)

Use `ruamel.yaml` (already a conductor dependency at `pyproject.toml:37` `ruamel.yaml>=0.18.0`; verified via `src/conductor/config/loader.py:15-17`). Do NOT add PyYAML.

```python
import re
from pathlib import Path
from ruamel.yaml import YAML
from ruamel.yaml.error import YAMLError

# Tolerant regex: handles CRLF, optional trailing whitespace on delimiters,
# closing --- at EOF without trailing newline.
_FRONTMATTER_RE = re.compile(
    r"\A---[ \t]*\r?\n(.*?)\r?\n---[ \t]*(?:\r?\n|\Z)",
    re.DOTALL,
)

# Module-level YAML parser (safe mode; does not execute custom constructors).
_yaml = YAML(typ="safe")


def _is_always_on_instructions_file(path: Path) -> bool:
    """Return True if a .github/instructions/*.instructions.md file is always-on
    per GitHub's documented convention semantics.

    Per https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses
    and https://code.visualstudio.com/docs/copilot/customization/custom-instructions:
    - applyTo: "**" → always applied (INCLUDE in conductor preamble)
    - applyTo: "<other glob>" → scoped (SKIP)
    - applyTo absent → "not applied automatically; manual attach only" (SKIP)

    Conductor's preamble is always-on for every agent. To honor the convention,
    only files explicitly marked applyTo: "**" are loaded.
    """
    try:
        # 'utf-8-sig' transparently strips a leading BOM if present
        text = path.read_text(encoding="utf-8-sig")
    except (OSError, UnicodeDecodeError) as e:
        logger.debug("Cannot read %s for frontmatter check: %s", path, e)
        return False

    match = _FRONTMATTER_RE.match(text)
    if not match:
        return False  # no frontmatter → manual-attach default; skip

    try:
        fm = _yaml.load(match.group(1))
    except YAMLError as e:
        logger.warning("Failed to parse frontmatter in %s: %s; skipping", path, e)
        return False

    # YAML may parse to non-dict (empty frontmatter, list, scalar). Skip safely.
    if not isinstance(fm, dict):
        return False

    return fm.get("applyTo") == "**"
```

**Edge cases covered (and tested)**:
- CRLF line endings (Windows-authored files)
- UTF-8 BOM (handled via `utf-8-sig` encoding)
- Closing `---` at EOF without trailing newline
- Non-dict YAML (empty frontmatter, list value, scalar)
- Malformed YAML (logged warning + skip, not crash)
- `applyTo: "**"` with single quotes, double quotes, or unquoted strings (all valid YAML)

**Dependency note**: conductor uses `ruamel.yaml>=0.18.0` (declared at `pyproject.toml:37`; imported at `src/conductor/config/loader.py:15-17`). No new dependency.

#### E. Symlink policy

Existing file convention discovery uses `Path.is_file()` which follows symlinks (verified at `instructions.py:83`). For directory recursion in this PR, use `os.walk(followlinks=False)` to:
- Prevent symlink loops in `.github/instructions/` (a symlinked subdirectory pointing back into the tree)
- Prevent out-of-tree expansion (a symlink pointing to `/etc` etc.)

**Explicit policy**:
- Symlinked instruction FILES inside `.github/instructions/` are read like regular files (`Path.read_text()` follows symlinks)
- Symlinked DIRECTORIES inside `.github/instructions/` are NOT traversed

Tested explicitly.

#### F. Strict opt-in flag (single existing flag)

Single flag `--workspace-instructions` (no new flag). Backward-compat audit:

| Pre-existing behavior | After this PR |
|---|---|
| `--workspace-instructions` discovers 3 file conventions | Unchanged for repos without `.github/instructions/` |
| Repo with `.github/instructions/` containing only scoped/manual files | Identical (all filtered out) |
| Repo with always-on files (`applyTo: "**"`) in `.github/instructions/` | Those files added to preamble (intentional; matches convention semantics) |

The behavior change in the always-on case is intentional. A user who set `applyTo: "**"` already opted into "always apply" by the convention's rules.

#### G. Aggregate ordering (deterministic)

After this PR, `discover_workspace_instructions(start_dir)` returns paths in `CONVENTIONS` declaration order:

1. `AGENTS.md` (closest, if found)
2. `.github/copilot-instructions.md` (closest, if found)
3. `CLAUDE.md` (closest, if found)
4. `.github/instructions/*.instructions.md` (always-on subset, sorted by relative path within directory)

The first 3 are byte-identical to PR #141's existing output for repos without `.github/instructions/`.

---

## Files changed

| File | Change | LoC estimate |
|---|---|---|
| `src/conductor/config/instructions.py` | Refactor `CONVENTION_FILES` → polymorphic `CONVENTIONS` list with `ConventionFile` / `ConventionDirectory` types; add `_is_always_on_instructions_file()` helper; generalize walker to dispatch on type | +120 / -25 |
| `src/conductor/cli/app.py` | Update `--workspace-instructions` help text to mention new convention | +1 / -1 |
| `tests/test_config/test_instructions.py` | Update existing 38 tests' references from `CONVENTION_FILES` to `CONVENTIONS` (or maintain `CONVENTION_FILES = [c.path for c in CONVENTIONS if isinstance(c, ConventionFile)]` as a backward-compat alias if cleaner); add 14 new tests in 1 new class | +200 / -10 |
| `docs/cli-reference.md` | Document the new convention | +5 / -0 |
| `docs/workflow-syntax.md` | Note: `.github/instructions/` is auto-discovered with `applyTo` filtering | +5 / -0 |

Total: ~330 net new LoC. Refactor is bounded; existing 38 tests pass with reference update only.

**No changes to**: schema, engine, executor, checkpoint, sub-workflow merging, YAML loader, bg_runner, resume command. All PR #141 foundation reused.

---

## Test plan (14 new + maintenance for existing)

### Existing 38 tests

Continue to pass. Most reference `discover_workspace_instructions()` behavior, not the constant directly. Tests that DO reference `CONVENTION_FILES` (TBD via re-grep at impl time; likely 0-2) get updated to reference `CONVENTIONS` or a backward-compat alias.

### New `TestDiscoverGithubInstructionsDir` (10 tests)

1. `test_discovers_always_on_file` — `applyTo: "**"` → loaded
2. `test_skips_scoped_file` — `applyTo: "**/*.ts"` → NOT loaded
3. `test_skips_no_frontmatter` — no frontmatter → NOT loaded (manual-attach default)
4. `test_skips_no_apply_to` — frontmatter present but `applyTo` absent → NOT loaded
5. `test_recursive_subdirs` — `lang/csharp.instructions.md` with `applyTo: "**"` → loaded
6. `test_closest_wins_per_relative_path` — root + subproject mix; correct shadowing per relative path
7. `test_root_only_file_loads_when_subproject_missing` — no shadow without counterpart
8. `test_symlinked_directory_not_traversed` — `os.walk(followlinks=False)` validated
9. `test_directory_and_file_conventions_coexist` — `AGENTS.md` AND always-on file both loaded
10. `test_pattern_must_match` — `.github/instructions/foo.md` (no `.instructions.md`) → NOT loaded

### New `TestFrontmatterRobustness` (4 tests)

11. `test_crlf_line_endings` — file with `\r\n`, valid `applyTo: "**"` → loaded
12. `test_utf8_bom_handling` — file with BOM, valid `applyTo: "**"` → loaded
13. `test_malformed_yaml_skipped_with_warning` — invalid YAML → skipped, warning logged
14. `test_non_dict_yaml_skipped` — frontmatter parses to list/scalar → skipped, no AttributeError

---

## Risks and explicit deferrals

| Risk | Severity | Mitigation in this PR | Defer rationale |
|---|---|---|---|
| Total preamble size > 50KB | Low | Existing warning fires | Already handled in PR #141 |
| Symlink loops | Low | `os.walk(followlinks=False)` + test | Explicit policy, not deferred |
| `applyTo: "**/*.cs"` files have global value (some teams) | Med | NOT supported; conservative-correct | Future `--include-scoped-instructions` if demand emerges |
| User wants `.cursor/rules/` etc. discovered | Med | NOT in scope | Future PRs add via polymorphic list |
| `applyTo`-absent files conservatively skipped | Low | Documented; matches docs verbatim | Future `--include-unscoped-instructions` if demand emerges |
| Refactor breaks public API | Low | `CONVENTION_FILES` not in `__all__`; verified internal only | If found to be exported in practice, add backward-compat alias |

---

## Future PRs (enabled by polymorphic shape)

Each future convention is a minimal addition:

```python
# Cursor (.cursor/rules/*.mdc)
def _is_cursor_alwaysapply(path: Path) -> bool:
    fm = _parse_frontmatter(path)
    if fm is None: return False
    # Cursor rule types: "Always", "Auto Attached", "Agent Requested", "Manual"
    # Always: alwaysApply: true
    return fm.get("alwaysApply") is True

CONVENTIONS.append(
    ConventionDirectory(".cursor/rules", "*.mdc", _is_cursor_alwaysapply)
)
```

```python
# Cline (.clinerules/*.md, NON-recursive per docs)
def _is_cline_unconditional(path: Path) -> bool:
    fm = _parse_frontmatter(path)
    # Cline: paths frontmatter for conditional; no paths = unconditional
    return fm is None or "paths" not in fm

CONVENTIONS.append(
    ConventionDirectory(".clinerules", "*.md", _is_cline_unconditional, recursive=False)
)
```

Each future PR is small (10-20 LoC) and follows the same pattern: filter function + list entry + tests.

---

## PR Title and Description

### PR Title

```
feat(instructions): polymorphic conventions + auto-discover .github/instructions/ with applyTo filtering
```

### PR Body

```markdown
Closes #<issue-number>

## What

1. Refactors `CONVENTION_FILES: list[str]` into a polymorphic `CONVENTIONS:
   list[Convention]` using `ConventionFile | ConventionDirectory` union types.
   Generalizes the walker to dispatch on type.

2. Adds `.github/instructions/*.instructions.md` as a `ConventionDirectory`
   with `applyTo` frontmatter filtering — only files marked always-on
   (`applyTo: "**"`) are loaded.

## Why

Today's `--workspace-instructions` discovers exactly 3 hard-coded file
conventions. GitHub Copilot itself already discovers more (per the
[GitHub Copilot Chat docs](https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses)
and [VS Code Copilot docs](https://code.visualstudio.com/docs/copilot/customization/custom-instructions);
the convention is `.github/instructions/**/*.instructions.md`).
in git root & cwd). This PR closes the gap.

The polymorphic refactor primes future PRs that add other directory
conventions (Cursor, Cline, Continue, Windsurf, Claude rules-dir) — each
becomes a 10-line addition (filter function + list entry + tests) without
walker changes.

## Why honor `applyTo`?

Conductor's preamble is always-on for every agent. The convention says
files with scoped `applyTo` (e.g., `applyTo: "**/*.ts"`) should only apply
to TypeScript files. Loading them globally would violate the convention.

Conservative-correct: only `applyTo: "**"` files are loaded. Files with
no `applyTo` are skipped (the convention treats them as manual-attach,
per https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses).

## Tests

14 new tests (10 in TestDiscoverGithubInstructionsDir + 4 in
TestFrontmatterRobustness). Existing 38 tests continue to pass.

## Backward compatibility

- Repos without `.github/instructions/` → identical behavior
- Repos with the directory containing only scoped/manual files → identical behavior
- Repos with always-on files (`applyTo: "**"`) → those files added to preamble
  (intentional; matches convention semantics)

`CONVENTION_FILES` is not in `__all__`; refactor is internal. If found to be
imported by downstream consumers, a backward-compat alias can be added.

## Out of scope (deferred to future PRs)

- Other directory conventions (Cursor, Claude rules-dir, Continue, Windsurf, Cline)
- `--instructions <directory>` CLI support
- YAML `!file <directory>` error message improvement
- `--include-unscoped-instructions` / `--include-scoped-instructions` flags

Each is small enough for its own PR; bundling expands scope.
```

---

## Issue Title and Body (for you to file at microsoft/conductor)

### Issue Title

```
enhancement(instructions): polymorphic conventions + auto-discover .github/instructions/ with applyTo filtering
```

### Issue Body

```markdown
## Summary

Two changes proposed:

1. Refactor today's `CONVENTION_FILES: list[str]` (single-purpose constant)
   into a polymorphic `CONVENTIONS: list[Convention]` with `ConventionFile`
   and `ConventionDirectory` types. Walker dispatches on type.

2. Add `.github/instructions/*.instructions.md` as a `ConventionDirectory`
   with `applyTo` frontmatter filtering — only files marked always-on
   (`applyTo: "**"`) are loaded.

## Why

Today's `--workspace-instructions` discovers 3 hard-coded files. GitHub
Copilot itself already discovers more — including `.github/instructions/`
files (per [GitHub Copilot Chat docs](https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses) and [VS Code Copilot docs](https://code.visualstudio.com/docs/copilot/customization/custom-instructions)).

The polymorphic refactor pays a one-time cost so future conventions
(Cursor's `.cursor/rules/`, Cline's `.clinerules/`, etc.) can be added
via filter function + list entry, without walker changes.

## Convention reference (truth-grounded)

GitHub Copilot's `.github/instructions/` convention is supported across:
- Copilot Chat on github.com — https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses
- Copilot in VS Code — https://code.visualstudio.com/docs/copilot/customization/custom-instructions
- Copilot CLI — supports the same convention (shared agentic harness with VS Code Copilot); `--no-custom-instructions` and `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` env-var verifiable in CLI

`applyTo` semantics:
- `applyTo: "**"` → always applied (INCLUDE)
- `applyTo: "<other>"` → scoped (SKIP)
- `applyTo` absent → manual-attach (SKIP)

## Implementation outline

`src/conductor/config/instructions.py`:
- Replace `CONVENTION_FILES: list[str]` with `CONVENTIONS: list[Convention]`
- Add `ConventionFile` and `ConventionDirectory` dataclasses
- Add `_is_always_on_instructions_file()` filter function
- Generalize walker to dispatch on type

Update help text in `cli/app.py:309-320`.

Add 14 new tests in `tests/test_config/test_instructions.py`. Existing 38
tests continue to pass (with possible reference update from `CONVENTION_FILES`
to `CONVENTIONS`).

## Backward compatibility

- Repos without `.github/instructions/` → identical behavior
- `CONVENTION_FILES` is internal (not in `__all__`); rename safe

## Estimated diff

~120 net LoC in `instructions.py`, ~200 LoC in tests, ~10 LoC docs/help-text.

## Implementation availability

Ready on `franklixuefei/conductor:feat/discover-github-instructions-dir`.
(I'm an enterprise-managed user and cannot fork microsoft/conductor directly.)

## Out of scope (deferred)

- Other directory conventions (each needs its own PR with proper activation semantics)
- `--instructions <directory>` CLI support
- `--include-unscoped-instructions` / `--include-scoped-instructions` flags
```

---

## Truth-grounded references

### Conductor source (verified line-by-line at `franklixuefei/conductor:main`)

| Claim | Source |
|---|---|
| Existing `CONVENTION_FILES` has 3 entries | `src/conductor/config/instructions.py:21-25` |
| Discovery walks CWD → git-root | `instructions.py:_find_git_root` + `instructions.py:78-90` |
| `is_file()` follows symlinks | `instructions.py:83` |
| `CONVENTION_FILES` not exported in `__all__` | `src/conductor/config/__init__.py` (no instructions imports) |
| Existing 38 tests in 8 classes | `tests/test_config/test_instructions.py` |
| `Path.cwd()` used for discovery | `cli/run.py:1223` |
| Sub-workflow merging unwraps + concats + rewraps | `engine/workflow.py:794-797` |
| Checkpoint backward compat via `.get()` | `engine/checkpoint.py:319` |
| `WorkflowDef.instructions: list[str]` | `config/schema.py:964-984` |

### GitHub Copilot convention sources

- Copilot Chat docs: https://docs.github.com/en/copilot/customizing-copilot/about-customizing-github-copilot-chat-responses
- VS Code Copilot docs: https://code.visualstudio.com/docs/copilot/customization/custom-instructions
- Copilot CLI: convention support shared with VS Code Copilot (same agentic harness); `--no-custom-instructions` and `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` env-var verifiable in CLI

---

## Review checklist

- [ ] **Polymorphic refactor justified**: pays small one-time cost; future conventions become 10-LoC additions. Agree?
- [ ] **`Convention = Union[ConventionFile, ConventionDirectory]`**: simplest discriminated union; no enum/factory pattern needed. Agree?
- [ ] **`include_file` as `Callable[[Path], bool]`**: pure function per directory; testable in isolation. Agree?
- [ ] **`recursive: bool = True` default**: matches GitHub's recursive convention; defaultable for future Cline-style flat directories. Agree?
- [ ] **Existing 38 tests pass with reference update**: rename `CONVENTION_FILES` (or alias). Agree?
- [ ] **Honor `applyTo` per docs verbatim**: only `applyTo: "**"` loaded. Agree?
- [ ] **Conservative skip for `applyTo` absent**: matches "manual attach" docs. Agree (vs. include by default)?
- [ ] **Closest-wins per relative path** within directory. Agree (vs. whole-dir shadow)?
- [ ] **`os.walk(followlinks=False)`**: explicit symlink policy. Agree?
- [ ] **Single `--workspace-instructions` flag** (no new flag): existing flag's contract is "discover known conventions"; this PR adds one. Agree?
- [ ] **14 new tests sufficient**. Agree?

When you sign off:

1. I write 14 tests + implementation
2. Self-review pass against same standards
3. Push to `franklixuefei/conductor:feat/discover-github-instructions-dir`
4. You file the issue at `microsoft/conductor` using body above
5. You file the PR (description above)
6. Octane integration ships separately, dependent on this PR's release


Claim	Source
Existing `CONVENTION_FILES` has 3 entries	`src/conductor/config/instructions.py:21-25`
Discovery walks CWD → git-root	`instructions.py:_find_git_root` + `instructions.py:78-90`
`is_file()` follows symlinks	`instructions.py:83`
`CONVENTION_FILES` not exported in `__all__`	`src/conductor/config/__init__.py` (no instructions imports)
Existing 38 tests in 8 classes	`tests/test_config/test_instructions.py`
`Path.cwd()` used for discovery	`cli/run.py:1223`
Sub-workflow merging unwraps + concats + rewraps	`engine/workflow.py:794-797`
Checkpoint backward compat via `.get()`	`engine/checkpoint.py:319`
`WorkflowDef.instructions: list[str]`	`config/schema.py:964-984`

Capability	Source	This PR
`--workspace-instructions` flag	`cli/app.py:309-320`	Same; help-text update for new convention
Git-root walk (`_find_git_root` + walker loop)	`instructions.py:_find_git_root` + `instructions.py:78-90`	Walker generalized to dispatch on Convention type; `_find_git_root` reused as-is
Closest-wins per filename	`instructions.py:80-86`	Generalized: closest-wins-per-key, where key is filename for files, relative-path-within-dir for directories
`load_instruction_files()` content reader	`instructions.py:96-130`	Reused as-is
`<workspace_instructions>` wrapping	`instructions.py:_wrap_preamble`	Reused as-is
3-source merge (auto + YAML + CLI)	`instructions.py:build_inner_instructions`	Reused as-is
Engine preamble injection	`executor/agent.py:157-166`	Untouched
Sub-workflow merging	`engine/workflow.py:794-797`	Untouched
Checkpoint persistence	`engine/checkpoint.py:319`	Untouched
Bg-runner subprocess flag forwarding	`cli/bg_runner.py:199-204`	Untouched
Resume command	`cli/app.py:resume()`	Untouched
`WorkflowDef.instructions: list[str]`	`config/schema.py:964-984`	Untouched

Pre-existing behavior	After this PR
`--workspace-instructions` discovers 3 file conventions	Unchanged for repos without `.github/instructions/`
Repo with `.github/instructions/` containing only scoped/manual files	Identical (all filtered out)
Repo with always-on files (`applyTo: "**"`) in `.github/instructions/`	Those files added to preamble (intentional; matches convention semantics)

File	Change	LoC estimate
`src/conductor/config/instructions.py`	Refactor `CONVENTION_FILES` → polymorphic `CONVENTIONS` list with `ConventionFile` / `ConventionDirectory` types; add `_is_always_on_instructions_file()` helper; generalize walker to dispatch on type	+120 / -25
`src/conductor/cli/app.py`	Update `--workspace-instructions` help text to mention new convention	+1 / -1
`tests/test_config/test_instructions.py`	Update existing 38 tests' references from `CONVENTION_FILES` to `CONVENTIONS` (or maintain `CONVENTION_FILES = [c.path for c in CONVENTIONS if isinstance(c, ConventionFile)]` as a backward-compat alias if cleaner); add 14 new tests in 1 new class	+200 / -10
`docs/cli-reference.md`	Document the new convention	+5 / -0
`docs/workflow-syntax.md`	Note: `.github/instructions/` is auto-discovered with `applyTo` filtering	+5 / -0

Risk	Severity	Mitigation in this PR	Defer rationale
Total preamble size > 50KB	Low	Existing warning fires	Already handled in PR #141
Symlink loops	Low	`os.walk(followlinks=False)` + test	Explicit policy, not deferred
`applyTo: "*/.cs"` files have global value (some teams)	Med	NOT supported; conservative-correct	Future `--include-scoped-instructions` if demand emerges
User wants `.cursor/rules/` etc. discovered	Med	NOT in scope	Future PRs add via polymorphic list
`applyTo`-absent files conservatively skipped	Low	Documented; matches docs verbatim	Future `--include-unscoped-instructions` if demand emerges
Refactor breaks public API	Low	`CONVENTION_FILES` not in `__all__`; verified internal only	If found to be exported in practice, add backward-compat alias

Conductor Design — Polymorphic conventions: extend --workspace-instructions to recognize .github/instructions/*.instructions.md with applyTo-aware filtering #168

Description

TL;DR

Why polymorphic (and why now)

Building on PR #141 (verified line-by-line)

The Convention (truth-grounded)

Design

Hard standards (user-imposed)

Architectural choices

A. Polymorphic Convention types

A.1 Refactor scope (truth-grounded, verified via grep)

B. Existing file-convention semantics preserved

C. Directory-convention semantics

D. applyTo filter (truth-grounded, robust)

E. Symlink policy

F. Strict opt-in flag (single existing flag)

G. Aggregate ordering (deterministic)

Files changed

Test plan (14 new + maintenance for existing)

Existing 38 tests

New TestDiscoverGithubInstructionsDir (10 tests)

New TestFrontmatterRobustness (4 tests)

Risks and explicit deferrals

Future PRs (enabled by polymorphic shape)

PR Title and Description

PR Title

PR Body

Issue Title and Body (for you to file at microsoft/conductor)

Issue Title

Issue Body

Truth-grounded references

Conductor source (verified line-by-line at franklixuefei/conductor:main)

GitHub Copilot convention sources

Review checklist

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Conductor Design — Polymorphic conventions: extend `--workspace-instructions` to recognize `.github/instructions/*.instructions.md` with applyTo-aware filtering #168

A. Polymorphic `Convention` types

D. `applyTo` filter (truth-grounded, robust)

New `TestDiscoverGithubInstructionsDir` (10 tests)

New `TestFrontmatterRobustness` (4 tests)

Conductor source (verified line-by-line at `franklixuefei/conductor:main`)