Add resilient filesystem error handling with retry logic

[gregpriday]

Summary

Implement automatic retry with exponential backoff for transient filesystem errors (EBUSY, EPERM, EMFILE, EAGAIN) to improve reliability on Windows, network drives, and under high file descriptor pressure.

Current Behavior

Filesystem operations silently fail or return null when encountering transient errors:

In ignoreWalker.js:

try {
  entries = await fs.readdir(dir, { withFileTypes: true });
} catch (error) {
  // Can't read directory - skip it
  return;
}

In fileLoader.js:

} catch (error) {
  logger.error('Failed to load file', {
    path: relativePath,
    error: error.message,
  });
  return null;  // File silently skipped
}

Common failure scenarios:

• Windows: Antivirus holds file locks → EBUSY, EPERM
• Network drives: Temporary disconnects → ETIMEDOUT, EIO
• High concurrency: File descriptor limits → EMFILE, ENFILE
• Race conditions: Files being written → ENOENT (transient)

Expected Behavior

Transient filesystem errors should trigger automatic retry with exponential backoff:

1. First attempt fails with EBUSY
2. Wait 100ms, retry (attempt 2)
3. Wait 200ms, retry (attempt 3)
4. Wait 400ms, retry (attempt 4)
5. If all retries fail, log detailed error and continue (or fail based on --fail-on-errors policy)

Environment

This issue affects all platforms but is most critical on:

• Windows (antivirus, indexing services, file locks)
• macOS with aggressive Spotlight indexing
• Network-mounted filesystems (NFS, SMB, SSHFS)
• CI environments with high concurrency

Evidence

Existing retry infrastructure (AI services only):

• src/utils/errors.js - Defines RETRYABLE_ERROR_CODES
• src/utils/errors.js - isRetryableError() function

Current codes (network-focused):

export const RETRYABLE_ERROR_CODES = [
  'RATE_LIMIT', 'TIMEOUT', 'SERVICE_UNAVAILABLE',
  'NETWORK_ERROR', 'TEMPORARY_FAILURE',
  'ENOTFOUND', 'ETIMEDOUT', 'ECONNRESET', 'ECONNABORTED'
];

Missing filesystem codes:

• EBUSY - Resource busy (file locked)
• EPERM - Permission denied (transient on Windows)
• EMFILE - Too many open files
• ENFILE - File table overflow
• EAGAIN - Resource temporarily unavailable
• EIO - I/O error (network drives)

Affected files:

• src/utils/ignoreWalker.js - readdir() failures
• src/utils/ignoreWalker.js - stat() failures
• src/utils/fileLoader.js - stat() and readFile() failures
• src/utils/fileLoader.js - Binary file fallback handling

Root Cause

• No retry logic for filesystem operations (only for AI service calls)
• Silent failures instead of structured error aggregation
• Binary fallback (line 91-116) masks underlying transient errors

Proposed Solution

1. Add Filesystem Error Codes

Update src/utils/errors.js:

export const RETRYABLE_ERROR_CODES = [
  // Network (existing)
  'RATE_LIMIT', 'TIMEOUT', 'SERVICE_UNAVAILABLE', 
  'ENOTFOUND', 'ETIMEDOUT', 'ECONNRESET', 'ECONNABORTED',
  
  // Filesystem (new)
  'EBUSY',    // Resource busy
  'EPERM',    // Permission denied (Windows antivirus)
  'EMFILE',   // Too many open files
  'ENFILE',   // File table overflow
  'EAGAIN',   // Resource temporarily unavailable
  'EIO',      // I/O error (network drives)
];

2. Create Retry Utility

New file: src/utils/retryableFs.js

import { isRetryableError } from './errors.js';

export async function withRetry(operation, options = {}) {
  const maxAttempts = options.maxAttempts || 3;
  const initialDelay = options.initialDelay || 100;
  const maxDelay = options.maxDelay || 2000;
  
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxAttempts || !isRetryableError(error)) {
        throw error;
      }
      
      const delay = Math.min(initialDelay * Math.pow(2, attempt - 1), maxDelay);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

3. Apply to Filesystem Operations

Wrap critical operations in withRetry():

// In ignoreWalker.js
entries = await withRetry(() => fs.readdir(dir, { withFileTypes: true }));

// In fileLoader.js
const stats = await withRetry(() => fs.stat(fullPath));
const content = await withRetry(() => fs.readFile(fullPath, 'utf8'));

4. Error Aggregation

Replace silent failures with structured error collection:

// Track errors for final report
const errors = {
  retried: [],    // Succeeded after retry
  failed: [],     // Failed after all retries
  permanent: []   // Non-retryable errors
};

Tasks

• Add filesystem error codes to RETRYABLE_ERROR_CODES
• Create src/utils/retryableFs.js with withRetry() utility
• Add retry wrapper to readdir() in ignoreWalker.js:131
• Add retry wrapper to stat() in ignoreWalker.js:149
• Add retry wrapper to stat() in fileLoader.js:59
• Add retry wrapper to readFile() in fileLoader.js:72
• Implement error aggregation and reporting
• Add --fail-on-fs-errors CLI flag for strict mode
• Add configuration option: fs.retryAttempts, fs.retryDelay
• Write tests for retry logic (mock filesystem errors)
• Write tests for error aggregation
• Update troubleshooting docs with retry behavior

Acceptance Criteria

• Transient errors (EBUSY, EPERM, EMFILE) trigger up to 3 retry attempts
• Exponential backoff delays: 100ms, 200ms, 400ms (configurable)
• Maximum delay capped at 2000ms to prevent excessive waits
• Non-retryable errors (ENOENT permanent, EISDIR) fail immediately
• Error summary includes: total retries, successes after retry, permanent failures
• --fail-on-fs-errors flag causes exit code 1 if any files fail
• Retry behavior is configurable via copytree.fs.retryAttempts and copytree.fs.retryDelay
• Tests cover Windows-specific scenarios (EBUSY, EPERM)
• Tests cover file descriptor exhaustion (EMFILE, ENFILE)

Additional Context

Windows antivirus impact:
Windows Defender and third-party antivirus can hold file locks for 50-500ms while scanning, causing EBUSY/EPERM errors that resolve after brief delays.

File descriptor limits:

• macOS default: 256 per process
• Linux default: 1024 per process
• CI environments: Often lower limits

Related tools with retry:

• rsync: Built-in retry for transient errors
• aws-cli: Automatic retry with exponential backoff
• rclone: Configurable retry logic

[gregpriday]

• Expand retryable error codes to include filesystem transients.
• Introduce a withFsRetry() utility with exponential backoff.
• Wrap readdir, stat, and readFile in retry logic in walker/loader.
• Aggregate and report FS errors (retried, failed, permanent) at the end.
• Add config (copytree.fs.*) and --fail-on-fs-errors CLI flag.
• Add targeted unit/integration tests and update docs.

Goal

Implement automatic retry with exponential backoff for transient filesystem errors so directory walking and file loading become resilient on Windows, network filesystems, and under FD pressure, while providing structured error reporting and an optional strict failure mode.

Approach Overview

• Extend retryable codes in [src/utils/errors.js](https://github.com/gregpriday/copytree/issues/src/utils/errors.js) and expose a FS-specific predicate.
• Add src/utils/retryableFs.js to centralize exponential backoff with caps and optional jitter.
• Wrap FS calls in [src/utils/ignoreWalker.js](https://github.com/gregpriday/copytree/issues/src/utils/ignoreWalker.js) and [src/utils/fileLoader.js](https://github.com/gregpriday/copytree/issues/src/utils/fileLoader.js).
• Add lightweight error aggregation in src/utils/fsErrorReport.js, and print a summary in src/cli/index.js.
• Introduce config options (copytree.fs.retryAttempts, copytree.fs.retryDelay, copytree.fs.maxDelay) and a --fail-on-fs-errors flag.
• Add tests in tests/utils/retryableFs.test.js, tests/utils/fsErrorReport.test.js, and tests/integration/fs-retry.e2e.test.js.

Step-by-Step Plan

1. Retryable error codes and predicate

   • Edits: [src/utils/errors.js](https://github.com/gregpriday/copytree/issues/src/utils/errors.js)

   • Actions: Add FS codes and expose isRetryableFsError(code: string).

   • Snippet:

     --- a/src/utils/errors.js
     +++ b/src/utils/errors.js
     @@
     -export const RETRYABLE_ERROR_CODES = [
     -  'RATE_LIMIT', 'TIMEOUT', 'SERVICE_UNAVAILABLE',
     -  'NETWORK_ERROR', 'TEMPORARY_FAILURE',
     -  'ENOTFOUND', 'ETIMEDOUT', 'ECONNRESET', 'ECONNABORTED'
     -];
     +export const RETRYABLE_ERROR_CODES = [
     +  // Network (existing)
     +  'RATE_LIMIT', 'TIMEOUT', 'SERVICE_UNAVAILABLE',
     +  'NETWORK_ERROR', 'TEMPORARY_FAILURE',
     +  'ENOTFOUND', 'ETIMEDOUT', 'ECONNRESET', 'ECONNABORTED',
     +  // Filesystem (new)
     +  'EBUSY', 'EPERM', 'EMFILE', 'ENFILE', 'EAGAIN', 'EIO'
     +];
     +
     +export function isRetryableFsError(err) {
     +  const code = err?.code || err;
     +  return RETRYABLE_ERROR_CODES.includes(code);
     +}

   • Verification: Run lint/type checks; ensure no circular imports: pnpm -w run build (or npm run build).

2. Add withFsRetry() utility

   • Edits: src/utils/retryableFs.js (new)

   • Actions: Implement exponential backoff (initialDelay, maxDelay, maxAttempts) and hooks for aggregation.

   • Snippet:

     // src/utils/retryableFs.js
     import { isRetryableFsError } from './errors.js';
     
     export async function withFsRetry(op, {
       maxAttempts = 3,
       initialDelay = 100,
       maxDelay = 2000,
       jitter = false,
       onRetry = () => {},
       signal
     } = {}) {
       let attempt = 0;
       // eslint-disable-next-line no-constant-condition
       while (true) {
         attempt++;
         try {
           if (signal?.aborted) throw Object.assign(new Error('Aborted'), { code: 'ABORT_ERR' });
           return await op();
         } catch (err) {
           const retryable = isRetryableFsError(err);
           if (!retryable || attempt >= maxAttempts) throw err;
           const base = Math.min(initialDelay * 2 ** (attempt - 1), maxDelay);
           const delay = jitter ? Math.floor(base * (0.5 + Math.random())) : base;
           await onRetry({ attempt, delay, code: err.code });
           await new Promise(r => setTimeout(r, delay));
         }
       }
     }

   • Verification: Add a quick local smoke test (run a dummy op throwing EBUSY twice, success on third).

3. Aggregate FS errors

   • Edits: src/utils/fsErrorReport.js (new)

   • Actions: Track retried attempts, successes after retry, and failures; expose recordRetry, recordFailure, recordPermanent, and summarize.

   • Snippet:

     // src/utils/fsErrorReport.js
     const stats = {
       totalRetries: 0,
       succeededAfterRetry: 0,
       failed: 0,
       permanent: 0,
       byPath: new Map(), // path -> { retries, lastCode, status }
     };
     
     export function recordRetry(path, code) {
       stats.totalRetries++;
       const s = stats.byPath.get(path) ?? { retries: 0, lastCode: null, status: 'pending' };
       s.retries++; s.lastCode = code;
       stats.byPath.set(path, s);
     }
     
     export function recordGiveUp(path, code) {
       stats.failed++;
       const s = stats.byPath.get(path) ?? { retries: 0, lastCode: null, status: 'pending' };
       s.lastCode = code; s.status = 'failed';
       stats.byPath.set(path, s);
     }
     
     export function recordPermanent(path, code) {
       stats.permanent++;
       const s = stats.byPath.get(path) ?? { retries: 0, lastCode: null, status: 'pending' };
       s.lastCode = code; s.status = 'permanent';
       stats.byPath.set(path, s);
     }
     
     export function recordSuccessAfterRetry(path) {
       const s = stats.byPath.get(path);
       if (s && s.retries > 0 && s.status === 'pending') {
         stats.succeededAfterRetry++;
         s.status = 'ok';
         stats.byPath.set(path, s);
       }
     }
     
     export function summarize() {
       return {
         totalRetries: stats.totalRetries,
         succeededAfterRetry: stats.succeededAfterRetry,
         failed: stats.failed,
         permanent: stats.permanent,
       };
     }

   • Verification: Import the module and exercise functions in a REPL/test.

4. Wrap FS calls in ignoreWalker.js

   • Edits: [src/utils/ignoreWalker.js](https://github.com/gregpriday/copytree/issues/src/utils/ignoreWalker.js)

   • Actions: Use withFsRetry() for readdir and stat. Replace silent skips with aggregation.

   • Snippet:

     --- a/src/utils/ignoreWalker.js
     +++ b/src/utils/ignoreWalker.js
     @@
     -  try {
     -    entries = await fs.readdir(dir, { withFileTypes: true });
     -  } catch (error) {
     -    // Can't read directory - skip it
     -    return;
     -  }
     +  try {
     +    entries = await withFsRetry(
     +      () => fs.readdir(dir, { withFileTypes: true }),
     +      {
     +        maxAttempts: config.copytree?.fs?.retryAttempts ?? 3,
     +        initialDelay: config.copytree?.fs?.retryDelay ?? 100,
     +        maxDelay: config.copytree?.fs?.maxDelay ?? 2000,
     +        onRetry: ({ attempt, code }) => recordRetry(dir, code),
     +      }
     +    );
     +  } catch (error) {
     +    if (isRetryableFsError(error)) recordGiveUp(dir, error.code);
     +    else recordPermanent(dir, error.code);
     +    return; // keep previous behavior of skipping the dir
     +  }
     @@
     -    const st = await fs.stat(fullPath);
     +    const st = await withFsRetry(
     +      () => fs.stat(fullPath),
     +      {
     +        maxAttempts: config.copytree?.fs?.retryAttempts ?? 3,
     +        initialDelay: config.copytree?.fs?.retryDelay ?? 100,
     +        maxDelay: config.copytree?.fs?.maxDelay ?? 2000,
     +        onRetry: ({ code }) => recordRetry(fullPath, code),
     +      }
     +    ).catch(err => {
     +      if (isRetryableFsError(err)) recordGiveUp(fullPath, err.code);
     +      else recordPermanent(fullPath, err.code);
     +      return null;
     +    });
     +    if (!st) continue;
     +    recordSuccessAfterRetry(fullPath);

   • Verification: Run the walker against a small fixture; ensure errors are logged/aggregated and traversal proceeds.

5. Wrap FS calls and tighten binary fallback in fileLoader.js

   • Edits: [src/utils/fileLoader.js](https://github.com/gregpriday/copytree/issues/src/utils/fileLoader.js)

   • Actions: Wrap stat/readFile; only attempt binary fallback after retries either succeed or we classify the error as non-retryable; log/aggregate accordingly.

   • Snippet:

     --- a/src/utils/fileLoader.js
     +++ b/src/utils/fileLoader.js
     @@
     -  try {
     -    const stats = await fs.stat(fullPath);
     -    const content = await fs.readFile(fullPath, 'utf8');
     -    // ...
     -  } catch (error) {
     -    logger.error('Failed to load file', { path: relativePath, error: error.message });
     -    return null;
     -  }
     +  try {
     +    const stats = await withFsRetry(() => fs.stat(fullPath), {
     +      maxAttempts: config.copytree?.fs?.retryAttempts ?? 3,
     +      initialDelay: config.copytree?.fs?.retryDelay ?? 100,
     +      maxDelay: config.copytree?.fs?.maxDelay ?? 2000,
     +      onRetry: ({ code }) => recordRetry(fullPath, code)
     +    });
     +    const content = await withFsRetry(() => fs.readFile(fullPath, 'utf8'), {
     +      maxAttempts: config.copytree?.fs?.retryAttempts ?? 3,
     +      initialDelay: config.copytree?.fs?.retryDelay ?? 100,
     +      maxDelay: config.copytree?.fs?.maxDelay ?? 2000,
     +      onRetry: ({ code }) => recordRetry(fullPath, code)
     +    });
     +    recordSuccessAfterRetry(fullPath);
     +    // ...
     +  } catch (error) {
     +    if (isRetryableFsError(error)) {
     +      recordGiveUp(fullPath, error.code);
     +      logger.warn('File failed after retries', { path: relativePath, code: error.code });
     +    } else {
     +      recordPermanent(fullPath, error.code);
     +    }
     +    // Binary fallback should not mask transient errors; only try for non-retryable cases:
     +    if (!isRetryableFsError(error)) {
     +      try {
     +        const buf = await fs.readFile(fullPath);
     +        // existing binary handling...
     +        return { /* ... */ };
     +      } catch {
     +        // fall through to null
     +      }
     +    }
     +    return null;
     +  }

   • Verification: Exercise loader with controlled mocks that throw EBUSY then succeed; ensure content is loaded and recordSuccessAfterRetry increments.

6. Wire configuration and CLI flag

   • Edits: src/config/index.js, src/cli/index.js (or equivalent locations where config is read and CLI parsed)

   • Actions:

     • Add defaults: copytree.fs.retryAttempts=3, copytree.fs.retryDelay=100, copytree.fs.maxDelay=2000.
     • Parse --fail-on-fs-errors and propagate to the run context.
     • On process completion, print summary and set exit code if failures present under strict mode.

   • Snippet:

     --- a/src/config/index.js
     +++ b/src/config/index.js
     @@
     export const defaultConfig = {
       copytree: {
         // ...
     +    fs: {
     +      retryAttempts: 3,
     +      retryDelay: 100,
     +      maxDelay: 2000
     +    }
       }
     };

     --- a/src/cli/index.js
     +++ b/src/cli/index.js
     @@
       program
         .option('--fail-on-errors', 'Exit with non-zero code on errors')
     +   .option('--fail-on-fs-errors', 'Exit with non-zero code if filesystem operations fail after retries');
     @@
       const opts = program.opts();
     + const strictFs = Boolean(opts.failOnFsErrors);
     @@
       await runCopyTree(config);
     + import { summarize } from '../utils/fsErrorReport.js';
     + const fsSummary = summarize();
     + logger.info('Filesystem error summary', fsSummary);
     + if (strictFs && (fsSummary.failed > 0 || fsSummary.permanent > 0)) {
     +   process.exitCode = 1;
     + }

   • Verification: node ./bin/copytree --fail-on-fs-errors against fixtures; confirm exit code 1 only when FS failures exist.

7. Docs update

   • Edits: docs/troubleshooting.md, README.md
   • Actions: Document retry behavior, config keys, error summary, and --fail-on-fs-errors flag; call out Windows AV and network-drive guidance.
   • Verification: Render markdown locally.

Tests

• Unit

  • tests/utils/retryableFs.test.js

    • Retries on EBUSY/EPERM/EAGAIN and succeeds on final attempt; assert total attempts and backoff schedule ≈ [100, 200] by injecting a fake timer.

    • Stops immediately on ENOENT/EISDIR; assert no sleep calls.

    • Caps delays at maxDelay.

    • Example:

      import { withFsRetry } from '../../src/utils/retryableFs.js';
      import { vi } from 'vitest';
      
      test('retries EBUSY then succeeds', async () => {
        const op = vi.fn()
          .mockRejectedValueOnce(Object.assign(new Error(), { code: 'EBUSY' }))
          .mockRejectedValueOnce(Object.assign(new Error(), { code: 'EBUSY' }))
          .mockResolvedValue('ok');
        const onRetry = vi.fn();
        const t0 = Date.now();
        const res = await withFsRetry(op, { maxAttempts: 3, initialDelay: 100, onRetry, jitter: false });
        expect(res).toBe('ok');
        expect(onRetry).toHaveBeenCalledTimes(2);
      });

  • tests/utils/fsErrorReport.test.js

    • Recording retries, success-after-retry, give-ups, permanents → summarize() matches expectations.

• Integration/E2E

  • tests/integration/fs-retry.e2e.test.js

    • Mock fs/promises methods to throw EMFILE twice then succeed; run the real ignoreWalker/fileLoader flow and assert items processed.
    • Verify binary fallback only engages for non-retryable code paths.
    • Verify CLI --fail-on-fs-errors sets non-zero exit when failures remain.

  • Commands

    • pnpm test (or npm test)
    • pnpm test -w -i utils to run unit subset, pnpm test -w -i integration for e2e if split.

Edge Cases & Risks

• EMFILE storms: Backoff may be insufficient under heavy concurrency. If flakes persist, throttle concurrent FS ops or add a process-wide semaphore; keep out of scope for this patch but leave utility withFsRetry extensible.
• Error objects without code: Guard for missing err.code and treat as non-retryable unless explicitly mapped.
• ENOENT semantics: Default to non-retryable per acceptance; if future races demand it (e.g., writer process), add an optional copytree.fs.retryOnENOENT=false toggle (default false).
• Long waits: Honor maxDelay cap; consider optional copytree.fs.totalTimeoutMs later if needed.
• Jitter: Disabled by default to meet deterministic schedule; make available for power users on CI fleets.

Rollout & Monitoring

• Flags/configs:

  • --fail-on-fs-errors (CLI).
  • copytree.fs.retryAttempts (default 3), copytree.fs.retryDelay (default 100), copytree.fs.maxDelay (default 2000) in src/config/index.js.

• Telemetry/logs:

  • Ensure logger captures retry attempts with codes and paths (via recordRetry/recordGiveUp).
  • Print one-line summary at end; in verbose mode, optionally dump top offending paths.

• Compatibility:

  • Backwards compatible defaults mirror current behavior except retries (no breaking changes). Setting --fail-on-fs-errors is opt-in strictness.
  • Rollback: revert imports of withFsRetry and the new aggregator; no data migrations.

Acceptance Checklist

• EBUSY, EPERM, EMFILE, ENFILE, EAGAIN, EIO included in retryable codes.
• Exponential backoff with defaults 100ms, 200ms, 400ms, capped by maxDelay=2000ms, configurable via copytree.fs.*.
• ignoreWalker and fileLoader wrap FS calls in withFsRetry().
• Non-retryable errors (incl. ENOENT, EISDIR) fail fast (no retries).
• Aggregated FS error summary printed; counts: total retries, succeeded after retry, permanent/failed.
• --fail-on-fs-errors causes exit code 1 when FS failures remain.
• Unit + integration tests cover Windows-style locks (EBUSY/EPERM) and FD exhaustion (EMFILE/ENFILE).
• Docs updated for configuration, flag, and troubleshooting on Windows and network FS.