Expose public programmatic Node.js API

[gregpriday]

Summary

Expose a clean, typed programmatic API for using CopyTree as a library in Node.js applications, enabling developers to integrate file discovery, transformation, and formatting into their build tools and scripts.

Problem Statement

CopyTree is currently CLI-only with no programmatic API:

No entry point:

// ❌ This doesn't work
import { scan, format } from 'copytree';
import copytree from 'copytree';  // No default export

Current workarounds:

// Ugly: Spawn CLI process
import { exec } from 'child_process';
exec('copytree --format json -o output.json', ...);

// Brittle: Directly import internal modules
import copyCommand from 'copytree/src/commands/copy.js'; // Undocumented

Blocked use cases:

• Custom build tools (Vite plugins, Webpack loaders)
• GitHub Actions (programmatic file collection)
• Documentation generators (introspecting codebases)
• Testing frameworks (fixture generation)
• CI/CD pipelines (programmatic output processing)

User Story

As a tool developer
I want a stable programmatic API for CopyTree
So that I can integrate file discovery and transformation into my Node.js applications

Current State

Package structure:

• Entry point: None (package.json only defines bin)
• Exports: Only copyCommand() from src/commands/copy.js
• Internal modules: Not documented for public use
• TypeScript: No .d.ts type definitions

Key internal modules (undocumented):

• src/commands/copy.js - Main copy logic
• src/pipeline/Pipeline.js - Pipeline orchestration
• src/profiles/ProfileLoader.js - Profile loading
• src/transforms/TransformerRegistry.js - Transformer system
• src/utils/ignoreWalker.js - File discovery

Expected Behavior

Public API Surface

// Main functions
import { scan, format, copy } from 'copytree';

// Core classes (for advanced usage)
import { Pipeline, ProfileLoader, TransformerRegistry } from 'copytree';

// Utilities
import { parseProfile, validateConfig } from 'copytree/utils';

// Types (if TypeScript support added)
import type { ScanOptions, FormatOptions, FileResult } from 'copytree';

Primary API: scan()

Discover and filter files without formatting:

import { scan } from 'copytree';

// Simple scan
const files = await scan('/path/to/project');
// Returns: AsyncIterable<FileResult>

// With options
const files = await scan('/path/to/project', {
  profile: 'default',          // Profile name or object
  filter: ['**/*.js'],         // Additional patterns
  exclude: ['node_modules'],   // Exclusions
  respectGitignore: true,      // Use .gitignore
  modified: true,              // Git modified files only
  maxDepth: 3,                 // Directory depth limit
  transform: true,             // Apply transformers
  transformers: ['pdf', 'ai-summary']  // Specific transformers
});

// Iterate results
for await (const file of files) {
  console.log(file.path, file.size, file.content);
}

// Collect all (be careful with memory)
const allFiles = [];
for await (const file of files) {
  allFiles.push(file);
}

Secondary API: format()

Format a list of files into output:

import { format } from 'copytree';

const files = [...]; // From scan() or custom source

// Format to string
const xml = await format(files, { format: 'xml' });
const json = await format(files, { format: 'json' });
const markdown = await format(files, { format: 'markdown' });
const ndjson = await format(files, { format: 'ndjson' });

// Format with options
const output = await format(files, {
  format: 'xml',
  onlyTree: false,
  addLineNumbers: true,
  basePath: '/project',
  instructions: 'Please review this code'
});

Convenience API: copy()

Complete end-to-end operation (equivalent to CLI):

import { copy } from 'copytree';

// Simple copy (returns formatted string)
const result = await copy('/path/to/project');

// With options (matches CLI flags)
const result = await copy('/path/to/project', {
  profile: 'default',
  format: 'json',
  output: './output.json',  // Write to file
  display: true,            // Also log to console
  clipboard: false,         // Don't copy to clipboard
  modified: true,           // Git modified only
  charLimit: 50000,         // Character budget
  // ... all CLI options supported
});

// Result structure
{
  output: '...',           // Formatted string
  files: [...],            // File list
  stats: {
    totalFiles: 123,
    duration: 523,
    totalSize: 456789,
    secretsGuard: {...}    // If enabled
  }
}

Advanced API: Classes

For complex use cases:

import { Pipeline, ProfileLoader, TransformerRegistry } from 'copytree';

// Custom pipeline
const pipeline = new Pipeline({ continueOnError: true });
pipeline.through([
  new FileDiscoveryStage({ basePath: '/project' }),
  new ProfileFilterStage({ profile }),
  new TransformStage({ transformers: registry }),
  new OutputFormattingStage({ format: 'json' })
]);

const result = await pipeline.process({ basePath: '/project' });

// Load custom profile
const loader = new ProfileLoader();
const profile = await loader.load('my-profile');

// Create transformer registry
const registry = await TransformerRegistry.createDefault();
const transformers = registry.getAll();

Affected Components

• New file: src/index.js - Main API exports
• New file: types/index.d.ts - TypeScript definitions
• package.json - Add main, exports, types fields
• src/commands/copy.js - Refactor for programmatic use
• New file: docs/api/programmatic-usage.md - API documentation

Implementation Approach

1. Create src/index.js

// Main API
export { scan } from './api/scan.js';
export { format } from './api/format.js';
export { copy } from './api/copy.js';

// Core classes (advanced)
export { default as Pipeline } from './pipeline/Pipeline.js';
export { default as ProfileLoader } from './profiles/ProfileLoader.js';
export { default as TransformerRegistry } from './transforms/TransformerRegistry.js';

// Utilities
export { parseProfile, validateProfile } from './profiles/ProfileLoader.js';
export { config, validateConfig } from './config/ConfigManager.js';

// Errors
export * from './utils/errors.js';

// Default export (convenience)
export { copy as default } from './api/copy.js';

2. Create src/api/scan.js

export async function* scan(basePath, options = {}) {
  const pipeline = new Pipeline();
  pipeline.through([
    new FileDiscoveryStage({ basePath, ...options }),
    new ProfileFilterStage({ ...options }),
    ...(options.transform ? [new TransformStage({ ...options })] : [])
  ]);
  
  const result = await pipeline.process({ basePath, options });
  
  for (const file of result.files) {
    yield file;
  }
}

3. Update package.json

{
  "main": "./src/index.js",
  "types": "./types/index.d.ts",
  "exports": {
    ".": {
      "import": "./src/index.js",
      "types": "./types/index.d.ts"
    },
    "./utils": {
      "import": "./src/api/utils.js",
      "types": "./types/utils.d.ts"
    }
  },
  "files": [
    "src/",
    "bin/",
    "types/",
    "profiles/",
    "config/"
  ]
}

4. Create TypeScript Definitions

// types/index.d.ts
export interface FileResult {
  path: string;
  absolutePath: string;
  size: number;
  modified: Date;
  content?: string;
  isBinary: boolean;
  encoding?: string;
}

export interface ScanOptions {
  profile?: string | Profile;
  filter?: string[];
  exclude?: string[];
  respectGitignore?: boolean;
  modified?: boolean;
  changed?: string;
  maxDepth?: number;
  transform?: boolean;
  transformers?: string[];
}

export interface FormatOptions {
  format?: 'xml' | 'json' | 'markdown' | 'tree' | 'ndjson' | 'sarif';
  onlyTree?: boolean;
  addLineNumbers?: boolean;
  basePath?: string;
  instructions?: string;
}

export interface CopyResult {
  output: string;
  files: FileResult[];
  stats: {
    totalFiles: number;
    duration: number;
    totalSize: number;
    secretsGuard?: object;
  };
}

export function scan(basePath: string, options?: ScanOptions): AsyncIterable<FileResult>;
export function format(files: FileResult[], options?: FormatOptions): Promise<string>;
export function copy(basePath: string, options?: ScanOptions & FormatOptions): Promise<CopyResult>;

// Classes
export class Pipeline { /* ... */ }
export class ProfileLoader { /* ... */ }
export class TransformerRegistry { /* ... */ }

Tasks

• Create src/index.js with public API exports
• Create src/api/scan.js implementing async iteration
• Create src/api/format.js implementing formatting
• Create src/api/copy.js (wrapper around copyCommand)
• Refactor src/commands/copy.js to support both CLI and programmatic use
• Update package.json with main, exports, types fields
• Create types/index.d.ts with TypeScript definitions
• Add JSDoc comments to all public APIs
• Write unit tests for scan(), format(), copy()
• Write integration tests for programmatic usage
• Create docs/api/programmatic-usage.md with examples
• Add examples to examples/ directory:
  • examples/scan-files.js - Basic scanning
  • examples/custom-pipeline.js - Advanced pipeline
  • examples/build-tool-integration.js - Vite/Webpack example
• Update README with programmatic usage section
• Create migration guide for users importing internal modules

Acceptance Criteria

• import { scan } from 'copytree' works without errors
• scan() returns AsyncIterable that can be used with for await
• format() accepts file list and returns formatted string
• copy() supports all CLI options programmatically
• TypeScript definitions provide autocomplete in IDEs
• JSDoc provides inline documentation in editors
• All public APIs have JSDoc comments
• Breaking changes to internal APIs don't affect public API
• Tests achieve 80% coverage for API surface
• Documentation includes at least 5 practical examples
• Semver versioning promise documented (e.g., minor for new features, major for breaking)

Additional Context

Semver commitment:

• Major (1.0.0): Breaking changes to public API
• Minor (0.x.0): New features, backward compatible
• Patch (0.0.x): Bug fixes, no API changes

Competitor examples:

• fast-glob: import fg from 'fast-glob'; const files = await fg('**/*.js');
• globby: import { globby } from 'globby'; const paths = await globby('**/*.js');
• eslint: import { ESLint } from 'eslint'; const eslint = new ESLint();

Usage examples:

Vite plugin:

import { scan, format } from 'copytree';

export function copytreePlugin() {
  return {
    name: 'copytree',
    async buildStart() {
      const files = await scan('./src', { profile: 'typescript' });
      const xml = await format(files, { format: 'xml' });
      this.emitFile({ type: 'asset', fileName: 'structure.xml', source: xml });
    }
  };
}

GitHub Action:

import { copy } from 'copytree';
import * as core from '@actions/core';

const result = await copy(process.env.GITHUB_WORKSPACE, {
  format: 'sarif',
  output: 'results.sarif'
});

core.setOutput('file-count', result.stats.totalFiles);

Related issues:

• Depends on Add NDJSON and SARIF output formatters #31 (NDJSON/SARIF formatters) for complete format support

[gregpriday]

Goal

Expose a stable, typed, streaming programmatic API (V1) that:

• Lets consumers scan(), format(), copy() without shelling to the CLI.
• Keeps the CLI as a thin wrapper over the new API.
• Guarantees deterministic file ordering, clear error semantics, and backpressure-safe streaming.
• Ships .d.ts with accurate types, supports ESM (primary) with CJS fallback.
• Preserves existing CLI behavior; no breaking changes to flags.

Approach Overview

1. Establish a clean public surface in /src/index.js and hide internals behind an internal/ boundary.
2. Extract CLI orchestration from /src/commands/copy.js into a pure “application service” in /src/app/CopyService.js used by both CLI and library (copy()).
3. Make scan() a streaming AsyncIterable<FileResult> built on the existing pipeline with clear invariants and cancellation (AbortSignal).
4. Refactor formatters into /src/formatters/ with a common Formatter interface; make format() accept Iterable | AsyncIterable.
5. Define validation and config schema in /src/config/schema.js; unify CLI/library options parsing; add validateConfig().
6. Introduce error hierarchy in /src/errors/, and event/logging interfaces in /src/telemetry/ for observability.
7. Package/types: configure exports map, .d.ts, side-effect flags, and dual ESM/CJS entry points; document semver & migration.
8. Tests and examples: unit/integration/e2e parity between CLI and API; add programmatic examples under /examples/.

Step-by-Step Plan

1. Public API module boundary

   • Create /src/index.js exporting only the public API:

     • Functions: scan(), format(), copy()
     • Advanced classes: Pipeline, ProfileLoader, TransformerRegistry
     • Utilities: parseProfile(), validateConfig()

   • Move anything not intended for public use under /src/internal/ to discourage deep imports.

   • Expected outcome: import { scan } from 'copytree' resolves to /src/index.js; deep imports outside the surface fail under Node’s exports.

2. Extract application service (composition root)

   • Create /src/app/CopyService.js that wires: discovery → profile filter → transforms → formatter → writers.
   • Replace orchestration code in /src/commands/copy.js to call CopyService.run(); keep argument parsing and TTY/clipboard only in the CLI layer.
   • Expected outcome: CLI becomes a thin wrapper; copy() reuses exactly the same code path for full parity.

3. Streaming scan with invariants & cancellation

   • Implement /src/api/scan.js as an async generator that yields FileResult as soon as each file is ready (do not buffer entire sets).

   • Add support for AbortSignal and onEvent callback: scan(basePath, { signal, onEvent, ... }).

   • Invariants:

     • FileResult.path: POSIX-style relative path (forward slashes), unique and stable.
     • FileResult.absolutePath: platform path (not normalized to POSIX).
     • FileResult.content: optional; populated only when includeContent: true or a transformer requests it.
     • FileResult.isBinary: true iff content cannot be safely decoded as UTF-8.
     • Emission order: stable, lexicographic by path unless order: 'git'|'mtime' is requested; document default.

   • Performance controls: concurrency, maxDepth, maxBytes, includeContent, respectGitignore, exclude, filter.

   • Expected outcome: for await (const f of scan(...)) works with bounded memory and predictable ordering.

4. Formatter interface and format() over streams

   • Define /src/formatters/BaseFormatter.js (id, supportsStream, format(iterable, options)).

   • Move each formatter (json, xml, markdown, ndjson, tree, sarif) into /src/formatters/ with consistent options.

   • Implement /src/api/format.js that:

     • Accepts Iterable<FileResult> | AsyncIterable<FileResult> | FileResult[].
     • Streams when possible; falls back to buffered modes for formats that require lookahead.
     • Validates options with validateConfig().

   • Expected outcome: await format(files, { format: 'json' }) returns a string; NDJSON/SARIF stream efficiently when inputs are large.

5. End-to-end copy() orchestration

   • Implement /src/api/copy.js that delegates to /src/app/CopyService.js and returns:

     • output: string (or empty if output file was written),
     • files: FileResult[] (respecting summarizeFiles: true to omit heavy content),
     • stats (duration, counts, sizes, optional secretsGuard summary).

   • Add options parity with CLI flags; ensure display, clipboard, output are side-effectful and default to false in the programmatic API.

   • Expected outcome: copy() mirrors CLI behavior without spawning subprocesses.

6. Validation, types, and config schema

   • Create /src/config/schema.js (runtime schema) and /types/index.d.ts (authoritative types).
   • Provide validateConfig() returning { ok, errors } and throwing ConfigValidationError when strict: true.
   • Generate or hand-maintain .d.ts; add a tsd test to verify typings.
   • Expected outcome: Editors get intellisense; invalid options fail fast with actionable messages.

7. Error hierarchy and observability

   • Create /src/errors/ with CopytreeError base and subclasses: ProfileNotFoundError, TransformerError, FormatError, IOError, ConfigValidationError.
   • Introduce onEvent(event) hook where event ∈ { scan:start, scan:file, scan:end, transform:applied, format:start, format:end, error, stats }.
   • Ship a default Logger that writes debug traces only when process.env.COPYTREE_DEBUG is set.
   • Expected outcome: Consumers can trace and measure without coupling to a logging library.

8. Package and exports

   • Update package.json with conditional exports, dual ESM/CJS, and types; mark sideEffects: false for tree-shaking.
   • Provide CJS shim at /dist/cjs/index.cjs that re-exports ESM via createRequire if needed.
   • Expected outcome: import and require both work on Node 18+; bundlers tree-shake unused formatters.

Small example (≤20 lines) for package.json exports:

{
  "type": "module",
  "main": "./src/index.js",
  "types": "./types/index.d.ts",
  "exports": {
    ".": {
      "types": "./types/index.d.ts",
      "import": "./src/index.js",
      "require": "./dist/cjs/index.cjs"
    }
  },
  "sideEffects": false,
  "files": ["src/", "dist/cjs/", "types/", "bin/", "profiles/", "config/"]
}

9. CLI refactor (thin wrapper)

   • In /src/commands/copy.js, replace internal calls with import { copy } from '../api/copy.js' and map parsed flags to API options.
   • Expected outcome: One source of truth; easier testing; less drift.

10. Documentation and examples

• Add /docs/api/programmatic-usage.md and examples under /examples/ matching the new surface.
• Include migration notes for users currently importing internals.

Tests

• Unit (Jest or vitest)

  • /test/unit/scan.test.js: yields order, respects exclude/filter, AbortSignal, maxDepth, respectGitignore.
  • /test/unit/format.test.js: accepts arrays and streams; json/xml/markdown/ndjson/sarif outputs stable.
  • /test/unit/errors.test.js: error classification and cause preservation.
  • /test/unit/types.test-d.ts: typings via tsd (e.g., ScanOptions.transformers type).

• Integration

  • /test/integration/cli-parity.test.js: same inputs produce byte-identical outputs between CLI and copy().
  • /test/integration/transforms.test.js: TransformerRegistry customization affects scan() and copy() consistently.
  • /test/integration/format-streaming.test.js: large repositories stream NDJSON without OOM.

• E2E

  • /test/e2e/programmatic-basic.test.js: Vite-like plugin flow (scan → format → asset).
  • /test/e2e/github-action.test.js: copy() with SARIF writes file and returns expected stats.

• Commands

  • npm run test standard; npm run test:types for tsd; npm run test:e2e for long-running checks.

• Coverage target

  • 80%+ lines/functions/branches for the API surface packages; exclude CLI parser files.

Edge Cases & Risks

• Very large trees (100k+ files): require streaming; guard with maxFiles, maxBytes, and onProgress.
• Symlinks and cycles: decide default (followSymlinks: false), detect loops, emit IOError with code: 'ELOOP'.
• Path normalization across OS: enforce POSIX in FileResult.path; ensure matching/globbing uses forward slashes regardless of platform.
• Binary vs text detection: use BOM + heuristic; never decode as UTF-8 if isBinary true; avoid accidental memory copies.
• Git-only modes: if modified:true and no git repo, either fallback (documented) or throw CopytreeError('GIT_NOT_FOUND').
• Determinism: sorting and formatter output must be stable regardless of parallelism; define and test determinism invariants.
• Transforms safety: untrusted transformers should be opt-in; document that transformers run in-process with the current user’s privileges.
• API misuse: discourage await scan() pattern; document correct usage; optionally export collect(files) helper.

Rollout & Monitoring

• Flags & toggles

  • Environment toggle: COPYTREE_API_V1=1 enables verbose debug logs and additional runtime assertions; off by default for performance.
  • CLI remains unchanged; the library API ships as a minor release.

• Telemetry & logs

  • Emit structured events via onEvent; default logger writes to stderr when COPYTREE_DEBUG=1.
  • Code pointers: /src/telemetry/events.js, /docs/observability.md.

• Rollback

  • Immediate: revert exports to CLI-only (re-publish patch), or publish dist-tag rollback.
  • Code-level: guard new exports behind a small export shim file so a revert is one-file change in /src/index.js.

• Abort criteria

  • If >3 critical issues (crashes, data loss, wrong file selection) arise within 48 hours of release or if p95 memory for copy() exceeds target by 2× on standard fixture (see SLO below), revert and ship a patch.

Acceptance Checklist

• Public API available from import { scan, format, copy } from 'copytree'.

• scan() is an AsyncIterable<FileResult> with AbortSignal support and stable path ordering.

• format() accepts Iterable | AsyncIterable | Array and streams where possible (NDJSON/SARIF).

• copy() uses the same orchestration as the CLI and supports all flags programmatically; no console/clipboard side effects unless requested.

• TypeScript: .d.ts complete; tsd passes; editors show intellisense for all public symbols.

• Errors: all thrown errors inherit from CopytreeError with name, code, and optional cause.

• Observability: onEvent emits documented events; debug logs behind COPYTREE_DEBUG.

• Performance SLO (standard 10k-file fixture):

  • p95 scan() ≤ 3.5s, p95 copy() (json format) ≤ 6s on Node 18 LTS, memory peak ≤ 300MB.
  • Alert when exceeded via a failing e2e check in CI.

• Documentation: /docs/api/programmatic-usage.md, /docs/migration.md, and three runnable examples under /examples/.

• Metrics/log pointers: /src/telemetry/, /docs/observability.md.

Naming note: use consistent prefixes and casing:

• Modules/dirs: camelCase for files, singular service names (e.g., CopyService, ProfileLoader).
• Events: copytree.scan.start, copytree.scan.file, copytree.format.end.
• Metrics keys: copytree.files.count, copytree.duration.ms, copytree.bytes.total.
• Options: prefer nouns (basePath, profile, respectGitignore), boolean positives (display, not noDisplay).