Add exit code differentiation for external vs internal crawl errors

[YusukeHirao]

Summary

This PR introduces differentiated exit codes for the crawl and pipeline commands to distinguish between fatal errors (exit code 1) and non-fatal external link errors (exit code 2). It also adds a --strict flag to treat external link errors as fatal when needed.

closes: #36

Key Changes

• New exit code system: Created ExitCode enum with three states:

  • 0 (Success): No errors
  • 1 (Fatal): Internal errors or missing arguments
  • 2 (Warning): External link errors only (DNS failures, certificate errors, etc.)

• Enhanced CrawlAggregateError:

  • Added hasOnlyExternalErrors property to distinguish error types
  • Updated error message to include breakdown of internal vs external errors
  • Added isCrawlerExternalError() type guard to classify errors

• New --strict flag:

  • Added to both crawl and pipeline commands
  • When enabled, treats external link errors as fatal (exit code 1)
  • Allows CI/CD pipelines to tolerate temporary external link failures by default

• Error handling in pipeline:

  • Catches CrawlAggregateError with only external errors
  • Exits with code 2 (Warning) unless --strict is specified
  • Propagates other errors normally

• Updated CrawlerError type: Added isExternal field to distinguish between external and internal errors at the crawler level

• Global error handler: Added try-catch in main CLI entry point to format and handle uncaught errors consistently

Implementation Details

• External errors are identified by checking CrawlerError.isExternal field; plain Error objects are treated as internal
• Error messages now include a breakdown: "Crawl completed with 3 error(s) (2 internal, 1 external)."
• The --strict flag is passed through the entire pipeline from CLI flags to the crawl command
• Comprehensive test coverage added for all exit code scenarios and error classification logic

https://claude.ai/code/session_01HtVd24jAUKYMjuMu1wgJs4