Add stress tests, production docs, and prettier setup

[mrekh]

Summary

• Add comprehensive stress tests validating performance with large files (1MB+, 100K lines), pathological wildcard patterns, and bulk URL checking (10K URLs)
• Document production usage guidance including file size limits (500 KiB per RFC 9309) and timeout recommendations
• Document Google-specific behaviors vs RFC 9309 standard (line length limits, typo tolerance, index.html normalization)
• Add prettier with tabs configuration and format script
• Improve JSDoc documentation for URL handling methods with graceful error handling notes
• Remove unused index.ts hello world file

Test plan

• Run bun test to verify all tests pass including new stress tests
• Verify stress tests complete within time limits
• Run bun run format to verify prettier works

[greptile-apps]

Greptile Overview

Greptile Summary

This PR enhances the robots.txt parser library with production-ready features and comprehensive testing. The changes add stress tests validating performance under extreme conditions (1MB+ files, 100K lines, pathological wildcard patterns), document production safeguards (file size limits per RFC 9309, timeout recommendations), and clarify Google-specific behaviors vs RFC 9309 standard.

Key improvements:

• 10 new stress tests covering large file handling, pathological pattern matching, and bulk URL checking (10K URLs)
• Production usage documentation with code examples for file size validation (500 KiB limit per RFC 9309)
• Google-specific behavior comparison table (line length limits, typo tolerance, index.html normalization)
• Enhanced JSDoc comments explaining graceful error handling for malformed URLs
• Prettier configuration with tabs formatting
• Removed unused index.ts hello world file

All tests pass successfully and the code maintains the existing architecture while improving documentation and testing coverage.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• All changes are additive (tests, documentation, tooling) with no modifications to core parsing logic. Stress tests validate performance characteristics, JSDoc improvements clarify existing behavior, and the production documentation helps users implement proper safeguards. All tests pass successfully.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
tests/stress.test.ts	5/5	Added comprehensive stress tests for large files, pathological patterns, and bulk operations
src/matcher.ts	5/5	Enhanced JSDoc documentation explaining graceful error handling for malformed URLs
src/parsed-robots.ts	5/5	Enhanced JSDoc documentation with graceful error handling notes
README.md	5/5	Added production usage guidance and Google-specific behavior documentation
TESTS.md	5/5	Updated test metrics and documented new stress test suite

Sequence Diagram

sequenceDiagram
    participant Test as Stress Test
    participant PR as ParsedRobots
    participant Parser as parseRobotsTxt
    participant Handler as RulesCollectorHandler
    participant Matcher as Pattern Matcher

    Note over Test: Large File Test (1MB)
    Test->>Test: Generate 1MB robots.txt content
    Test->>PR: ParsedRobots.parse(content)
    PR->>Handler: Create RulesCollectorHandler
    PR->>Parser: parseRobotsTxt(content, handler)
    Parser->>Handler: handleRobotsStart()
    Parser->>Handler: handleUserAgent(*, "line")
    Parser->>Handler: handleDisallow(pattern)
    Handler->>Handler: Store rules in groups
    Parser->>Handler: handleRobotsEnd()
    Handler-->>PR: Return collected rules
    PR-->>Test: Return ParsedRobots instance
    Test->>Test: Verify performance < 5s

    Note over Test: Bulk URL Check Test (10K URLs)
    Test->>PR: ParsedRobots.parse(robotsTxt)
    PR-->>Test: ParsedRobots instance
    Test->>Test: Generate 10,000 URLs
    Test->>PR: checkUrls("Googlebot", urls)
    loop For each URL
        PR->>PR: getPathParamsQuery(url)
        PR->>Matcher: matches(path, pattern)
        Matcher-->>PR: Match result
        PR->>PR: Track best allow/disallow
    end
    PR-->>Test: Array of 10K results
    Test->>Test: Verify performance < 1s

    Note over Test: Pathological Pattern Test
    Test->>Test: Create pattern with many wildcards
    Test->>PR: oneAgentAllowedByRobots(robotsTxt, agent, url)
    PR->>Parser: Parse robots.txt
    Parser->>Handler: Collect rules
    PR->>Matcher: Match URL against pattern
    Matcher->>Matcher: Efficient wildcard matching
    Matcher-->>PR: Match result
    PR-->>Test: Allowed/Disallowed
    Test->>Test: Verify performance < 100ms

Loading

[greptile-apps]

_{8 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

style: String concatenation in a tight loop - inefficient for building large strings

Suggested change

	const rule = "Disallow: /path/to/some/resource/\n";
	while (content.length < 1_000_000) {
	content += rule;
	}
	// Generate ~1MB of valid robots.txt content
	const rule = "Disallow: /path/to/some/resource/\n";
	const rulesNeeded = Math.ceil(1_000_000 / rule.length);
	const content = "User-agent: *\n" + rule.repeat(rulesNeeded);

Prompt To Fix With AI

This is a comment left during a code review.
Path: tests/stress.test.ts
Line: 23:26

Comment:
**style:** String concatenation in a tight loop - inefficient for building large strings

```suggestion
		// Generate ~1MB of valid robots.txt content
		const rule = "Disallow: /path/to/some/resource/\n";
		const rulesNeeded = Math.ceil(1_000_000 / rule.length);
		const content = "User-agent: *\n" + rule.repeat(rulesNeeded);
```

How can I resolve this? If you propose a fix, please make it concise.