[pull] main from regenrek:main

[pull]

See Commits and Changes for more details.

---

Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

Summary by Sourcery

Update dependencies and add prompt support.

New Features:

• Introduce prompt support with built-in and custom prompts, allowing users to guide LLM interactions.
• Implement token limiting strategies (sequential and truncated) for better control over token usage with --max-tokens.
• Add support for various token encoders (simple, p50k, o200k, cl100k) to accommodate different AI models.

Tests:

• Add integration tests for prompt functionality and codebase fixture.
• Add unit tests for chunk-based token limit and token limiting strategies.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request introduces several new dependencies and updates existing ones in the pnpm-lock.yaml file. It also includes changes to the markdown.ts file to improve token handling and file processing.

Sequence diagram for the new file processing flow

sequenceDiagram
    participant User
    participant CLI
    participant Config
    participant FileCollector
    participant TokenHandler
    participant MarkdownGen

    User->>CLI: Run command
    CLI->>Config: Load configuration
    Config->>Config: Merge CLI args
    Config->>FileCollector: Initialize with config
    FileCollector->>FileCollector: Apply ignore patterns
    FileCollector->>TokenHandler: Process files
    TokenHandler->>TokenHandler: Count tokens
    TokenHandler->>MarkdownGen: Generate content
    MarkdownGen->>CLI: Return markdown
    CLI->>User: Output result

Loading

Class diagram showing the updated configuration structure

classDiagram
    class CodefetchConfig {
        +string outputFile
        +string outputPath
        +number maxTokens
        +string[] includeFiles
        +string[] excludeFiles
        +string[] includeDirs
        +string[] excludeDirs
        +number verbose
        +string[] extensions
        +boolean defaultIgnore
        +boolean gitignore
        +number projectTree
        +TokenEncoder tokenEncoder
        +TokenLimiter tokenLimiter
        +string[] trackedModels
        +boolean dryRun
        +boolean disableLineNumbers
        +string defaultPromptFile
        +string defaultChat
        +Record~string,string~ templateVars
    }

    class TokenEncoder {
        <<enumeration>>
        simple
        p50k
        o200k
        cl100k
    }

    class TokenLimiter {
        <<enumeration>>
        sequential
        truncated
    }

    CodefetchConfig --> TokenEncoder
    CodefetchConfig --> TokenLimiter

Loading

File-Level Changes

Change	Details	Files
Added new dependencies for enhanced functionality.	Added c12, consola, defu, fast-glob, js-tiktoken, mri, and pathe as dependencies. Updated various devDependencies including @vitest/coverage-v8, eslint, unbuild, and vitest. Introduced new packages and updated existing ones in the packages section. Added new snapshots for the new and updated packages.	pnpm-lock.yaml
Improved token handling and file processing in markdown.ts.	Replaced node:fs and node:path imports with pathe. Added token counting and limiting logic using js-tiktoken. Implemented chunk-based reading to handle large files and token limits. Added support for disabling line numbers. Added support for prompt templates. Added support for token limiting strategies (truncated, sequential).	src/markdown.ts

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.