diff --git a/.github/prompts/sparc.md b/.github/prompts/sparc.md new file mode 100644 index 0000000000..cafff565de --- /dev/null +++ b/.github/prompts/sparc.md @@ -0,0 +1,279 @@ +// from: https://gist.githubusercontent.com/ruvnet/7d4e1d5c9233ab0a1d2a66bf5ec3e58f/raw/923916fa92cd225c3d5eeec12a184a4a2a2462ff/.cursorules.txt + +# SPARC Agentic Development Rules + +Core Philosophy + +1. Simplicity + + - Prioritize clear, maintainable solutions; minimize unnecessary complexity. + +2. Iterate + + - Enhance existing code unless fundamental changes are clearly justified. + +3. Focus + + - Stick strictly to defined tasks; avoid unrelated scope changes. + +4. Quality + + - Deliver clean, well-tested, documented, and secure outcomes through structured workflows. + +5. Collaboration + - Foster effective teamwork between human developers and autonomous agents. + +Methodology & Workflow + +- Structured Workflow + - Follow clear phases from specification through deployment. +- Flexibility + - Adapt processes to diverse project sizes and complexity levels. +- Intelligent Evolution + - Continuously improve codebase using advanced symbolic reasoning and adaptive complexity management. +- Conscious Integration + - Incorporate reflective awareness at each development stage. + +Agentic Integration with Cline and Cursor + +- Cline Configuration (.clinerules) + + - Embed concise, project-specific rules to guide autonomous behaviors, prompt designs, and contextual decisions. + +- Cursor Configuration (.cursorrules) + - Clearly define repository-specific standards for code style, consistency, testing practices, and symbolic reasoning integration points. + +Memory Bank Integration + +- Persistent Context + - Continuously retain relevant context across development stages to ensure coherent long-term planning and decision-making. +- Reference Prior Decisions + - Regularly review past decisions stored in memory to maintain consistency and reduce redundancy. +- Adaptive Learning + - Utilize historical data and previous solutions to adaptively refine new implementations. + +General Guidelines for Programming Languages + +1. Clarity and Readability + + - Favor straightforward, self-explanatory code structures across all languages. + - Include descriptive comments to clarify complex logic. + +2. Language-Specific Best Practices + + - Adhere to established community and project-specific best practices for each language (Python, JavaScript, Java, etc.). + - Regularly review language documentation and style guides. + +3. Consistency Across Codebases + - Maintain uniform coding conventions and naming schemes across all languages used within a project. + +Project Context & Understanding + +1. Documentation First + + - Review essential documentation before implementation: + - Product Requirements Documents (PRDs) + - README.md + - docs/architecture.md + - docs/technical.md + - tasks/tasks.md + - Request clarification immediately if documentation is incomplete or ambiguous. + +2. Architecture Adherence + + - Follow established module boundaries and architectural designs. + - Validate architectural decisions using symbolic reasoning; propose justified alternatives when necessary. + +3. Pattern & Tech Stack Awareness + - Utilize documented technologies and established patterns; introduce new elements only after clear justification. + +Task Execution & Workflow + +Task Definition & Steps + +1. Specification + + - Define clear objectives, detailed requirements, user scenarios, and UI/UX standards. + - Use advanced symbolic reasoning to analyze complex scenarios. + +2. Pseudocode + + - Clearly map out logical implementation pathways before coding. + +3. Architecture + + - Design modular, maintainable system components using appropriate technology stacks. + - Ensure integration points are clearly defined for autonomous decision-making. + +4. Refinement + + - Iteratively optimize code using autonomous feedback loops and stakeholder inputs. + +5. Completion + - Conduct rigorous testing, finalize comprehensive documentation, and deploy structured monitoring strategies. + +AI Collaboration & Prompting + +1. Clear Instructions + + - Provide explicit directives with defined outcomes, constraints, and contextual information. + +2. Context Referencing + + - Regularly reference previous stages and decisions stored in the memory bank. + +3. Suggest vs. Apply + + - Clearly indicate whether AI should propose ("Suggestion:") or directly implement changes ("Applying fix:"). + +4. Critical Evaluation + + - Thoroughly review all agentic outputs for accuracy and logical coherence. + +5. Focused Interaction + + - Assign specific, clearly defined tasks to AI agents to maintain clarity. + +6. Leverage Agent Strengths + + - Utilize AI for refactoring, symbolic reasoning, adaptive optimization, and test generation; human oversight remains on core logic and strategic architecture. + +7. Incremental Progress + + - Break complex tasks into incremental, reviewable sub-steps. + +8. Standard Check-in + - Example: "Confirming understanding: Reviewed [context], goal is [goal], proceeding with [step]." + +Advanced Coding Capabilities + +- Emergent Intelligence + - AI autonomously maintains internal state models, supporting continuous refinement. +- Pattern Recognition + - Autonomous agents perform advanced pattern analysis for effective optimization. +- Adaptive Optimization + - Continuously evolving feedback loops refine the development process. + +Symbolic Reasoning Integration + +- Symbolic Logic Integration + - Combine symbolic logic with complexity analysis for robust decision-making. +- Information Integration + - Utilize symbolic mathematics and established software patterns for coherent implementations. +- Coherent Documentation + - Maintain clear, semantically accurate documentation through symbolic reasoning. + +Code Quality & Style + +1. TypeScript Guidelines + + - Use strict types, and clearly document logic with JSDoc. + +2. Maintainability + + - Write modular, scalable code optimized for clarity and maintenance. + +3. Concise Components + + - Keep files concise (under 300 lines) and proactively refactor. + +4. Avoid Duplication (DRY) + + - Use symbolic reasoning to systematically identify redundancy. + +5. Linting/Formatting + + - Consistently adhere to ESLint/Prettier configurations. + +6. File Naming + + - Use descriptive, permanent, and standardized naming conventions. + +7. No One-Time Scripts + - Avoid committing temporary utility scripts to production repositories. + +Refactoring + +1. Purposeful Changes + + - Refactor with clear objectives: improve readability, reduce redundancy, and meet architecture guidelines. + +2. Holistic Approach + + - Consolidate similar components through symbolic analysis. + +3. Direct Modification + + - Directly modify existing code rather than duplicating or creating temporary versions. + +4. Integration Verification + - Verify and validate all integrations after changes. + +Testing & Validation + +1. Test-Driven Development + + - Define and write tests before implementing features or fixes. + +2. Comprehensive Coverage + + - Provide thorough test coverage for critical paths and edge cases. + +3. Mandatory Passing + + - Immediately address any failing tests to maintain high-quality standards. + +4. Manual Verification + - Complement automated tests with structured manual checks. + +Debugging & Troubleshooting + +1. Root Cause Resolution + + - Employ symbolic reasoning to identify underlying causes of issues. + +2. Targeted Logging + + - Integrate precise logging for efficient debugging. + +3. Research Tools + - Use advanced agentic tools (Perplexity, AIDER.chat, Firecrawl) to resolve complex issues efficiently. + +Security + +1. Server-Side Authority + + - Maintain sensitive logic and data processing strictly server-side. + +2. Input Sanitization + + - Enforce rigorous server-side input validation. + +3. Credential Management + - Securely manage credentials via environment variables; avoid any hardcoding. + +Version Control & Environment + +1. Git Hygiene + + - Commit frequently with clear and descriptive messages. + +2. Branching Strategy + + - Adhere strictly to defined branching guidelines. + +3. Environment Management + + - Ensure code consistency and compatibility across all environments. + +4. Server Management + - Systematically restart servers following updates or configuration changes. + +Documentation Maintenance + +1. Reflective Documentation + + - Keep comprehensive, accurate, and logically structured documentation updated through symbolic reasoning. + +2. Continuous Updates + - Regularly revisit and refine guidelines to reflect evolving practices and accumulated project knowledge. diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 1615660e2f..dfa5797436 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -2,7 +2,7 @@ name: build on: push: - branches: [main] + branches: [main, dev] pull_request: workflow_dispatch: jobs: diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 19bbff52b6..8cbb8bd979 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -4,6 +4,7 @@ on: push: branches: - main + - dev # tags: # - "[0-9]+.[0-9]+.[0-9]+" permissions: diff --git a/.github/workflows/genai-commander.yml b/.github/workflows/genai-commander.yml index 418253e5f9..32aedf7e7d 100644 --- a/.github/workflows/genai-commander.yml +++ b/.github/workflows/genai-commander.yml @@ -32,13 +32,13 @@ jobs: id: check_author uses: actions/github-script@v7 with: - script: | - const { owner, repo } = context.repo; - const commentAuthor = context.payload.comment.user.login; - const { data: collaborators } = await github.rest.repos.listCollaborators({ owner, repo }); - const isCollaborator = collaborators.some(collaborator => collaborator.login === commentAuthor); - if (!isCollaborator) - throw new Error(`user ${commentAuthor} is not a collaborator on this repository`); + script: | + const { owner, repo } = context.repo; + const commentAuthor = context.payload.comment.user.login; + const { data: collaborators } = await github.rest.repos.listCollaborators({ owner, repo }); + const isCollaborator = collaborators.some(collaborator => collaborator.login === commentAuthor); + if (!isCollaborator) + throw new Error(`user ${commentAuthor} is not a collaborator on this repository`); # # Resolve the PR sha and ref # @@ -54,20 +54,20 @@ jobs: console.log(res) return JSON.stringify(res) # - # Checkout both main and PR branches + # Checkout both dev and PR branches # - name: checkout uses: actions/checkout@v4 with: fetch-depth: 0 - name: diff PR branch - run: git diff origin/main...origin/${{ fromJSON(steps.sha.outputs.result).ref }} + run: git diff origin/dev...origin/${{ fromJSON(steps.sha.outputs.result).ref }} - name: diff PR commit - run: git diff origin/main...${{ fromJSON(steps.sha.outputs.result).sha }} + run: git diff origin/dev...${{ fromJSON(steps.sha.outputs.result).sha }} - name: checkout PR commit run: git checkout ${{ fromJSON(steps.sha.outputs.result).sha }} - - name: diff main - run: git diff origin/main + - name: diff dev + run: git diff origin/dev # # Setup and build project # @@ -88,14 +88,14 @@ jobs: # - name: genaiscript pr-describe if: startsWith(github.event.comment.body, '/genai describe') - run: node packages/cli/built/genaiscript.cjs run pr-describe -prd --out-output $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-describe -prd --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev env: GITHUB_ISSUE: ${{ github.event.issue.number }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_COMMIT_SHA: ${{ fromJSON(steps.sha.outputs.result).sha }} - name: genaiscript pr-review if: startsWith(github.event.comment.body, '/genai review') - run: node packages/cli/built/genaiscript.cjs run pr-review -prc --out-output $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-review -prc --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev env: GITHUB_ISSUE: ${{ github.event.issue.number }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/genai-pr-commit-review.yml b/.github/workflows/genai-pr-commit-review.yml index 91ad511936..5d16429ce4 100644 --- a/.github/workflows/genai-pr-commit-review.yml +++ b/.github/workflows/genai-pr-commit-review.yml @@ -33,9 +33,9 @@ jobs: - name: start ollama run: yarn ollama:start - name: git stuff - run: git fetch origin && git pull origin main:main + run: git fetch origin && git pull origin dev:dev - name: genaiscript pr-review-commit - run: node packages/cli/built/genaiscript.cjs run pr-review-commit --out ./temp/genai/pr-review-commit -prr --out-trace $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-review-commit --out ./temp/genai/pr-review-commit -prr --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev continue-on-error: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/genai-pr-docs-commit-review.yml b/.github/workflows/genai-pr-docs-commit-review.yml index 47d5d6ed47..15c316aae8 100644 --- a/.github/workflows/genai-pr-docs-commit-review.yml +++ b/.github/workflows/genai-pr-docs-commit-review.yml @@ -30,9 +30,9 @@ jobs: - name: start ollama run: yarn ollama:start - name: git stuff - run: git fetch origin && git pull origin main:main + run: git fetch origin && git pull origin dev:dev - name: genaiscript pr-review-commit - run: node packages/cli/built/genaiscript.cjs run pr-docs-review-commit --out ./temp/genai/pr-docs-review-commit -prr --out-trace $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-docs-review-commit --out ./temp/genai/pr-docs-review-commit -prr --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev continue-on-error: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/genai-pr-review.yml b/.github/workflows/genai-pr-review.yml index c3859ed077..9f90c0d9b7 100644 --- a/.github/workflows/genai-pr-review.yml +++ b/.github/workflows/genai-pr-review.yml @@ -41,11 +41,11 @@ jobs: run: git fetch origin && git pull origin main:main - name: genaiscript pr-describe continue-on-error: true - run: node packages/cli/built/genaiscript.cjs run pr-describe --out ./temp/genai/pr-describe -prd -m reasoning --out-trace $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-describe --out ./temp/genai/pr-describe -prd -m reasoning --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: genaiscript pr-review - run: node packages/cli/built/genaiscript.cjs run pr-review --out ./temp/genai/pr-review -prc -m reasoning --out-trace $GITHUB_STEP_SUMMARY + run: node packages/cli/built/genaiscript.cjs run pr-review --out ./temp/genai/pr-review -prc -m reasoning --out-trace $GITHUB_STEP_SUMMARY --vars defaultBranch=dev continue-on-error: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/licenses.yml b/.github/workflows/licenses.yml index 387ae58ec6..83c3330e79 100644 --- a/.github/workflows/licenses.yml +++ b/.github/workflows/licenses.yml @@ -3,7 +3,7 @@ name: refresh 3rd party licenses on: workflow_dispatch: push: - branches: [main] + branches: [main, dev] paths: - "yarn.lock" concurrency: @@ -13,9 +13,9 @@ jobs: build: runs-on: ubuntu-latest permissions: - actions: read - contents: write - pull-requests: write + actions: read + contents: write + pull-requests: write steps: - uses: actions/checkout@v4 with: diff --git a/.github/workflows/playwright.yml b/.github/workflows/playwright.yml index 3154168ac2..95408645ec 100644 --- a/.github/workflows/playwright.yml +++ b/.github/workflows/playwright.yml @@ -10,6 +10,7 @@ on: push: branches: - main + - dev paths: - yarn.lock - ".github/workflows/playwright.yml" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 57ef514fd3..a1b27ee368 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ You can open this repo in GitHub CodeSpace/Docker to get the build environment n - Go to https://github.com/microsoft/genaiscript - Click on **Code** - Select Create new Codespace - +- Select the **dev** branch ### Manual setup - Install [Node.JS LTS](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index 2acd1b471b..4b3029114c 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -1164,7 +1164,7 @@ The following npm packages may be included in this product: - @grpc/proto-loader@0.7.13 - detect-libc@2.0.3 - docker-modem@5.0.6 - - dockerode@4.0.4 + - dockerode@4.0.5 These packages each contain the following license: @@ -2244,7 +2244,7 @@ Apache License The following npm package may be included in this product: - - @aws-sdk/client-bedrock-runtime@3.778.0 + - @aws-sdk/client-bedrock-runtime@3.779.0 This package contains the following license: @@ -2890,8 +2890,8 @@ The following npm packages may be included in this product: - @types/http-cache-semantics@4.0.4 - @types/node-fetch@2.6.12 - @types/node@16.9.1 - - @types/node@18.19.84 - - @types/node@22.13.14 + - @types/node@18.19.86 + - @types/node@22.13.17 - @types/turndown@5.0.5 - @types/uuid@9.0.8 - @types/yauzl@2.10.3 @@ -2924,7 +2924,7 @@ MIT License The following npm package may be included in this product: - - genaiscript-vscode@1.123.0 + - genaiscript-vscode@1.124.3 This package contains the following license: @@ -2994,7 +2994,7 @@ SOFTWARE. The following npm package may be included in this product: - - express@5.0.1 + - express@5.1.0 This package contains the following license: @@ -3245,7 +3245,7 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. The following npm package may be included in this product: - - cookie@0.7.1 + - cookie@0.7.2 This package contains the following license: @@ -3583,38 +3583,6 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ----------- -The following npm package may be included in this product: - - - methods@1.1.2 - -This package contains the following license: - -(The MIT License) - -Copyright (c) 2013-2014 TJ Holowaychuk -Copyright (c) 2015-2016 Douglas Christopher Wilson - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -'Software'), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, -TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE -SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - ------------ - The following npm packages may be included in this product: - body-parser@2.2.0 @@ -3844,12 +3812,11 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - debug@4.3.6 - debug@4.4.0 -These packages each contain the following license: +This package contains the following license: (The MIT License) @@ -4091,12 +4058,11 @@ THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH RE ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - path-scurry@1.11.1 - path-scurry@2.0.0 -These packages each contain the following license: +This package contains the following license: # Blue Oak Model License @@ -4156,12 +4122,11 @@ software or this license, under any kind of legal claim.*** ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - jackspeak@3.4.3 - jackspeak@4.1.0 -These packages each contain the following license: +This package contains the following license: # Blue Oak Model License @@ -6060,12 +6025,11 @@ Apache-2.0 ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - qs@6.13.0 - qs@6.14.0 -These packages each contain the following license: +This package contains the following license: BSD 3-Clause License @@ -8809,7 +8773,7 @@ THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI The following npm package may be included in this product: - - minizlib@3.0.1 + - minizlib@3.0.2 This package contains the following license: @@ -8885,10 +8849,10 @@ The following npm packages may be included in this product: - @tokenizer/token@0.3.0 - agent-base@6.0.2 - eastasianwidth@0.2.0 - - genaiscript-core-internal@1.123.0 - - genaiscript-sample@1.123.0 - - genaiscript-web@1.123.0 - - genaiscript@1.123.0 + - genaiscript-core-internal@1.124.3 + - genaiscript-sample@1.124.3 + - genaiscript-web@1.124.3 + - genaiscript@1.124.3 - https-proxy-agent@5.0.1 - isarray@1.0.0 - javascript-natural-sort@0.7.1 @@ -10699,7 +10663,8 @@ SOFTWARE. The following npm packages may be included in this product: - - @azure/msal-browser@4.9.1 + - @azure/msal-browser@4.8.0 + - @azure/msal-common@15.3.0 - @azure/msal-common@15.4.0 These packages each contain the following license: @@ -10747,7 +10712,7 @@ The following npm packages may be included in this product: - file-type@20.4.1 - get-stream@5.2.0 - get-stream@9.0.1 - - got@14.4.6 + - got@14.4.7 - is-docker@3.0.0 - is-inside-container@1.0.0 - is-plain-obj@4.1.0 @@ -10773,7 +10738,7 @@ The following npm packages may be included in this product: - temp-dir@3.0.0 - tempy@3.1.0 - type-fest@1.4.0 - - type-fest@4.38.0 + - type-fest@4.39.0 - uint8array-extras@1.4.0 - unicorn-magic@0.3.0 - unique-string@3.0.0 @@ -11389,12 +11354,11 @@ OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - glob@10.4.5 - glob@11.0.1 -These packages each contain the following license: +This package contains the following license: The ISC License @@ -11414,12 +11378,11 @@ IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - - lru-cache@10.4.3 - lru-cache@11.1.0 -These packages each contain the following license: +This package contains the following license: The ISC License @@ -11513,13 +11476,11 @@ IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ----------- -The following npm packages may be included in this product: +The following npm package may be included in this product: - minimatch@10.0.1 - - minimatch@9.0.5 - - rimraf@5.0.10 -These packages each contain the following license: +This package contains the following license: The ISC License @@ -12408,35 +12369,6 @@ THE SOFTWARE. ----------- -The following npm package may be included in this product: - - - utils-merge@1.0.1 - -This package contains the following license: - -The MIT License (MIT) - -Copyright (c) 2013-2017 Jared Hanson - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - ------------ - The following npm package may be included in this product: - path-to-regexp@8.2.0 @@ -12561,7 +12493,7 @@ The following npm packages may be included in this product: - end-of-stream@1.4.4 - pump@3.0.2 - - tar-fs@2.0.1 + - tar-fs@2.1.2 - tar-stream@2.2.0 These packages each contain the following license: @@ -12919,36 +12851,6 @@ SOFTWARE. ----------- -The following npm package may be included in this product: - - - ms@2.1.2 - -This package contains the following license: - -The MIT License (MIT) - -Copyright (c) 2016 Zeit, Inc. - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. - ------------ - The following npm package may be included in this product: - stoppable@1.1.0 diff --git a/docs/package.json b/docs/package.json index 625b7e2de5..ebc2d1ad9b 100644 --- a/docs/package.json +++ b/docs/package.json @@ -2,7 +2,7 @@ "name": "docs", "type": "module", "private": true, - "version": "1.124.2", + "version": "1.124.3", "license": "MIT", "scripts": { "install:force": "rm yarn.lock && yarn install", diff --git a/docs/src/content/docs/blog/ast-grep-and-transform.mdx b/docs/src/content/docs/blog/ast-grep-and-transform.mdx new file mode 100644 index 0000000000..837c7e899e --- /dev/null +++ b/docs/src/content/docs/blog/ast-grep-and-transform.mdx @@ -0,0 +1,184 @@ +--- +title: AST Grep and Transform +date: 2025-04-01 +tags: + - ast + - tree-sitter + - ast-grep + - genai +authors: pelikhan +canonical_url: https://microsoft.github.io/genaiscript/blog/ast-grep-and-transform +description: Learn how to parse source code using ast-grep and update nodes in the tree. +cover: + alt: The image is an abstract 8-bit style illustration using five colors. It + shows a stylized tree with nodes connected in a branching pattern, + representing an Abstract Syntax Tree (AST) used in code transformation. + Geometric shapes and lines indicate the ideas of parsing and modifying + source code. The design is simple and corporate, with no characters or text + present. + image: ./ast-grep-and-transform.png +--- + +import { Code } from "@astrojs/starlight/components" +import code from "../../../../../genaisrc/docs.genai.mjs?raw" + +This page describes a strategy to build GenAI scripts that use Abstract Syntax Trees (AST) to parse +and modify source code. When applicable, it provides an extremely flexible and stable way to +apply large scale changes to source code. Interested? Let's dive in! + +:::tip + +If you're in hurry, jump to the **last documentation generator/updater you'll ever need** \* +in the [sample section](#sample). + +_\*: for TypeScript, maybe._ +::: + +## The strategy of AST-based code transformation + +One of the challenge when creating GenAI scripts that update source code is to correctly locate and update source code. +The slightest mistake in the location of the code to update can lead to a broken code. This is especially true when +the code to update is not a simple string, but a complex structure like an object or a function call. + +In some cases, you know "precisely" which part of the code you want +to update. For example, you want to refresh the documentation of a function after a change. You know +that the documentation is located just before the function definition at least in the sense of the programming language +but the number of empty lines or space may vary. + +```ts title="math.ts" +/** sums a and b */ +function fn(a: number, b: number): number { + return a - b // oops outdated +} +``` + +In such scenario, you can use the Abstract Syntax Tree (AST) to locate the code to update. The AST is a tree representation of +code. + +```mermaid +graph TD + A(comment
/** sums a and b */) --> B(function_declaration
fn) + B --> D(statement
return a - b) +``` + +So instead of fighting spaces and new lines, you can just locate the `function_declaration` node that follows +a `comment` node. + +```ts title="docs.genai.mts" wrap +const node = sg.search("functions without comments") +``` + +Once you've located the node to update, you could do any transformation you want, e.g. replace it with another text. +In terms of GenAI script, this means that you can build a prompt that includes as much context as you need, generate a response. + +```ts title="docs.genai.mts" wrap +$`Update the documentation of the function 'fn' to reflect the new behavior of the function.` +fence(node.text()) +``` + +```txt +/* subs a and b */ +``` + +Once the LLM responds with the new comment, you could insert it as the new content of the node in the AST. + +```ts title="docs.genai.mts" +edits.replace(node.comment(), response) +``` + +```mermaid +graph TD + A(comment
/** subs a and b */) --> B(function_declaration
fn) + B --> D(statement
return a - b) +``` + +Voila! You've only touched the part of the file you wanted to update! + +```ts title="math.ts" +/** subs a and b */ +function fn(a: number, b: number): number { + return a - b +} +``` + +To recap, this strategy is based on the following steps: + +1. **search** Use the AST to locate the node to update. +2. **transform and replace** Use the LLM to generate the new content of the node. +3. **commit** Update the node in the AST with the new content. + +## ast-grep + +![AST-Grep Logo](https://ast-grep.github.io/logo.svg) + +[ast-grep(sg)](https://ast-grep.github.io/) is a fast and polyglot tool for code structural search, +lint, rewriting at large scale. **sg** provides us the AST-search-and-replace capabilities we need to +implement the strategy above. + +GenAIScript benefits from the excellent [Node.JS integration](https://ast-grep.github.io/reference/api.html#napi), +which is available through the [`host.astGrep()`](https://microsoft.github.io/genaiscript/reference/scripts/ast-grep) method. + +### Searching + +The `sg.search` method allows you to search for nodes in the AST. It takes the language, file glob +and pattern syntax and returns a list of matches. + +```ts title="docs.genai.mts" wrap +// search +const { matches, replace } = await sg.search("ts", "src/*fib*.ts", { + rule: { + kind: "function_declaration", + not: { + precedes: { + kind: "comment", + stopBy: "neighbor", + }, + }, + }, +}) +``` + +### Editing + +The `sg.changeset` method creates a changeset that can be used to apply a set of edits to a set of files. + +```ts +// transform +const edits = sg.changeset() +for (const match of matches) { + const { text } = await prompt`Generate new docs for ${match.text()}` + // replace + edits.replace(match.comment(), text) // it's somewhat more involved +} +// commit all edits to file +await workspace.writeFiles(edits.commit()) +``` + +## Sample: Doc generator / updater + +You will find a full write down of the making of the documentation generator/updater script below +in the [documentation](/genaiscript/reference/scripts/ast-grep). I encourage you to read it to dig deeper. + +The `docs` scripts is a documentation generator/updater. + +- uses ast-grep to find and generate missing documentation for exported TypeScript function. A second LLM-as-Judge + request is used to check that the generated documentation is correct. +- if the `diff` option is selected, it will filter out functions that do not intersect with the + diff (this is rather naive but a good start...). +- it can also be used to update the documentation of a function that has changed. +- it works regardless of the file size or the number of files as most transformations are hyper-localized. + +```sh title="generate and refresh my docs plz" +genaiscript run docs -- --diff +``` + +Here are some example of applications of the scripts (one-shot, no human edit, multi-edit per file): + +- [TypeScript codebase](https://github.com/pelikhan/TypeScript/pull/1), I stopped the script after a while, it was humming. +- [GenAIScript codebase](https://github.com/microsoft/genaiscript/pull/1376/files), this change also contains the update to GenAIScript as I was building the feature. + +Here it goes: + + + +- [original source](https://github.com/microsoft/genaiscript/blob/main/genaisrc/docs.genai.mts) diff --git a/docs/src/content/docs/blog/ast-grep-and-transform.png b/docs/src/content/docs/blog/ast-grep-and-transform.png new file mode 100644 index 0000000000..55ab07587f Binary files /dev/null and b/docs/src/content/docs/blog/ast-grep-and-transform.png differ diff --git a/docs/src/content/docs/blog/mcp-resources.mdx b/docs/src/content/docs/blog/mcp-resources.mdx index 5809f2d1ce..fe6c42ec58 100644 --- a/docs/src/content/docs/blog/mcp-resources.mdx +++ b/docs/src/content/docs/blog/mcp-resources.mdx @@ -2,7 +2,7 @@ title: MCP Resources description: Learn how to publish MCP resources seamlessly using GenAIScript. keywords: MCP, GenAIScript, resources, publishing -date: 2025-30-03 +date: 2025-03-30 cover: alt: A colorful 2D illustration in an 8-bit retro style, depicting a stylized diff --git a/docs/src/content/docs/reference/cli/commands.md b/docs/src/content/docs/reference/cli/commands.md index 145ae5bce8..00ab6e850e 100644 --- a/docs/src/content/docs/reference/cli/commands.md +++ b/docs/src/content/docs/reference/cli/commands.md @@ -266,7 +266,7 @@ Commands: folder to enable type checking. compile [folders...] Compile all scripts in workspace model [options] [script] List model connection information for scripts - help [command] display help for command + help|info