Improved walkDir performance #1814
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>
* Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <rleidle@tesla.com> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <rleidle@tesla.com> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: inimaz <imaz@ai-verse.com> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <rleidle@gmail.com> Co-authored-by: Rob Leidle <rleidle@tesla.com> Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com> Co-authored-by: inimaz <imaz@ai-verse.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>
* fix: scroll issues w/ code blocks * chore: name offset
* docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <sestinj@gmail.com> Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
* feat: multistep quick pick * fix: multi step for context provider * Update test.js
* update styling * remove testing code * remove comments
* feat: create tutorial card * Update TutorialCard.tsx
* feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming
* config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages
In set-up-codestral.md "Trobleshooting" -> "Troubleshooting"
* Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com> Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com> Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com> Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com> Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>
* Add an enableDebugLogs flag * Send logs to VSCode
* Use the resultType given to markComplete This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog. * Add a Compute and UpdateLastUpdated to refreshIndex Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead. * Don't insert compute tags They are REPLACEd in markComplete now.
* feat: allow JetBrains users to index docs * update docs service * update global context * throw error popup on jetbrains + transformers * update toast notifications * update typings * fix tsc errors
* Reload the submenu items when the context provider data is refreshed * update version * Just call refreshSubmenuItems * remove log statements * cleanup * Remove the Promise declaration * Copy currentFiles to a new dict since we're going to mutate it Don't mutate the caller's currentFiles unexpectedly * Remove reIndexFile and just call index/forceReIndex * We MUST await the refreshCodebaseIndex call * Expose send on core * remove unused import * Reindex the workspaces on save * Call core.send Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension * Always refreshSubmenuItems in refreshCodebaseIndex * whitespace * Change messenger back to private
* Add a skipInitialSync function * Add pauseInitialCodebaseIndex * docs * version bump * Rename to pauseCodebaseIndexOnStart
* allow FileContextProvider to handle files * remove commented-out lines
…IndexOnStart is set (#1804)
* set to flex flex-column * Remove fixed height and always show the StyledDivs content to hold its size
* Add a delete button * add a delete message action to the GUI * Fix off by one error * format * This index should be less than length * Update to trash icon
…r repos (#1806) The performance of WalkDir is improved by replacing the recursive DFS with a stack based implementation (less memory in-use during peak exploration). In addition, some unnecessary flags and options were removed which allowed the code to be overall simplified. Much less rule matching is occuring with the same files returned. Some performance results on my linux host: small repo: - 649 files returned by walkDir() - 9 .gitignore files - walkDir() old vs new duration: 918 ms to 143 ms medium sized repo: - 9271 returned by walkDir() - 35 .gitignore files - walkDir() old vs new duration: 10,886 ms to 970ms large sized repo: - 197,900 files returned by walkDir() - 929 .gitignore files - walkDir() old vs new duration: 498,151 ms to 37,353 ms Co-authored-by: Rob Leidle <rleidle@tesla.com>
Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com>
* add support for llama3 405b * 3.1 * name mapping * updates to model setup
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.