feat: filter context with .cody/.ignore #1382

abeatrix · 2023-10-12T17:25:05Z

For details, please see RFC 852 File Exclusion Patterns for Cody

feat: filter context from ignored files

IMPORTANT: Currently behind the cody.internal.unstable config and must be enabled to try the feature. When this is out of internal experimental stage, this should be enabled by default.

Add isCodyIgnoreFile utility to check if file is defined in .cody/.ignore (at workspace level)
Update Interaction to filter context messages from ignored files when first created
Update context gathering in completions to skip ignored files
Add check in editor to skip ignored files
ignore all .env files / path by default

All clients should provide file content from the .cody/.ignore file at start up, and update the ignoreList when the file is changed.

Note

This PR includes update to the VS Code clients so that the ignore list is set at Editor start up, and whenever the .cody/.ignore file is updated, the ignore list will be updated accordingly.

This does not make any changes to the current embedding steps or BFG, which will be a feature to be implemented in the future as listed by Dom in the discussion below.

PRFAQ to agree on the .cody/.ignorename

Test plan

Prep:

Build Cody from this branch, start in VS Code debug mode
Turn on the Unstable Experimental Features from the Cody Settings
Open the cody repository
Create a .codyignore file
Add the following to the .codyignore file

vscode/**
utils.ts

Chat

Ask cody about fixup in Chat View: "what is fixup?"

You should see Cody did not read any files from the vscode directory to answer your question:

Autocomplete

Go to any of the file inside the vscode directory, you should not get any autocomplete suggestions.

Command

Go to lib/shared/src/utils.ts file, and then select some code, right click and select Explain Code:

You should see cody explain a random code (hallucinating because we didn't provide any code as context) that is not your selected code.

You can check the output channel to confirm no context was sent to the LLM:

Demo

DEMO_.cody_.ignore.file.mp4

Update

Here is a loom video of codyignore working in Simple chat panel:

https://www.loom.com/share/83fe3e0b0728449385d130d45c317cfa

philipp-spiess

Awesome stuff! Left some early comments but feel free to ignore

lib/shared/src/chat/context-filter.ts

lib/shared/src/chat/transcript/interaction.ts

vscode/src/services/context-filter.ts

lib/shared/src/chat/context-filter.ts

vscode/src/services/context-filter.ts

Co-authored-by: Philipp Spiess <hello@philippspiess.com>

lib/shared/src/chat/context-filter.test.ts

lib/shared/src/chat/context-filter.ts

vscode/src/services/context-filter.ts

abeatrix · 2023-11-03T13:33:58Z

@DanTup Thank you so much for reviewing this PR!

Updated the PR to address your very helpful feedback, especially the point regarding handling relative file paths in isCodyIgnoredFile:

Update isCodyIgnoredFile to handle relative file paths instead of absolute
Extract current ignore file directory and make file paths relative to it
Handle errors in isCodyIgnoredFile instead of returning false
Update setCodyIgnoreList docs to mention it takes ignore file path

I haven't looked into passing the entire ignore file which seems to make things a lot easier, will continue the discussion with you on Slack!

--------- Co-authored-by: Beatrix <68532117+abeatrix@users.noreply.github.com>

slimsag · 2023-11-15T19:14:13Z

Q:

IMPORTANT: Currently behind the cody.internal.unstable config and must be enabled to try the feature. When this is out of internal experimental stage, this should be enabled by default.

What's the motivation behind this vs. making the existence of a .cody/.ignore file opt-in to this functionality? When would it not be behind the feature flag?

slimsag · 2023-11-15T19:16:57Z

lib/shared/src/chat/ignore-helper.ts

+/**
+ * The Cody ignore file path in POSIX style (always forward slashes).
+ */
+export const CODY_IGNORE_FILENAME_POSIX_GLOB = path.posix.join('**', '.cody', '.ignore')


I notice the PR title and description mentions .codyignore, while the code here seems to suggest .cody/.ignore. (not commenting on the awkwardness of two . files..)

which one is right / expected to work?

It'll be .cody/.ignore! Will attach the RFC to the issue to clarify! RFC 852 File Exclusion Patterns for Cody

slimsag

This is quite useful to us on the Cody strat team, thank you for working on this!

abeatrix · 2023-11-15T19:21:46Z

Q:

IMPORTANT: Currently behind the cody.internal.unstable config and must be enabled to try the feature. When this is out of internal experimental stage, this should be enabled by default.

What's the motivation behind this vs. making the existence of a .cody/.ignore file opt-in to this functionality? When would it not be behind the feature flag?

I wanted to emphasize on the fact that this feature is still in experimental and unstable (not throughly tested so users must manually enable this feature to confirm before they can test it out.

I'm thinking once we have tested this internally and have .Cody/.ignore working for embedding jobs, we can have it enabled by default.

Do you think this is a good idea, or am I overthinking 😂?

lib/shared/src/chat/ignore-helper.ts

…into bee/codyignore

DanTup

LGTM with a few minor comments (although I'm less familiar with some of this code so you may want another reviewer too).

lib/shared/src/chat/transcript/interaction.ts

vscode/src/repository/repositoryHelpers.ts

vscode/src/services/context-filter.ts

vscode/src/repository/repositoryHelpers.ts

DanTup · 2023-12-29T19:00:59Z

vscode/src/repository/repositoryHelpers.ts

+        }
+    }
+
+    await extension?.activate().then(() => init())


Do we need to worry about any exceptions here (for example if you disable the Git extension does it throw?)

ahh i added try catch to this for cases where Git extension is disabled. we probably will need to do some follow up works on UX improvement but I think what we have is a good start. I am going to do some final testings before I merge it but consider this is currently behind the cody.internal.unstable config flag, i think we should be good 🙏

Thank you so much for your help and your throughout review 🙇‍♀️

feat: filter context from .codyignore

84d6cda

abeatrix mentioned this pull request Oct 12, 2023

feat: allow configuring file exclusions #1358

Closed

philipp-spiess reviewed Oct 13, 2023

View reviewed changes

abeatrix and others added 5 commits October 13, 2023 06:05

Apply suggestions from code review

d00e1d2

Co-authored-by: Philipp Spiess <hello@philippspiess.com>

use ignore npm package

107fa0b

update test

3c09549

use fileWatcher

f9490b7

move removeCodyIgnoredFiles in Interaction

defa542

DanTup reviewed Nov 3, 2023

View reviewed changes

abeatrix added 2 commits November 3, 2023 05:25

Merge branch 'main' into bee/codyignore

09d75fa

Update isCodyIgnoredFile to handle relative file paths

8bfec0b

DanTup and others added 7 commits November 11, 2023 10:55

Support nested ignore files (#1634)

3ee3520

--------- Co-authored-by: Beatrix <68532117+abeatrix@users.noreply.github.com>

Merge branch 'main' into bee/codyignore

16bcf9b

clean up vscode-context

49be541

Merge branch 'main' of https://github.com/sourcegraph/cody

ffb8a42

Merge branch 'main' into bee/codyignore

7ee2824

filter embedding results from current ws

179db5d

New cody.internal.unstable feature flag

17940c2

abeatrix requested review from chwarwick and a team November 15, 2023 18:24

Merge branch 'main' into bee/codyignore

621ed9a

abeatrix marked this pull request as ready for review November 15, 2023 18:26

fix tests

8139eb7

slimsag reviewed Nov 15, 2023

View reviewed changes

slimsag approved these changes Nov 15, 2023

View reviewed changes

DanTup reviewed Nov 15, 2023

View reviewed changes

lib/shared/src/chat/ignore-helper.ts Outdated Show resolved Hide resolved

set ws in isIgnored instead of setIgnoreFiles

8ae6f6f

abeatrix mentioned this pull request Nov 16, 2023

proof of concept - .cody/.ignore per codebase in workspace #1769

Merged

abeatrix and others added 3 commits November 21, 2023 21:30

map .cody/.ignore file per codebase in workspace (#1769)

a38fab6

Merge branch 'bee/codyignore'

300f2fc

Merge branch 'bee/codyignore' of https://github.com/sourcegraph/cody …

9ba681d

…into bee/codyignore

DanTup mentioned this pull request Dec 11, 2023

Introduce exclusion patterns for files that shouldn't be sent over the network #1049

Closed

DanTup mentioned this pull request Dec 22, 2023

bug: files in bin/ are filtered out of @-mention in chat #2401

Closed

abeatrix added 5 commits December 28, 2023 12:01

Merge branch 'bee/codyignore' of https://github.com/sourcegraph/cody …

52afe1e

…into bee/codyignore

Merge branch 'main' into bee/codyignore

55cbcaf

Merge branch 'main' into bee/codyignore

20a0449

codyignore to work with simple chat

b836727

use string for findFiles

45ab0d4

DanTup approved these changes Dec 29, 2023

View reviewed changes

abeatrix commented Dec 29, 2023

View reviewed changes

vscode/src/services/context-filter.ts Outdated Show resolved Hide resolved

abeatrix added 4 commits December 29, 2023 09:28

move setup logic to git startup

8a3762b

remove 'Unstable Experimental Features' flag

d9fab6f

add error logging

b7b3d7d

document code

28689c6

DanTup reviewed Dec 29, 2023

View reviewed changes

vscode/src/repository/repositoryHelpers.ts Outdated Show resolved Hide resolved

DanTup reviewed Dec 29, 2023

View reviewed changes

vscode/src/repository/repositoryHelpers.ts Outdated Show resolved Hide resolved

abeatrix added 2 commits December 29, 2023 10:40

move gitAPIinit to main

974fb08

update gitAPIinit

e62c34d

DanTup reviewed Dec 29, 2023

View reviewed changes

abeatrix added 3 commits December 29, 2023 11:29

add error handling

e1ccb5b

refactor gitAPIinit

58c252f

add changelog entry

6eb1823

abeatrix merged commit c9a4f7e into main Dec 29, 2023
13 checks passed

abeatrix deleted the bee/codyignore branch December 29, 2023 21:15

abeatrix changed the title ~~feat: filter context from .cody/.ignore~~ feat: filter context with .cody/.ignore Jan 9, 2024

abeatrix self-assigned this Jan 9, 2024

abeatrix mentioned this pull request Jan 16, 2024

.cody/ignore tracking - client side #2756

Closed

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: filter context with .cody/.ignore #1382

feat: filter context with .cody/.ignore #1382

abeatrix commented Oct 12, 2023 •

edited

philipp-spiess left a comment

abeatrix commented Nov 3, 2023

slimsag commented Nov 15, 2023

slimsag Nov 15, 2023

abeatrix Nov 15, 2023

slimsag left a comment

abeatrix commented Nov 15, 2023

DanTup left a comment

DanTup Dec 29, 2023

abeatrix Dec 29, 2023

feat: filter context with .cody/.ignore #1382

feat: filter context with .cody/.ignore #1382

Conversation

abeatrix commented Oct 12, 2023 • edited

Note

Test plan

Chat

Autocomplete

Command

Demo

Update

philipp-spiess left a comment

Choose a reason for hiding this comment

abeatrix commented Nov 3, 2023

slimsag commented Nov 15, 2023

slimsag Nov 15, 2023

Choose a reason for hiding this comment

abeatrix Nov 15, 2023

Choose a reason for hiding this comment

slimsag left a comment

Choose a reason for hiding this comment

abeatrix commented Nov 15, 2023

DanTup left a comment

Choose a reason for hiding this comment

DanTup Dec 29, 2023

Choose a reason for hiding this comment

abeatrix Dec 29, 2023

Choose a reason for hiding this comment

abeatrix commented Oct 12, 2023 •

edited