feat: test (edit) command & add tests code lenses #2959

abeatrix · 2024-01-30T23:37:09Z

PR to introduce /unit command that is a smarter version of the /test command.

/unit command

/unit command generates unit tests in chat (old /test)
/test command generates unit tests in line
- if a test file exists, the new unit test suit will be appended to the end of the current test file
- if a test file does not exist, cody will suggest a file name and we will create a temporary test file with the new test suit appended along with the import statements (if any)

Add Tests code lenses

add-tests is a command that can be executed via the Cody Command CodeLenses (when enabled) in test files. It is not available via the command menu or context menu.

Behind internal unstable flag.

Summary

This PR includes the following change:

Refactor the /unit command, rename it to /test, and move it out of unstable
Rename the old /test command to /unit
Add test-case command
Add executeUnitCaseCommand to execute unit test cases
Added support for specifying a destination file in the executeEditArguments interface in the edit/execute.ts file.
Added support for specifying a destination file in the executeEdit method in the edit/manager.ts file.
Removed the NewFixupFileMap import and added support for specifying a destination file in the EditProvider class in the edit/provider.ts file.
Removed the NewFixupFileMap import in the FixupController class in the non-stop/FixupController.ts file.

Test plan

Added e2e test.

Test

Run the Test command to confirm the newly generated tests are added to a new / exisiting test file automatically for you.

Add Tests

Enable Code Lenses in your Cody Setting

Locate the Add Tests code lenses in a test file to check if new test cases have been inserted in line for you

Follow-up

option to append to current selection instead of insert
context improvement for test

…tArguments

vscode/package.json

vscode/src/services/treeViewItems.ts

vscode/src/non-stop/FixupTask.ts

toolmantim

Woo!

Why the rename to /unit? I don't see a strong argument for renaming — I'd recommend we keep a single /test command (the edit version) and replace the old /test (chat version) with this one
I couldn't get this to work well with Python or Typescript… it seems to replace the code with the tests (see videos below)
We might need to add some kind of progress that isn't just a spinner, because tests can take quite a while to generate. Maybe something like \Generating tests… (${n} lines)``?
Is the "unit-test-case" command — it's an interesting idea, but I don't fully understand what problem it's trying to solve. Could you expand more on how it fits into the overall flow?

CleanShot.2024-01-31.at.16.38.16.mp4

CleanShot.2024-01-31.at.16.31.13.mp4

abeatrix · 2024-01-31T06:00:09Z

Why the rename to /unit? I don't see a strong argument for renaming — I'd recommend we keep a single /test command (the edit version) and replace the old /test (chat version) with this one

Because jetBrains is using the old one so if I was worried switching it would cause issue for them, but I think I can find a workaround for that.

abeatrix · 2024-01-31T06:03:51Z

Is the "unit-test-case" command — it's an interesting idea, but I don't fully understand what problem it's trying to solve. Could you expand more on how it fits into the overall flow?

That's for adding more tests to an existing test suite inside the test file, vs create a new test suite with tests using other test context as reference, adding more tests to an existing test suite should be easier for the llm and the result should have a higher quality from what I've tested so far.

abeatrix · 2024-01-31T06:07:03Z

I couldn't get this to work well with Python or Typescript… it seems to replace the code with the tests (see videos below)

Let me try to reproduce it. Can you try it on non-react typescript code in the meantime?

toolmantim · 2024-01-31T06:25:40Z

That's for adding more tests to an existing test suite inside the test file, vs create a new test suite with tests using other test context as reference, adding more tests to an existing test suite should be easier for the llm and the result should have a higher quality from what I've tested so far.

That's interesting about higher quality — is that higher quality output for the same user problem? (i.e. same source and test file, but running the command from the test file first rather than the source file first).

I still don't quite understand what "Add Tests" quite means, or how it fits into their overall flow though. When would someone know to hit and not hit the "Add Tests" button?

abeatrix · 2024-01-31T06:36:08Z

i.e. same source and test file, but running the command from the test file first rather than the source file first

Yes. You will only see this code lense in test files. You will not see this in source file.

I still don't quite understand what "Add Tests" quite means, or how it fits into their overall flow though. When would someone know to hit and not hit the "Add Tests" button?

it only shows up on top of each test suite, and clicking "Add Tests" adds new tests to the associated test suite. You don't see them in non-test files.

abeatrix · 2024-01-31T06:38:29Z

For example, i started the suite with only 1 test:

Press Add tests:

Ended up with more tests:

abeatrix · 2024-01-31T06:58:19Z

@toolmantim Can you share the link to the repo you tested with to see if I can reproduce the issue? I can't reproduce the issue when testing it on the langchain python repository.

python.mov

abeatrix · 2024-01-31T07:43:42Z

rename the new unit test command to /test
replace the old /test (chat version) with /unit

umpox · 2024-01-31T14:50:52Z

Noticed one thing, it seems to often not stream the text into the new file for me. It's a bit off putting because it feels like nothing is working, but actually the file seems to be just created at the end.

I got a good video of it doing this:

TestStreaming.mov

It looks like the file name isn't properly returned by the LLM. WDYT about maybe firing two LLM requests for this command:

The edit one
A separate one to get the file name, if it doesn't exist. Then we maybe have a better chance of getting a valid response?

We had something similar for the intent detector: https://sourcegraph.com/github.com/sourcegraph/cody@bee/new-commands/-/blob/lib/shared/src/intent-detector/client.ts?L66%3A1-125%3A6

We could do the same here?

umpox

Left some minor comments. Looking really good!

vscode/src/commands/execute/test-case.ts

vscode/src/commands/execute/test-file.ts

vscode/src/commands/services/CommandCodeLenses.ts

vscode/src/commands/execute/test-case.ts

umpox · 2024-01-31T14:33:30Z

vscode/src/commands/services/CommandCodeLenses.ts

+        const symbols =
+            allSymbols?.filter(
+                symbol =>
+                    symbol.kind === vscode.SymbolKind.Function ||
+                    symbol.kind === vscode.SymbolKind.Class ||
+                    symbol.kind === vscode.SymbolKind.Method ||
+                    symbol.kind === vscode.SymbolKind.Constructor
+            ) ?? []

-        // Add code lenses for each symbol
-        if (symbols) {
+        // For test files, adds code lenses for each symbol
+        if (isTest) {
            for (const symbol of symbols) {


I guess the downside here is that this will show for random functions too, e.g.

I guess we don't really have a way to determine which symbols are actually used for tests here :/

Yea we should update the code lenses logic in a follow up PR!

vscode/src/non-stop/FixupController.ts

vscode/src/non-stop/FixupTask.ts

Co-authored-by: Tom Ross <tom@umpox.com>

abeatrix · 2024-01-31T18:54:26Z

@toolmantim @umpox i've updated the prompt so that it will always generate the file name. Before this change, if cody doesn't reply with a file name suggestion, we would wait for the test to finish before adding the code, causing a delay in showing the response.
The updated prompt now reminds the LLM provider to start the response with a file name suggestion.

abeatrix · 2024-01-31T18:55:26Z

@toolmantim also put the Add Tests code lense behind the unstable flag so the new /test command won't get blocked by it.

toolmantim

Works great with the examples I found now! Let's just go with a single /test command (edit) for now though.

abeatrix added 5 commits January 30, 2024 14:51

feat: unit test files & unit test cases commands

65cc867

feat(edit): Add support for specifying destination file in executeEdi…

e09e947

…tArguments

move out of unstable

8079158

add e2e test

ba43e00

remove .only

d721374

abeatrix commented Jan 30, 2024

View reviewed changes

vscode/package.json Show resolved Hide resolved

abeatrix requested review from toolmantim, umpox and a team January 30, 2024 23:39

abeatrix commented Jan 31, 2024

View reviewed changes

vscode/src/services/treeViewItems.ts Outdated Show resolved Hide resolved

abeatrix commented Jan 31, 2024

View reviewed changes

vscode/src/non-stop/FixupTask.ts Show resolved Hide resolved

abeatrix added 2 commits January 30, 2024 20:37

Merge branch 'main' into bee/new-commands

a3e1e81

update codelenses for /unit

2cdd35e

abeatrix mentioned this pull request Jan 31, 2024

VSCE: Improve Generate Unit Tests command #1475

Closed

toolmantim reviewed Jan 31, 2024

View reviewed changes

abeatrix added 2 commits January 30, 2024 23:34

switch /test and /unit

b707928

update e2e test

ae08fab

abeatrix changed the title ~~feat: unit test command & unit test case command~~ feat: test (edit) command & add tests code lenses Jan 31, 2024

umpox reviewed Jan 31, 2024

View reviewed changes

abeatrix and others added 2 commits January 31, 2024 07:24

Update vscode/src/commands/execute/test-case.ts

439b31e

Co-authored-by: Tom Ross <tom@umpox.com>

Apply suggestions from code review

f6801b6

Co-authored-by: Tom Ross <tom@umpox.com>

abeatrix added 3 commits January 31, 2024 10:27

update prompt, add test intent

6368256

Merge branch 'main' into bee/new-commands

cfc76a8

move add-tests behind unstable flag

29b8358

abeatrix requested review from toolmantim and umpox January 31, 2024 18:56

toolmantim approved these changes Jan 31, 2024

View reviewed changes

abeatrix added 3 commits January 31, 2024 15:38

remove old test command from menu

7fd932a

add changelog entry

bce3768

update expected events for command e2e test

fe0291a

abeatrix merged commit 0513f7a into main Feb 1, 2024
15 checks passed

abeatrix deleted the bee/new-commands branch February 1, 2024 00:43

umpox mentioned this pull request Feb 2, 2024

Edit: Add support for multiple models #2951

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: test (edit) command & add tests code lenses #2959

feat: test (edit) command & add tests code lenses #2959

abeatrix commented Jan 30, 2024 •

edited

toolmantim left a comment

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

toolmantim commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

umpox commented Jan 31, 2024

umpox left a comment

umpox Jan 31, 2024

abeatrix Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

toolmantim left a comment

feat: test (edit) command & add tests code lenses #2959

feat: test (edit) command & add tests code lenses #2959

Conversation

abeatrix commented Jan 30, 2024 • edited

/unit command

Add Tests code lenses

Summary

Test plan

Test

Add Tests

Follow-up

toolmantim left a comment

Choose a reason for hiding this comment

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

toolmantim commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

umpox commented Jan 31, 2024

umpox left a comment

Choose a reason for hiding this comment

umpox Jan 31, 2024

Choose a reason for hiding this comment

abeatrix Jan 31, 2024

Choose a reason for hiding this comment

abeatrix commented Jan 31, 2024

abeatrix commented Jan 31, 2024

toolmantim left a comment

Choose a reason for hiding this comment

abeatrix commented Jan 30, 2024 •

edited