Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command: Remove model override for Test command #3343

Merged
merged 5 commits into from
Mar 8, 2024

Conversation

abeatrix
Copy link
Contributor

@abeatrix abeatrix commented Mar 7, 2024

Removes model override for the Generate Unit Test command

This pull request removes the model override for the Generate Unit Test command, granting users the flexibility to utilize any edit model for generating unit tests.

Previously, we defaulted to using Claude 2.0 due to observed regressions with other models, as detailed in this discussion. However, with the introduction of Claude 3 and the availability of more alternative models, Claude 2.0 is no longer the sole option for generating effective unit tests. Therefore, we can remove the override for the test command, allowing users to choose from a wider range of models.

Moreover, the regressions we noticed previously in ChatGPT 4.0 and Claude 2.1 models are no longer present from what I've tested so far.

ChatGPT 4.0:
image

Claude 2.1:
image

ChatGPT 3.5 and Claude Instant struggle to follow the instructions fully due to their smaller context windows, but I don't see a point of overriding them if we have other alternatives that are proven to work fine that the users can use.

Test plan

Pick any edit model in the edit menu before running the unit test command.

@abeatrix abeatrix requested review from umpox and a team March 7, 2024 22:53
@abeatrix abeatrix merged commit 57cf117 into main Mar 8, 2024
15 of 16 checks passed
@abeatrix abeatrix deleted the bee/remove-override-test branch March 8, 2024 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants