Rearrange tests & fix CI #4596

Pwuts · 2023-06-06T15:25:20Z

Background

Not all tests are in the right location (tests/ vs tests/(unit|integration)/)
Challenges are run as part of the integration tests, while it can be useful to run only integration tests and not the challenges
We have a bunch of bogus unit tests which contribute more headache than valuable test coverage
Docker CI will succeed even if some of the tests fail

Changes

Rearrange all tests into tests/unit, tests/integration and tests/challenges
- Some tests also moved from tests/integration to tests/unit
Clean up bogus unit tests
Reduce Docker CI to only run unit+integration tests
Fix Docker CI to actually fail when the tests fail

Other changes:

Fixed get_text_summary which broke in Pass Configs to Commands and remove CFG = Config() in the commands/ folder #4328

Documentation

References to tests/integration/challenges or tests.integration.challenges in docs were also updated.

Test Plan

Run CI and check result + coverage

PR Quality Checklist

My pull request is atomic and focuses on a single change.
I have thoroughly tested my changes with multiple different prompts.
I have considered potential risks and mitigations for my changes.
I have documented my changes clearly and comprehensively.
I have not snuck in any "extra" small tweaks changes.

I have run the following commands against my code to ensure it passes our linters:

black .
isort .
mypy
autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports autogpt tests --in-place

vercel · 2023-06-06T15:25:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		Jun 6, 2023 4:49pm

github-actions · 2023-06-06T15:48:11Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

…) in CI

github-actions · 2023-06-06T15:52:19Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T15:53:06Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T16:33:38Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T16:36:42Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T16:39:11Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T16:44:57Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

github-actions · 2023-06-06T16:45:44Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

codecov · 2023-06-06T16:47:04Z

Codecov Report

Patch coverage has no change and project coverage change: -0.20 ⚠️

Comparison is base (ee6b97e) 69.58% compared to head (57eb545) 69.38%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4596      +/-   ##
==========================================
- Coverage   69.58%   69.38%   -0.20%     
==========================================
  Files          72       72              
  Lines        3551     3551              
  Branches      569      569              
==========================================
- Hits         2471     2464       -7     
- Misses        889      895       +6     
- Partials      191      192       +1

Impacted Files	Coverage Δ
autogpt/app.py	`50.00% <0.00%> (+8.88%)`	⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

github-actions · 2023-06-06T16:49:44Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

Pwuts · 2023-06-06T16:56:46Z

Someone will have to take a look at test_json_parser.py and _test_json_parser.py, determine what we actually want fix_and_parse_json() to be capable of, and consolidate.

ntindle

Make sure the mypy config is updated

ntindle · 2023-06-06T17:05:09Z

Someone will have to take a look at test_json_parser.py and _test_json_parser.py, determine what we actually want fix_and_parse_json() to be capable of, and consolidate.

We should make a new issue for this rather than holding this one up

waynehamadi · 2023-06-06T17:49:49Z

Someone will have to take a look at test_json_parser.py and _test_json_parser.py, determine what we actually want fix_and_parse_json() to be capable of, and consolidate.

We should make a new issue for this rather than holding this one up

#4597

Co-authored-by: Reinier van der Leer <github@pwuts.nl> Co-authored-by: Nicholas Tindle <nick@ntindle.com> Co-authored-by: Nicholas Tindle <nicktindle@outlook.com> Co-authored-by: k-boikov <64261260+k-boikov@users.noreply.github.com> Co-authored-by: merwanehamadi <merwanehamadi@gmail.com> Co-authored-by: Merwane Hamadi <merwanehamadi@gmail.com> Co-authored-by: Richard Beales <rich@richbeales.net> Co-authored-by: Luke K <2609441+lc0rp@users.noreply.github.com> Co-authored-by: Luke K (pr-0f3t) <2609441+lc0rp@users.noreply.github.com> Co-authored-by: Erik Peterson <e@eriklp.com> Co-authored-by: Auto-GPT-Bot <github-bot@agpt.co> Co-authored-by: Benny van der Lans <49377421+bfalans@users.noreply.github.com> Co-authored-by: Jan <jan-github@phobia.de> Co-authored-by: Robin Richtsfeld <robin.richtsfeld@gmail.com> Co-authored-by: Marc Bornträger <marc.borntraeger@gmail.com> Co-authored-by: Stefan Ayala <stefanayala3266@gmail.com> Co-authored-by: javableu <45064273+javableu@users.noreply.github.com> Co-authored-by: DGdev91 <DGdev91@users.noreply.github.com> Co-authored-by: Kinance <kinance@gmail.com> Co-authored-by: digger yu <digger-yu@outlook.com> Co-authored-by: David <scenaristeur@gmail.com> Co-authored-by: gravelBridge <john.tian31@gmail.com> Fix Python CI "update cassettes" step (#4591) fix CI (#4596) Fix inverted logic for deny_command (#4563) fix current_score.json generation (#4601) Fix duckduckgo rate limiting (#4592) Fix debug code challenge (#4632) Fix issues with information retrieval challenge a (#4622) fix issues with env configuration and .env.template (#4630) Fix prompt issue causing 'No Command' issues and challenge to fail (#4623) Fix benchmark logs (#4653) Fix typo in docs/setup.md (#4613) Fix run.sh shebang (#4561) Fix autogpt docker image not working because missing prompt_settings (#4680) Fix execute_command coming from plugins (#4730)

* Rearrange tests into unit/integration/challenge categories * Fix linting + `tests.challenges` imports * Fix obscured duplicate test in test_url_validation.py * Move VCR conftest to tests.vcr * Specify tests to run & their order (unit -> integration -> challenges) in CI * Fail Docker CI when tests fail * Fix import & linting errors in tests * Fix `get_text_summary` * Fix linting errors * Clean up pytest args in CI * Remove bogus tests from GoCodeo

Rearrange tests into unit/integration/challenge categories

66c669d

github-actions bot added the size/m label Jun 6, 2023

vercel bot temporarily deployed to Preview June 6, 2023 15:26 Inactive

Pwuts added ci testing labels Jun 6, 2023

Pwuts self-assigned this Jun 6, 2023

Fix linting + tests.challenges imports

e4534e3

github-actions bot added size/l and removed size/m labels Jun 6, 2023

vercel bot temporarily deployed to Preview June 6, 2023 15:36 Inactive

Pwuts added 2 commits June 6, 2023 17:39

Fix obscured duplicate test in test_url_validation.py

519d496

Move VCR conftest to tests.vcr

c1cf1a3

github-actions bot added size/xl and removed size/l labels Jun 6, 2023

Specify tests to run & their order (unit -> integration -> challenges…

0744b73

…) in CI

Fail Docker CI when tests fail

e9c6d31

Pwuts added 2 commits June 6, 2023 18:32

Fix import & linting errors in tests

bc86f55

Fix get_text_summary

0c42faf

Fix linting errors

150a387

Pwuts force-pushed the dx/rearrange+fix-tests branch from f8f1c0e to 150a387 Compare June 6, 2023 16:38

Clean up pytest args in CI

a16dce3

Pwuts force-pushed the dx/rearrange+fix-tests branch from 79a1e46 to a16dce3 Compare June 6, 2023 16:45

Remove bogus tests from GoCodeo

57eb545

Pwuts marked this pull request as ready for review June 6, 2023 16:50

ntindle approved these changes Jun 6, 2023

View reviewed changes

waynehamadi mentioned this pull request Jun 6, 2023

Refactor fix_and_parse_json #4597

Closed

1 task

waynehamadi approved these changes Jun 6, 2023

View reviewed changes

waynehamadi merged commit dafbd11 into master Jun 6, 2023
12 of 13 checks passed

waynehamadi deleted the dx/rearrange+fix-tests branch June 6, 2023 17:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rearrange tests & fix CI #4596

Rearrange tests & fix CI #4596

Pwuts commented Jun 6, 2023 •

edited

vercel bot commented Jun 6, 2023 •

edited

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

codecov bot commented Jun 6, 2023 •

edited

github-actions bot commented Jun 6, 2023

Pwuts commented Jun 6, 2023

ntindle left a comment

ntindle commented Jun 6, 2023

waynehamadi commented Jun 6, 2023

Rearrange tests & fix CI #4596

Rearrange tests & fix CI #4596

Conversation

Pwuts commented Jun 6, 2023 • edited

Background

Changes

Documentation

Test Plan

PR Quality Checklist

vercel bot commented Jun 6, 2023 • edited

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 6, 2023

codecov bot commented Jun 6, 2023 • edited

Codecov Report

github-actions bot commented Jun 6, 2023

Pwuts commented Jun 6, 2023

ntindle left a comment

Choose a reason for hiding this comment

ntindle commented Jun 6, 2023

waynehamadi commented Jun 6, 2023

Pwuts commented Jun 6, 2023 •

edited

vercel bot commented Jun 6, 2023 •

edited

codecov bot commented Jun 6, 2023 •

edited