Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make integration tests more robust #2688

Merged
merged 1 commit into from
Jun 7, 2023

Conversation

geyslan
Copy link
Member

@geyslan geyslan commented Feb 7, 2023

chore: make integration tests more robust

- Add generic functions to check received events in different manners:
  - ExpectAllInOrder
  - ExpectAllEqualTo
  - ExpectAtLeastOneOfEach
  - ExpectAnyOfEach
- Add waitForTraceeOutputEvents function to wait for tracee output
  events until buffer fills with certain number of events or
  timeout occurs.
- Add waitforTraceeStop function to wait for tracee to stop or until
  timeout occurs.
- Remove tester.sh concentrating all event filter tests in
  event_filters_test.go.
- Create syscaller tool to easily generate syscalls from the command
  line (strictly for integration tests). It reuses the tracee events
  package.
- Fix goroutine leaks, deadlocks and race conditions.

Fixes: #2681

Initial Checklist

  • There is an issue describing the need for this PR.

Type of change

  • Code refactor (code improvement and/or code removal)

How Has This Been Tested?

make test-unit
sudo make test-integration

@geyslan

This comment was marked as outdated.

@geyslan geyslan force-pushed the 2681-mscope-tests branch 4 times, most recently from 732b989 to 88ac808 Compare February 10, 2023 15:56
@geyslan geyslan changed the title chore: mscope: make scopes tests more robust mscope: make scopes tests more robust Feb 11, 2023
@geyslan geyslan changed the title mscope: make scopes tests more robust mscope: make integration tests more robust Feb 11, 2023
@geyslan geyslan force-pushed the 2681-mscope-tests branch 2 times, most recently from 08b5956 to 752c9df Compare February 14, 2023 22:43
@geyslan geyslan marked this pull request as ready for review February 14, 2023 22:56
.gitignore Show resolved Hide resolved
@geyslan geyslan changed the title mscope: make integration tests more robust make integration tests more robust Feb 14, 2023
@geyslan geyslan force-pushed the 2681-mscope-tests branch 3 times, most recently from 7910d6f to ff39fb8 Compare February 16, 2023 14:38
tests/integration/exec.go Outdated Show resolved Hide resolved
@geyslan geyslan force-pushed the 2681-mscope-tests branch 4 times, most recently from cdb67a1 to 03681b7 Compare May 8, 2023 21:48
@geyslan geyslan marked this pull request as ready for review May 8, 2023 21:58
@josedonizetti
Copy link
Collaborator

I want to review this one

@rafaeldtinoco
Copy link
Contributor

rafaeldtinoco commented May 27, 2023

TODO: We should merge this one at the end of the milestone (no matter if its on the bug fix window) just so we don't change tests while doing the release. @josedonizetti I'll let you take care of it (as you've requested to review and since we've spoken offline about it). Cheers!

- Add generic functions to check received events in different manners:
  - ExpectAllInOrder
  - ExpectAllEqualTo
  - ExpectAtLeastOneOfEach
  - ExpectAnyOfEach
- Add waitForTraceeOutputEvents function to wait for tracee output
  events until buffer fills with certain number of events or
  timeout occurs.
- Add waitforTraceeStop function to wait for tracee to stop or until
  timeout occurs.
- Remove tester.sh concentrating all event filter tests in
  event_filters_test.go.
- Create syscaller tool to easily generate syscalls from the command
  line (strictly for integration tests). It reuses the tracee events
  package.
- Fix goroutine leaks, deadlocks and race conditions.
@josedonizetti josedonizetti merged commit 6c9fcaa into aquasecurity:main Jun 7, 2023
25 checks passed
@rafaeldtinoco
Copy link
Contributor

rafaeldtinoco commented Jun 12, 2023

image

@geyslan shouldn't integration tests binary be inside ./dist/ like all other binaries (including tracee-gpt, for example). Its a convention, I wonder if we could make the same for this binary.

@geyslan
Copy link
Member Author

geyslan commented Jun 12, 2023

@geyslan shouldn't integration tests binary be inside ./dist/ like all other binaries (including tracee-gpt, for example). Its a convention, I wonder if we could make the same for this binary.

Currently the syscaller is used only for the golang integration tests that lie under ./tests/integration/syscaller. But if you have plans to run it from other scripts and so, I think the move makes sense.

@josedonizetti
Copy link
Collaborator

@geyslan shouldn't integration tests binary be inside ./dist/ like all other binaries (including tracee-gpt, for example). Its a convention, I wonder if we could make the same for this binary.

Currently the syscaller is used only for the golang integration tests that lie under ./tests/integration/syscaller. But if you have plans to run it from other scripts and so, I think the move makes sense.

+1, also think that if are not using it elsewhere, as it was created specifically for e2e, we should have it under ./tests/integrations

@rafaeldtinoco
Copy link
Contributor

rafaeldtinoco commented Jun 12, 2023

+1, also think that if are not using it elsewhere, as it was created specifically for e2e, we should have it under ./tests/integrations

Problem with that is now we are not "out-of-tree" building environment just because of this binary. We have just spread leftovers and artifacts to internal directories in the source tree directory (which is not what we should do IMO).

Every time a new binary is created, we have to ignore that binary (if its dir is not already ignored). Every time we clean things, we need to clean the generated binary instead of the out-of-tree directory (dist).

@geyslan geyslan deleted the 2681-mscope-tests branch July 31, 2023 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

make integration tests more robust
4 participants