Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to the integration tests build #1313

Merged
merged 14 commits into from
Jul 20, 2020

Conversation

leodido
Copy link
Member

@leodido leodido commented Jul 13, 2020

What type of PR is this?

/kind cleanup

/kind documentation

/kind feature

Any specific area of the project related to this PR?

/area build

/area tests

What this PR does / why we need it:

  • updating the CMakeLists to provide targets for obtaining and placing in the correct directory all the integration test fixtures (ie., trace files)
  • improving the docs to locally run integration tests
  • updating the run_regression_tests.sh script
  • building & publishing the new falcosecurity/falco-tester image
    • with a different tag (ie., update-integration-tests-build)
    • temporarily update the CI to use this falcosecurity/falco-tester:update-integration-tests-build image
    • tag it as latest and switch back the CI to normal flow
  • reporting the run_regression_tests.sh changes into the Falco website (update instructions to run integration test locally falco-website#227)

Which issue(s) this PR fixes:

Special notes for your reviewer:

Ready for review.

/cc @fntlnz
/cc @leogr

Does this PR introduce a user-facing change?:

docs(test): step-by-step instructions to run integration tests locally

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
This make target calls the `trace-files-psp`, `trace-files-k8s-audit`,
`trace-files-base-scap` targets to place all the integration test
fixtures in the proper position.

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
…t suites

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
@poiana poiana added size/M and removed size/S labels Jul 13, 2020
…suite

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
options

The following options have been added:
* -v (verbose)
* -p (prepare falco_traces test suite)
* -b (specify custom branch for downloading trace files)
* -d (specify the build directory)

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
@poiana poiana added size/L and removed size/M labels Jul 14, 2020
…uite locally

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
…ite in a non-interactive way

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
.circleci/config.yml Outdated Show resolved Hide resolved
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
@leodido
Copy link
Member Author

leodido commented Jul 20, 2020

/hold cancel

Copy link
Member

@leogr leogr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM,

though I still got some errors.

  1. When running locally using virtualenv and falco_test.yaml, although I executed all preparation steps (including ones for falco_traces),
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
BUILD_DIR="../build" avocado run --mux-yaml falco_tests.yaml --job-results-dir /tmp/job-results -- falco_test.py
deactivate

just one test is not passing:

 (83/96) falco_test.py:FalcoTest.test;in_operator_netmasks-ea28: ERROR: Falco command "../build/userspace/falco/falco -r /home/leogr/code/github.com/falcosecurity/falco/test/rules/detect_connect_using_in.yaml   -c /home/leogr/code/github.com/falcosecurity/falco/test/../falco.yaml -e ../build/test/trace_files/connect_localhost... (0.36 s)
  1. Also, when using ./run_regression_tests.sh -v although I executed all preparation (including ones for falco_tests_package) many tests are not passing.

I'm still investigating to figure out the problems

@leodido
Copy link
Member Author

leodido commented Jul 20, 2020

Verify your local build is a BUILD_TYPE=Release one @leogr

fntlnz
fntlnz previously approved these changes Jul 20, 2020
Copy link
Contributor

@fntlnz fntlnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works very good for me. You made it very easy now (again) - Thanks for crafting this PR.

LGTM

@poiana
Copy link

poiana commented Jul 20, 2020

LGTM label has been added.

Git tree hash: a84fd9c3ece72a7d70b52137243cb50ea08e5f39

Copy link
Member

@leogr leogr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have posted just two suggestions to avoid headaches when using this.

Could we include these clarifications before merging this?

test/README.md Outdated Show resolved Hide resolved
test/README.md Show resolved Hide resolved
…ild of Falco


Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
@poiana poiana added the lgtm label Jul 20, 2020
@poiana
Copy link

poiana commented Jul 20, 2020

LGTM label has been added.

Git tree hash: 2cd60f1671d7db0e4b39d9d27d83a3e6ecb06f4c

@poiana
Copy link

poiana commented Jul 20, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fntlnz, Kaizhe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

small adjustments to the integration test readme
6 participants