Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate moving integration tests to a more solid foundation #736

Closed
danielmitterdorfer opened this issue Jul 22, 2019 · 1 comment
Closed

Investigate moving integration tests to a more solid foundation #736

danielmitterdorfer opened this issue Jul 22, 2019 · 1 comment
Labels
enhancement Improves the status quo :misc Changes that don't affect users directly: linter fixes, test improvements, etc.

Comments

@danielmitterdorfer
Copy link
Member

We currently use a shell script for our integration tests. This has several limitiations:

  • We need to resort to adhoc solutions (e.g. we need Perl or require Docker for search / replace)
  • Limited ability for assertions
  • We cannot report test failures properly
  • Hard to extend

We should instead investigate how we can properly support integration tests with Python (e.g. via py.test).

Requirements:

  • Should avoid all of the shorcomings above
  • We should be able to run unit tests and integration tests separately. Ideally we can also clearly see in the source code structure which tests are unit tests and which ones are integration tests (separate root directories?)
  • We should not rely on external tools such as Perl (note: Docker is required for Docker tests). Note: In order to reduce scope, we might want to port the existing tests first 1:1 and then improve them in follow-up PRs
@danielmitterdorfer danielmitterdorfer added enhancement Improves the status quo :misc Changes that don't affect users directly: linter fixes, test improvements, etc. labels Jul 22, 2019
@danielmitterdorfer danielmitterdorfer added this to the 1.x milestone Jul 22, 2019
@danielmitterdorfer danielmitterdorfer added the help wanted We'd be happy about a community contribution label Sep 11, 2019
@danielmitterdorfer danielmitterdorfer mentioned this issue Dec 2, 2019
Merged
danielmitterdorfer added a commit to danielmitterdorfer/rally that referenced this issue Feb 26, 2020
@danielmitterdorfer
Run integration tests with pytest
169b113 
With this commit we add a proof-of-concept implementation of how
integration tests could look like with py.test instead of shell scripts.
For the proof of concept we migrate the integration tests for the list
subcommands and source builds.

It also spins up an Elasticsearch metrics store and checks that any
preconditions (basically Docker being present and up and running) are
met. Finally, it also integrates into CI by writing JUnit XML files for
test results.

Relates elastic#736
@danielmitterdorfer danielmitterdorfer mentioned this issue Feb 26, 2020
Merged
danielmitterdorfer added a commit that referenced this issue Mar 26, 2020
@danielmitterdorfer
Run integration tests with pytest (#919)
b33da19 
With this commit we add a proof-of-concept implementation of how
integration tests could look like with py.test instead of shell scripts.
For the proof of concept we migrate the integration tests for the list
subcommands and source builds.

It also spins up an Elasticsearch metrics store and checks that any
preconditions (basically Docker being present and up and running) are
met. Finally, it also integrates into CI by writing JUnit XML files for
test results.

Relates #736
@danielmitterdorfer
Copy link
Member Author

Initially addressed in #919. We will track the actual migration in #975.

@danielmitterdorfer danielmitterdorfer removed the help wanted We'd be happy about a community contribution label Apr 27, 2020
@danielmitterdorfer danielmitterdorfer removed this from the 1.x milestone Apr 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Improves the status quo :misc Changes that don't affect users directly: linter fixes, test improvements, etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@danielmitterdorfer