update BATS & helper + minor updates to BATS variables #2988

georglauterbach · 2023-01-08T14:33:10Z

Description

NOTE: Please review commit by commit :)

BATS core:
--> previous: 1.7.0
--> now: 1.8.2

BATS assert:
--> previous: ffe84ea5dd43b568851549b3e241db150c12929c
--> now: 2.1.0

Type of change

Improvement (non-breaking change that does improve existing functionality)

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
If necessary I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

BATS core: previous: 1.7.0 now: 1.8.2 BATS assert: previous: ffe84ea5dd43b568851549b3e241db150c12929c now: 2.1.0

Move from `TEST_NAME_PREFIX` to `BATS_TEST_NAME_PREFIX` which BATS will automatically prepend to tests in a given file. Didn't know why I did not see this earlier. The prefix names were made uniform too.

The test would very sporadically fail in case Amavis could not print its log in-time. We will now wait at most 5 secs for Amavis to print its log.

search-and-replace did not catch everything the last time

polarathene

Oh I somehow missed the assert lib finally getting a tagged release (I had been subscribed to the issue closed in Oct 2022) we could use instead of a master branch commit, thanks for sorting this out 🎉

polarathene · 2023-01-08T23:18:44Z

Makefile

+	@ shopt -s globstar ; ./test/bats/bin/bats $(BATS_FLAGS) \
+		--no-parallelize-within-files --jobs $(BATS_PARALLEL_JOBS) test/$@/**/*.bats


Prefer vertically listing the explicit options, easier to grok and track changes 😅

Suggested change

@ shopt -s globstar ; ./test/bats/bin/bats $(BATS_FLAGS) \

--no-parallelize-within-files --jobs $(BATS_PARALLEL_JOBS) test/$@/**/*.bats

@ shopt -s globstar ; ./test/bats/bin/bats $(BATS_FLAGS) \

--no-parallelize-within-files \

--jobs $(BATS_PARALLEL_JOBS) \

test/$@/**/*.bats

Side-notes (nothing below requiring change for this PR)

A reminder that --no-parallelize-within-files prevents --jobs 1 for running a parallel set sequentially.

Although I wouldn't worry about this much as I'm doubtful the default of --jobs 2 will cause anyone running these tests a problem, since we can still run a few individual tests with make test/file1,file2,file3 easily enough.

If of interest to you:

There is also a tagging feature to be selective on what test cases to run in a file(s).

--no-parallelize-within-files could be considered for removal if we can confidently evaluate which test files need to explicitly opt-out via export BATS_NO_PARALLELIZE_WITHIN_FILE=true in setup_file().

I have merged this PR to make CI pass again, but we can tackle the suggestions in another PR - I don't really mind the changes to the Makefile since it's only a stylistic change :)

polarathene · 2023-01-08T23:59:04Z

test/tests/parallel/set1/spam_virus/amavis.bats

+@test "SpamAssassin integration should be active" {
+  # give Amavis just a bit of time to print out its full debug log
+  run repeat_in_container_until_success_or_timeout 5 "${CONTAINER_NAME}" grep 'ANTI-SPAM-SA' /var/log/mail/mail.log
+  assert_success
  assert_output --partial 'loaded'
  refute_output --partial 'NOT loaded'


This is works too 👍 (I noticed this locally and was going to use an existing helper method to wait for the amavis service to be ready)

Alternatively, instead of "grep-ing until success", grep until a string appears:

tail -F /var/log/mail/mail.log | grep --max-count 1 'ANTI-SPAM-SA' timeout 5 tail -F /var/log/mail/mail.log | grep --max-count 1 'ANTI-SPAM-SA'

grep until a string appears

I have been refactoring / debugging a test of the past couple days where a similar requirement with counting was necessary and being a hassle for me to shell-fu my way through, I may want to switch to your tip here now 😂

fyi: if you care for the output, use --max-count 1, otherwise just -q (grep will then quit after first match)

update BATS & helper

1a6705b

BATS core: previous: 1.7.0 now: 1.8.2 BATS assert: previous: ffe84ea5dd43b568851549b3e241db150c12929c now: 2.1.0

georglauterbach added area/tests kind/update Update an existing feature, configuration file or the documentation labels Jan 8, 2023

georglauterbach added this to the v12.0.0 milestone Jan 8, 2023

georglauterbach requested review from casperklein and polarathene January 8, 2023 14:33

georglauterbach self-assigned this Jan 8, 2023

use BATS_TEST_NAME_PREFIX

050feb1

Move from `TEST_NAME_PREFIX` to `BATS_TEST_NAME_PREFIX` which BATS will automatically prepend to tests in a given file. Didn't know why I did not see this earlier. The prefix names were made uniform too.

georglauterbach changed the title ~~update BATS & helper~~ update BATS & helper + minor updates to BATS variables Jan 8, 2023

georglauterbach added 2 commits January 8, 2023 16:16

adjust Amavis test

5651052

The test would very sporadically fail in case Amavis could not print its log in-time. We will now wait at most 5 secs for Amavis to print its log.

remove left-over TEST_NAME_PREFIX

d7eaf32

search-and-replace did not catch everything the last time

casperklein approved these changes Jan 8, 2023

View reviewed changes

georglauterbach mentioned this pull request Jan 8, 2023

Add tools (ping & dig) to the image #2989

Merged

11 tasks

polarathene approved these changes Jan 8, 2023

View reviewed changes

polarathene reviewed Jan 8, 2023

View reviewed changes

Merge branch 'master' into update/bats

6274c9f

georglauterbach merged commit 41c44cb into master Jan 9, 2023

georglauterbach deleted the update/bats branch January 9, 2023 07:54

georglauterbach mentioned this pull request Jan 9, 2023

Fix several typos #2990

Merged

11 tasks

polarathene mentioned this pull request Jan 11, 2023

tests(refactor): Adjust mail_changedetector + change detection helpers #2997

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update BATS & helper + minor updates to BATS variables #2988

update BATS & helper + minor updates to BATS variables #2988

georglauterbach commented Jan 8, 2023 •

edited

Loading

polarathene left a comment •

edited

Loading

polarathene Jan 8, 2023 •

edited

Loading

georglauterbach Jan 9, 2023 •

edited

Loading

polarathene Jan 8, 2023

casperklein Jan 9, 2023

polarathene Jan 10, 2023 •

edited

Loading

casperklein Jan 10, 2023

		@ shopt -s globstar ; ./test/bats/bin/bats $(BATS_FLAGS) \
		--no-parallelize-within-files --jobs $(BATS_PARALLEL_JOBS) test/$@/*/.bats

update BATS & helper + minor updates to BATS variables #2988

update BATS & helper + minor updates to BATS variables #2988

Conversation

georglauterbach commented Jan 8, 2023 • edited Loading

Description

Type of change

Checklist:

polarathene left a comment • edited Loading

Choose a reason for hiding this comment

polarathene Jan 8, 2023 • edited Loading

Choose a reason for hiding this comment

Side-notes (nothing below requiring change for this PR)

georglauterbach Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

polarathene Jan 8, 2023

Choose a reason for hiding this comment

casperklein Jan 9, 2023

Choose a reason for hiding this comment

polarathene Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

casperklein Jan 10, 2023

Choose a reason for hiding this comment

georglauterbach commented Jan 8, 2023 •

edited

Loading

polarathene left a comment •

edited

Loading

polarathene Jan 8, 2023 •

edited

Loading

georglauterbach Jan 9, 2023 •

edited

Loading

polarathene Jan 10, 2023 •

edited

Loading