ci: initial acceptance test workflow#41552
Conversation
phil-davis
force-pushed
the
acceptance-workflow-20260519
branch
2 times, most recently
from
8951ddd to
f63cd28
Compare
| 'ldapAdminPassword' => 'admin', | ||
| 'ldapUsersOU' => 'TestUsers', | ||
| 'ldapGroupsOU' => 'TestGroups', | ||
| 'ldapInitialUserFilePath' => '/../../config/ldap-users.ldif', |
There was a problem hiding this comment.
As I expected, the --convert-config tool didn't do anything special with your YAML aliases (because it runs after the YAML is parsed) so it has duplicated all the blocks that were in your shared common_ldap_suite_context and common_feature_context_params definitions.
You might want to extract all those duplicates out to variables for readability / ease of maintenance?
| # If there were no scenarios in the requested suite or feature that match | ||
| # the requested combination of tags, then Behat exits with an error status | ||
| # and reports "No scenarios" in its output. | ||
| # This can happen, for example, when running core suites from an app and | ||
| # requesting some tag combination that does not happen frequently. Then | ||
| # sometimes there may not be any matching scenarios in one of the suites. | ||
| # In this case, consider the test has passed. | ||
| MATCHING_COUNT=`grep -ca '^No scenarios$' ${TEST_LOG_FILE}` | ||
| if [ ${MATCHING_COUNT} -eq 1 ] | ||
| then | ||
| echo "Information: no matching scenarios were found." | ||
| BEHAT_EXIT_STATUS=0 |
There was a problem hiding this comment.
Since Behat 3.20.0, we have an --allow-no-tests argument for exactly this situation - it will always exit 0 if no tests are found. That might allow you to simplify your bash script a bit?
| echo "Rerun failed scenario: ${FAILED_SCENARIO_PATH}" | ||
| ${BEHAT} ${COLORS_OPTION} --strict -c ${BEHAT_YML} -f pretty ${BEHAT_SUITE_OPTION} --tags ${BEHAT_FILTER_TAGS} ${FAILED_SCENARIO_PATH} -v 2>&1 | tee -a ${TEST_LOG_FILE} | ||
| ${BEHAT} ${COLORS_OPTION} --strict -c ${BEHAT_CONFIG_FILE} -f pretty ${BEHAT_SUITE_OPTION} --tags ${BEHAT_FILTER_TAGS} ${FAILED_SCENARIO_PATH} -v 2>&1 | tee -a ${TEST_LOG_FILE} | ||
| BEHAT_EXIT_STATUS=${PIPESTATUS[0]} | ||
| if [ ${BEHAT_EXIT_STATUS} -eq 0 ] | ||
| then | ||
| # The scenario was not expected to fail but had failed and is present in the | ||
| # unexpected_failures list. We've checked the scenario with a re-run and | ||
| # it passed. So remove it from the unexpected_failures list. | ||
| for i in "${!UNEXPECTED_FAILED_SCENARIOS[@]}" | ||
| do | ||
| if [ "${UNEXPECTED_FAILED_SCENARIOS[i]}" == "$SUITE_SCENARIO" ] | ||
| then | ||
| unset "UNEXPECTED_FAILED_SCENARIOS[i]" | ||
| fi | ||
| done | ||
| else | ||
| echo "webUI test rerun failed with exit status: ${BEHAT_EXIT_STATUS}" | ||
| # The scenario is not expected to fail but is failing also after the rerun. | ||
| # Since it is already reported in the unexpected_failures list, there is no | ||
| # need to touch that again. Continue processing the next scenario to rerun. | ||
| fi |
There was a problem hiding this comment.
I haven't explored / followed the full details of what your bash script does if scenarios fail but it looks like at least some of it is concerned with retrying failed scenarios.
You might want to look into Behat's --rerun and --rerun-only options:
--rerunis the original flag, it will run all scenarios that failed on the last execution, but it runs everything if nothing failed.--rerun-onlywas added in 3.15.0 - it will run all scenarios that failed on the last execution, or just exit 0 without running anything if there were no previous failures.
Note these run all the failures in a single Behat execution, it looks like you may be running a separate Behat process for each scenario - so if the failures are due to state conflicts between scenarios then those might happen again depending on what gets re-run.
There was a problem hiding this comment.
Note these run all the failures in a single Behat execution, it looks like you may be running a separate Behat process for each scenario - so if the failures are due to state conflicts between scenarios then those might happen again depending on what gets re-run.
From memory, I think that I reran them one scenario at a time because I was parsing the list of failed scenarios in Bash, and it was easiest to run just one failed scenario as I found it (I had the exact feature file and line number).
I will have a look at the --rerun-only option. That looks good for me.
| # Find the count of scenarios that passed | ||
| SCENARIO_RESULTS_COLORED=`grep -Ea '^[0-9]+[[:space:]]scenario(|s)[[:space:]]\(' ${TEST_LOG_FILE}` | ||
| SCENARIO_RESULTS=$(echo "${SCENARIO_RESULTS_COLORED}" | sed "s/\x1b[^m]*m//g") |
There was a problem hiding this comment.
We don't guarantee the output of the pretty formatter over time (though we don't expect to change it much).
Rather than attempting to parse it with grep / sed / awk etc, you could look into the JSON formatter added in 3.26.0. JSON is always rendered to a file, so you could also keep the pretty output but use jq or similar to parse the values you need out of the JSON file?
There was a problem hiding this comment.
Yes, I didn't like parsing the "pretty" output, as it can change at any time. I remember it changing one time when extra information was added to the output lines, and I had to realise that and code for it.
JSON (or any other structured data format) will be much better.
There was a problem hiding this comment.
Indeed. I think a few people parsed pretty in the past because the only other option was the JUnit XML and parsing that in the shell isn't that straightforward either. Hopefully the new JSON / jq will help.
phil-davis
force-pushed
the
acceptance-workflow-20260519
branch
from
eabecb8 to
bf458e0
Compare
phil-davis
force-pushed
the
acceptance-workflow-20260519
branch
from
bf458e0 to
0f0279c
Compare
This fixes log file messages like
{"reqId":"rRziQ2UaFono1r96TEdK","level":3,"time":"2026-05-19T12:44:35+00:00","remoteAddr":"::1",
"user":"Brian","app":"PHP","method":"PUT",
"url":"\/server\/remote.php\/webdav\/textfile0.txt-chunking-42-3-2",
"message":"Trying to access array offset on false at
\/var\/www\/html\/server\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#475"}
This will fix log file messages like
{"reqId":"oAKAHIupnZUz4ujYoFqY","level":3,"time":"2026-05-19T12:46:31+00:00","remoteAddr":"::1",
"user":"--","app":"PHP","method":"POST",
"url":"\/server\/ocs\/v1.php\/apps\/federation\/api\/v1\/request-shared-secret",
"message":"strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated
at \/var\/www\/html\/server\/apps\/federation\/lib\/DbHandler.php#298"}
testRemoteUrl was only checking for a responding ownCloud at the provided host. But if the server is installed in a subdirectory then the requests were not sent there. For example: - the cloud server is at http://mydomain.com/cloud - testRemoteUrl checked for a repsonse from http://mydomain.com/status.php Now it checks for a response from http://mydomain.com/cloud/status.php
Bump to the current minor releases. This is what is used in reality in CI, so record that here to make sure that everyone is using at least these versions.
Because 7.4.12 has a limit of 128 on the number of collection aliases. And we need over 200. A way to configure this may be needed.
Using YML for the Behat config has b een deprecated. So make the change to Behat PHP config.
This reverts commit 1d5dc1e.
phil-davis
force-pushed
the
acceptance-workflow-20260519
branch
from
6de7e4a to
d8e9307
Compare
2 participants
TBD