Configuration to exclude SKIPPED tests in Allure Report

[Sanjay-Bhatta]

I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

Current behavior is Allure Tests include skip tests as well

If the current behavior is a bug, please provide the steps to reproduce and if possible a

I have a total of 3 tests out of which 1 PASS, 1 FAIL & 1 SKIPPED
When I generate allure report, it says -
Total - 3
Pass - 1
Fail - 1
Skipped - 1

Pass percentage - 33.33%

What is the expected behavior?

There should be a way to ignore the Skipped test cases. It will be good to have a configurable parameter to calculate the percentage. In my case I would like the pass percentage to be 50% as the Skipped test did not execute

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Allure version	2.8.1
Test framework	testng@6.8
Allure integration	allure-testng@2.0-BETA11
Generate report using	allure-maven@2.18

Other information

Please let me know if there is already a way to configure this

[patotskiy]

@baev
Hi Dmitry, I also need such functionality. It is possible to exclude skip tests from final pass rate?

[pgavriluk]

A few months ago, I was looking for the same option, to remove skipped tests from the Allure statistics. Couldn't find it, so I ended writing my own statistics to display in the report email and slack channel, but the problem that if somebody clicks on the Allure report link it shows wrong numbers. So, yeah, it would be great to have it as part of the Allure report.
Thanks.

[mszajowskiwr]

I have similar issue, such configuration option would be a blast.

[JohnnyChiang]

Also need this config, too.

[Badhron]

yes please i need it too.

[fleytman]

Тоже очень нужна данная функция. Думал, что @muted должен это делать, но нет. Схожие запросы есть с 2017 года. Не хочется какие-то свои велосипеды горадить.

[shridharkalagi]

Is there any workaround to solve this for the time being?

[pgavriluk]

@shridharkalagi, yes, instead of using @Disabled/@ignore annotations (or any other annotation you might use in your framework to skip tests) I add tag "ignored" to the tests I want to ignore/skip. Like this @tag("ignored").

Now, when you run your tests just don't run tests with that tag "ignored".
For me, because I use mvn to run, it's:
mvn clean test -Dgroups="{any tags you want to run, or just keep it empty}" -DexcludedGroups="ignored"

This way no ignored tests will be shown in Allure reports, but they are also won't be part of your total tests that you run. If it’s fine for you, it can be a workaround to keep Allure statistics cleaner.

[ManishIBM]

I also need this functionality

[Isidor2811]

any news here?

[anagornaia]

Yeah, we need the same functionality. On our project, we need to skip tests on condition and would be nice for them not to appear in the report as the statistic of run shows wrong numbers

[bogalx]

same here. please add this functionality or some workaround (in Java)

[ManishIBM]

Is there any workaround to solve this for the time being?

you can use hook in conftest.py, following is the code sample:

def pytest_collection_modifyitems(items, config):
deselect_tcs = []
selected_tcs = []
for tc_item in items:
res = check_and_disable_test(tc_item)
if res is not None:
deselect_tcs.append(tc_item)
else:
selected_tcs.append(tc_item)

config.hook.pytest_deselected(items=deselect_tcs)
items[:] = selected_tcs

[bogalx]

Is there any workaround to solve this for the time being?

you can use hook in conftest.py, following is the code sample:

def pytest_collection_modifyitems(items, config): deselect_tcs = [] selected_tcs = [] for tc_item in items: res = check_and_disable_test(tc_item) if res is not None: deselect_tcs.append(tc_item) else: selected_tcs.append(tc_item)

config.hook.pytest_deselected(items=deselect_tcs)
items[:] = selected_tcs

Thanks but I mean in Java

[Khouloud-moalla]

@tag("ignored")

I have similar issue, such configuration option would be a blast.

what is the configuration!

[pgajeshwar-coupa]

Good to have this configuration option in allure

[easy-selenium-2022]

After spending more than two weeks and a long shell script that worked on my local runs, but failed on jenkins... this is what finally worked for me. Just two lines of code:

cd $PWD/target
grep -l status.*skipped allure-results/*result.json
grep -l status.*skipped allure-results/*result.json | xargs rm

Note: You need https://plugins.jenkins.io/postbuildscript/ jenkins post build and it needs to go right before Allure Report post build. Also, this is only removing the allure json result file that has the skipped status. You can still view the skipped results in the testng or junit reports.

[TimeInvestor]

We also need such a configuration.

[storenth]

Just use bash to remove skipped tests from the test results before report generation step:

grep -rl '"status": "skipped"' allure-results | xargs rm -rf

[romixb]

Ok, so i encountered a similar problem recently - ui tests that should be ignored with a specific configuration (e.g. specific browsers)through annotations are marked as Skipped, are displayed in TestOps report and included in the run statistics.

Using Ignore or tags was not an option because as mentioned before, one test is executed in different configs.
I tried to use allure result modification via grep in cmd but we use withAllureUpload function in Jenkins dsl pipeline that wraps the testrun and sends results on the go, so post modification didn't work

After a bit of debugging i found a solution that suites our case - cancel the default behaviour of createFakeResult(final ITestContext context, final ITestNGMethod method) from AllureTestNg listener by overriding it with an empty method

@Override
protected void createFakeResult(ITestContext context, ITestNGMethod method) {
}

This will affect those cases when skipping tests is allowed in your test run and expected in stats.

[QingqinLi]

I need this functionality too

[MrKozhin]

Need this config badly!

[wazzeps]

Also need this config.

[baev]

BTW, there is an option for TestNG integration to hide disabled tests from the report. See https://allurereport.org/docs/testng-configuration/#alluretestnghidedisabledtests

From what I see, there are multiple different requests from the community:

1. To exclude skipped tests in statistics. This is a language-agnostic report feature.
2. Hide disabled tests from TestNG — there is a configuration option; see the link above.
3. Some other "hide skipped" requests from other languages — please create a separate issue/discussion for that

I suggest only discussing the number one within the scope of the issue.

[baev]

After internal discussion, we decided to change the behaviour unconditionally and exclude unknown and skipped statuses from the passed rate calculation. I'll try to include this change in the next release of the Allure Report (aiming to make it this year).

In case of positive feedback from the community, we'll propagate the change to Allure TestOps as well.

[baev]

The RR allure-framework/allure2#2219

[baev]

After internal discussion, we decided to change the behaviour unconditionally and exclude unknown and skipped statuses from the passed rate calculation. I'll try to include this change in the next release of the Allure Report (aiming to make it this year).

released as 2.25.0

[tomkeiev]

@baev I've downloaded the released version from https://github.com/allure-framework/allure2/releases/download/2.25.0/allure-2.25.0.zip and generated a report on a local machine but I still have the same results as before

[KeaganGonsalvez]

I have also updated it to 2.25.0 and skipped tests are still being included in the summary.

[baev]

Since this is a front-end-only change, please, try to clear the browser cache and check again.

For me it working just fine:

[wazzeps]

Could anyone confirm that clearing the cache helps?

I tried clearing the cache / opening the page in incognito, but disabled tests continue to be taken into account.

[KeaganGonsalvez]

it doesn't work for me I cleared the cache but it still accounts for skipped tests.

[baev]

Could you please check https://baev.github.io/testng-java-maven/5/# ? It shows 100% for me, ignoring unknown & skipped statuses in the success rate calculation (source code https://github.com/baev/testng-java-maven )

[wazzeps]

Could you please check https://baev.github.io/testng-java-maven/5/# ? It shows 100% for me, ignoring unknown & skipped statuses in the success rate calculation (source code https://github.com/baev/testng-java-maven )

Can confirm it shows 100%.

I checked it on the browser where I tried to open my reports.

[wazzeps]

After some research I managed to find the source of the problem.

Allure Gradle plugin 2.11.2 with default configuration generates a report with disabled statuses included.
While calling allure generate manually creates the correct report.

@baev can we configure AGP to fix this?

[baev]

can we configure AGP to fix this?

allure {
    version.set("2.25.0")
}

[adenzhong]

It works as expected, thanks.