Response Assertion "Ignore Status" cancels preceding assertion failures.

[asfimport]

deane.sloan (Bug 38824):
Overview Description:
The Response Assertion setting "Ignore Status" causes a Response Assertion to
cancel/drop any preceding response assertion failures.

Steps to Reproduce:

1. Create a test plan (+thread group etc) with a HTTP Request sampler and two
   child Response Assertions.
2. Configure the thread group to "Stop Thread" after a sampler error and to
   loop forever.
3. Configure the HTTP Request sampler to a URL where you know it will return a
   certain string successfully.
4. Configure the first child Response Assertion with a pattern set to "NOT"
   contain the string that you know will be returned (i.e. it will cause it to
   fail).
5. Ensure the "Ignore status" checkbox on the first child Response Assertion is
   not set.
6. Configure the last child response assertion with a pattern set to contain
   the string that you know will be returned (i.e. it shouldn't fail).
7. Tick the "Ignore status" checkbox on the last child Response Assertion.

Actual Results:
The HTTP Request sample is not set into an error state when only the first
Response Assertion is triggered/fails on a pattern. The first assertion is
correctly recorded in the log as failed - however the HTTP sample shows as
being in success and the thread continues.

Expected Results:
When the first Response Assertion is triggered/fails on a pattern, set the HTTP
Request sample to be in error and stop the thread. The documented behaviour of
the "Ignore Status" does not indicate that preceding assertions are effectively
ignored if this value is set.

Build/Platform:
Windows XP SP2
Java 2 Runtime Environment, Standard Edition build 1.5.0_06-b05
Apache JMeter version 2.1.1

Thanks

Severity: trivial
OS: Windows XP

[asfimport]

Sebb (migrated from Bugzilla):
This is behaving as designed.

The documentation states:

"When the Ignore Status checkbox is selected, the Response status is forced to
successful before evaluating the Assertion."

The purpose of the checkbox is to allow the response status to be checked, even
if the request has failed, e.g. it allows one to check for a 404.

[asfimport]

deane.sloan (migrated from Bugzilla):
Thanks -

Probably led astray a bit by the doco statement "There is no difference between
setting up one Assertion with multiple patterns and setting up multiple
Assertions with one pattern each (assuming the other options are the same). " -
the statement seems to be incorrect based on your response and observations.

[asfimport]

deane.sloan (migrated from Bugzilla):
My bad - created a contrived example presuming the issue was a bug rather than
a doco thing.

Simply put - splitting a check where you want to make your own determination
about HTTP 400+/500+ responses into multiple response assertions using "Ignore
Status" won't reliably work - the patterns need to be in the one response
assertion.

The documentation around response assertion has a statement that would lead you
to believe otherwise ("There is no difference between...").
Just a minor doco thing - but still has the same impact if you made the choice
to split your assertion up based on the statement...

BTW - this seemed like a viable approach to let a known HTTP 500 error through
for a period of time by trapping everything but...

Thanks for taking the time to read this.

[asfimport]

Sebb (migrated from Bugzilla):
(In reply to comment 2)

Thanks -
Probably led astray a bit by the doco statement "There is no difference
between
setting up one Assertion with multiple patterns and setting up multiple
Assertions with one pattern each (assuming the other options are the
same). " -
the statement seems to be incorrect based on your response and observations.

Your example did not use the same options...

[asfimport]

Sebb (migrated from Bugzilla):
I'll see about improving the documentation for the check-box.

[asfimport]

Sebb (migrated from Bugzilla):
Documentation has been updated.

Ignore Status should only be used on the first Assertion