-
Notifications
You must be signed in to change notification settings - Fork 41
Tests Structure and Conventions (v2.x)
LightBDD is implemented on top of the well known test frameworks which makes it very easy to adopt.
Test classes are treated as Features, where:
Code | Meaning | Notes |
---|---|---|
class name | feature name | preferred method names with underscores |
class level [FeatureDescription] attribute, NUnit [Description] attribute | feature description (narration) | optional, [FeatureDescription] precedes implementation specific attributes |
class level [Label] attribute | can be used to correlate feature with ticket number | optional, multiple allowed |
class level [ScenarioCategory] attribute, NUnit [Category] attribute, MsTest [TestCategory] attribute, XUnit [Trait("Category", "...")] attribute | associates all feature scenarios with a logical group like business epic | optional, multiple allowed, attribute type mixing allowed, equivalent of applying those categories to each test method in class |
Note: The LightBDD feature classes should derive from FeatureFixture type.
Test methods are treated as Scenarios, where:
Code | Meaning | Notes |
---|---|---|
method level [Scenario] attribute | represents method to be scenario | It is equivalent of NUnit [Test], XUnit [Fact]/[Theory], MSTest [TestMethod] attributes, where test runner specific test method attributes are not supported. |
method name | scenario name | preferred method names with underscores |
method level [Label] attribute | can be used to correlate scenario with ticket number | optional, multiple allowed |
method level [ScenarioCategory] attribute, NUnit [Category] attribute, MsTest [TestCategory] attribute, XUnit [Trait("Category", "...")] attribute | associates scenario with a logical group like business epic | optional, multiple allowed, attribute type mixing allowed, scenario would be associated to all categories defined on test method level and class level |
method level [MultiAssert] attribute | can be used to instruct runner to run all its direct sub-steps no matter of their outcome and collect results after all are finished | optional; it overrides the default behavior where scenario execution will stop on first failed/ignored step |
method level [IgnoreScenario] attribute | can be used to instruct runner to mark scenario as ignored without executing it and include it in reports | optional |
As presented above, it is possible to categorize the scenarios by applying [ScenarioCategory]
or underlying test framework specific category attributes on feature class or scenario method.
Usage of those attributes will effect with including such categories in the feature reports.
Please note however that ScenarioCategoryAttribute
is pure LightBDD concept and it is not recognized by the underlying frameworks, which means that it is not possible to filter tests by it when executed from command line - if such behavior is desired, please use instead native attributes:
-
[Category]
for NUnit, -
[TestCategory]
for MsTest, -
[Trait("Category", "...")]
for XUnit (since version 2.3.5).
The test method body is a single call to Runner.RunScenario() with list of methods to execute in specified order, which are treated as Steps, where:
Code | Meaning | Notes |
---|---|---|
method name | step name | preferred method names with underscores |
method level [MultiAssert] attribute | if step is a composite, it can be used to instruct runner to run all its direct sub-steps no matter of their outcome and collect results after all are finished | optional; it overrides the default behavior where composite step execution will stop on first failed/ignored sub-step |
method level [IgnoreScenario] | can be used to instruct runner to mark step as ignored without executing it | optional |
See example below:
[TestFixture]
[FeatureDescription(
@"In order to access personal data
As an user
I want to login into system")] //feature description
[Label("Story-1")]
[ScenarioCategory("Security")]
public partial class Login_feature //feature name
{
[Test]
[Label("Ticket-1")]
public void Successful_login() //scenario name
{
Runner.RunScenario(
Given_the_user_is_about_to_login, //scenario steps
Given_the_user_entered_valid_login,
Given_the_user_entered_valid_password,
When_the_user_clicks_login_button,
Then_the_login_operation_should_be_successful,
Then_a_welcome_message_containing_user_name_should_be_returned);
}
}
Please note that LightBDD supports multiple formats of scenario definitions - for more, please see Scenario Steps Definition.
An important convention in LightBDD is that acceptance tests should be divided into two separate files:
- acceptance tests definition part containing easy to read scenarios,
- implementation part containing scenario test implementation. With it, all tests written with LightBDD would be easy to read even by people who are not experts in writing code and to maintain by developers.
It is suggested to use partial classes and private step methods over other ways of separating test definition and it's implementation, because it simplifies tests maintenance in future. If after requirements change some of steps become not used any more, VisualStudio (or Resharper) will help developers with keeping test code clean, by indicating unused methods.
The naming convention for all names used by LightBDD is that the _
character is replaced with white space - for more details see DefaultNameFormatter class.
The default steps behavior is interpreted as follows:
- if all steps exit without exception, the scenario finishes with passed status,
- if a given step has [IgnoreScenario] attribute or call Assert.Ignore() (NUnit) or Assert.Inconclusive() (NUnit/MsTest) or StepExecution.Current.IgnoreScenario() (xUnit), other steps are not executed and the scenario finishes with ignored status,
- if a given step throws any exception, other steps are not executed and scenario finishes with failed status,
Steps can also finish with bypassed status if StepExecution.Current.Bypass() method is called. Such steps however would not stop scenario execution. The scenario result would be as follows:
- if all others steps are passed/bypassed, the test would be reported as passed in testing framework, but LightBDD would report it as bypassed,
- if any consecutive step would be ignored or failed, the entire scenario would finish in a way described above.
A table below shows steps behavior. A first matching row will correspond to overall scenario status.
Trigger | Step result | Scenario behavior | Test result | Scenario result |
---|---|---|---|---|
Assert.Fail() / exception | Failed | Stop | Failed | Failed |
[IgnoreScenario] / Assert.Ignore() / Assert.Inconclusive() / StepExecution.Current.IgnoreScenario() | Ignored | Stop | Ignored / Skipped / Inconclusive | Ignored |
StepExecution.Current.Bypass() | Bypassed | Continue | Passed | Bypassed |
no exceptions | Passed | Continue | Passed | Passed |
Note: please note that applying underlying test framework [Ignore]
or [Fact(Skip="reason")]
attribute on scenario method will affect the test as being ignored in the test framework and it will not appear in the report as Runner.RunScenario()
would be never be executed.
If [MultiAssert] attribute is applied on a scenario method or composite step method, runner will always execute all direct sub-steps of the scenario/composite irrespectively to the previous sub-step result and their failure status will be reported after all sub-steps are finished. The overall status of parent step/scenario will be set to the most severe status of its sub-steps.
Continue reading: Scenario Steps Definition
- What Is New
- Quick Start
- Tests Structure and Conventions
- Scenario Steps Definition
- Composite Steps Definition
- SetUp and TearDown
- DI Containers
- Step parameters
- Test Progress Notification
- Generating Reports
- Execution Time Measurement
- Feature Fixture
- Step Comments
- Step Attachments
- Extending Test Behavior
- Test Utilities
- Test Framework Integrations
- LightBDD Configuration
- Visual Studio Extensions
- How to