How to support multiple parameterized tests

[ssg47]

Description

I have a legacy system and I'm adding tests prior to some changes. One scenario requires testing an method with ten input parameters, which generate twenty properties which must be tested. There are 54 sets of input parameters to be tested. How would you approach this with your framework?

[Suremaker]

Hello,
I am currently travelling with a limited access to the keyboard and no access to any IDE, so I would like to apologise in advance if given examples won't compile or won't run in a first go, but they should give you an idea of the possibilities with LightBDD.

LightBDD offers various ways to deal with parameterized scenarios, so to choose a best one would depend on the character of the code under test as well as your expectations (and test granularity).

Small analysis of the problem

It is hard for me to guess what is the nature of the method under the test.
There are various things to consider:

• if it is an algorithm that is fast and tests can be executed one by one, or it is slow so it would be better to run test cases in parallel;
• if the output values should be listed during execution (and included in reports) or it would be just a matter of specifying that given case (of 54) passed or not;
• if the test input data and output expectations are available in form of csv/xls file or some in memory structure, or they would be provided differently.

For below samples, I made an assumption that there is some source with inputs/expectations. Also, I will provide few samples to show different options, but all assume that feature methods are not executing in parallel (if that would be a case, I can provide a sample for it).

I hope the samples below will put a light how the problem could be implemented.
If you would have further questions I will be happy to answer them (but I will have a problem to provide any working sample in next 2 weeks due to lack of access to the VisualStudio).
Also, here is a link to wiki page that has the description to all the features I have used below: https://github.com/LightBDD/LightBDD/wiki

Input data

Here are some classes I assumed for samples below:

class Input
{
  public string Param1 {get;set;}
  public int OtherParam {get;set;}
  /* ... */
  // it would be used to render input parameter in steps
  public string ToString()
  {
    return $”Param1={Param1}, OtherParam={OtherParam}, ...”;
  }
}
class Output
{
  public string Output1 {get;set;}
  public string Output2 {get;set;}
  /* ... */
}
class Case
{
  public Input Input {get;set;}
  public Output Expectations {get;set;}
  // it would print only input details
  public ToString() { return Input.ToString(); }
}
class Generator
{
  public Output Generate (Input input);
}

Example 1: Scenario per case

This would be an example assuming that:

• generator is fast, so all cases can be tested serially,
• expected output values does not have to be displayed in the report (just success or failure),
• tests are written in LightBDD.Nunit3

It should produce a feature with 54 scenarios having Given_, When_ and Then_ step each.

using LightBDD.Framework;
using LightBDD.Framework.Scenarios.Extended;
using LightBDD.NUnit3;
public class Generating_output_feature: FeatureFixture
{
   //not thread safe fields
  Generator _generator;
  Case _case;
  Output _output;
  [Scenario]
  [TestCaseSource(“GetCases”)]
  // Scenario will have all input values listed
  public void Generator_should_produce_valid_output_for_case(Case @case)
  {
    _case = @case;
    Runner.RunScenario(
      _ => Given_a_generator(),
      _ => When_I_call_generator_with_input_values(),
      _ => Then_it_should_provide_valid_output_values());
  }
  void Given_a_generator() { _generator = new Generator(); }
  void When_I_call_generator_with_input_values()
  {
    _output = _generator.Generate(_case.Input);
  }
  void Then_it_should_provide_valid_output_values()
  {
        Assert.That(_output.Output1, Is.EqualTo(_case.Expectations.Output1));
        Assert.That(_output.Output2, Is.EqualTo(_case.Expectations.Output2));
        /* ... */
  }
  Case[] GetCases()
  {
    //return cases
  }
}

Example 2: Compact scenario

This would be an example assuming that:

• generator is fast, so all cases can be tested serially,
• expected output values does not have to be displayed in the report (just success or failure),
• tests are written in LightBDD.Nunit3

It should produce a feature with 1 scenario having Given_ step and 54 Then_ steps

using LightBDD.Framework;
using LightBDD.Framework.Scenarios.Extended;
using LightBDD.Framework.Scenarios.Fluent;
using LightBDD.NUnit3;
public class Generating_output_feature: FeatureFixture
{
  Generator _generator; //not thread safe
  [Scenario]
  [MultiAssert] //it would iterate through all steps allowing them to pass or fail
  public async Task Generator_should_produce_valid_output()
  {
    var scenarioBuilder = Runner
      .NewScenario()
      .AddSteps(_ => Given_a_generator());

    foreach (var @case in GetCases())
      scenarioBuilder.AddSteps(_ => Then_generator_should_provide_valid_result_for_case(@case));
  
    await scenarioBuilder.RunAsync();
  }
  void Given_a_generator() { _generator = new Generator(); }
  void Then_generator_should_provide_valid_result_for_case(Case @case)
  {
    var result = _generator.Generate(@case.Input);
    Assert.That(result.Output1, Is.EqualTo(@case.Expectations.Output1));
    Assert.That(result.Output2, Is.EqualTo(@case.Expectations.Output2));
    /* ... */
  }
  Case[] GetCases()
  {
    //return cases
  }
}

Example 3: Add explicit assertions for output values

This example would base on example 1, but will use composite step instead of Then_step with multiple asserts. The composite step would assert each property separately so it would show what has exactly failed

using LightBDD.Framework;
using LightBDD.Framework.Scenarios.Extended;
using LightBDD.NUnit3;
public class Generating_output_feature: FeatureFixture
{
   //not thread safe fields
  Generator _generator;
  Case _case;
  Output _output;
  [Scenario]
  [TestCaseSource(“GetCases”)]
  // Scenario will have all input values listed
  public void Generator_should_produce_valid_output_for_case(Case @case)
  {
    _case = @case;
    Runner.RunScenario(
      _ => Given_a_generator(),
      _ => When_I_call_generator_with_input_values(),
      _ => Then_it_should_provide_valid_output_values());
  }
  void Given_a_generator() { _generator = new Generator(); }
  void When_I_call_generator_with_input_values()
  {
    _output = _generator.Generate(_case.Input);
  }
  [MultiAssert] // to show all failures
  CompositeStep Then_it_should_provide_valid_output_values()
  {
    return CompositeStep.DefineNew().AddSteps(
      _ => Then_Output1_property_should_have_value(_case.Expectations.Output1),
      _ => Then_Output2_property_should_have_value(_case.Expectations.Output2),
    ).Build();
   }
  void Then_Output1_property_should_have_value(string value)     
  {
        Assert.That(_output.Output1, Is.EqualTo(value));
  }
  void Then_Output1_property_should_have_value(string value)     
  {
        Assert.That(_output.Output2, Is.EqualTo(value));
  }
  Case[] GetCases()
  {
    //return cases
  }
}

Outcome

After writing and running those scenarios, please take a look at the test output bin folder. It will contain report such as this one, showing the executed scenarios: http://htmlpreview.github.io/?https://github.com/LightBDD/LightBDD/blob/master/examples/ExampleReports/FeaturesReport.html

[ssg47]

Thanks for a comprehensive response. Example 3 looks like what I need.