LivingDoc not working after Reqnroll Version 3.0.0

[Moshex]

Reqnroll Version

3.3.2

Which test runner are you using?

NUnit

Test Runner Version Number

4.4.0

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
  "bindingAssemblies": [
  ],
  "livingDocExecutionDataCollector": {
    "enabled": true,
    "filepath": "TestExecution.json"
  }
}

Issue Description

Issue with iReqnrollOutputHelper.WriteLine and AddAttachment in Reqnroll Version 3.0.0 and Higher

Subject: iReqnrollOutputHelper.WriteLine and AddAttachment Not Working in LivingDoc After Reqnroll Version 3.0.0+

Description of the Issue:

After upgrading to Reqnroll version 3.0.0 or higher, the following methods:

iReqnrollOutputHelper.WriteLine(SpecHelperText);  
iReqnrollOutputHelper.AddAttachment("attachment");  

no longer work when generating a LivingDoc. Additionally, the "Show Test Output" button is missing from the LivingDoc generated by Reqnroll.

This issue seems to be caused by the changes introduced in version 3.0.0, where new reporting formats and methods were introduced, as documented here: Reqnroll Formatters Documentation.

Impact:

The LivingDoc format is highly valuable to me due to its simplicity and the way it presents test results.

The new reporting introduced in version 3.0.0, while powerful, does not fit my use case as well as LivingDoc does. Losing the "Show Test Output" button and the ability to use the above methods diminishes the effectiveness of reporting for my workflow.

Request:

1. Fix or Compatibility: Is there a way to restore compatibility for iReqnrollOutputHelper.WriteLine and AddAttachment methods so they work with LivingDoc in version 3.0.0 and higher?
2. LivingDoc Support: Can you confirm whether LivingDoc is still supported in the newer versions of Reqnroll, or if there is a plan to continue supporting it alongside the new reporting format?
3. Package Information: If the issue is related to the databinding.LivingDoc.ExecutionDataCollector NuGet package, could you provide information about the maintainer or author of this package so I can raise the issue with them directly?

Additional Notes:

I am aware of the new reporting features introduced in version 3.0.0 (Reqnroll Formatters Documentation), but I still prefer the LivingDoc format for my reports. The ability to use WriteLine and AddAttachment methods with LivingDoc was a crucial part of my workflow.

Please let me know if there is a way to make these methods work again with LivingDoc, or if you can provide information about who maintains the databinding.LivingDoc.ExecutionDataCollector package so I can escalate the issue to them.

Environment:

Reqnroll Version: 3.0.0 and higher
LivingDoc Version: Latest version
NuGet Package: databinding.LivingDoc.ExecutionDataCollector
Test Framework: SpecFlow / NUnit (if applicable)
Platform: Windows/MacOS/Linux (specify)

Steps to Reproduce

Steps to Reproduce:

1. Upgrade to Reqnroll version 3.0.0 or higher.
2. Use the following code snippet in a test:

iReqnrollOutputHelper.WriteLine("Test output");  
iReqnrollOutputHelper.AddAttachment("attachment_path");  

3. Generate a LivingDoc report.
4. Observe that the "Show Test Output" button is missing, and the above methods do not work.

Link to Repro Project

No response

[304NotModified]

LivingDoc was never supported by Reqnroll? It was a SpecFlow/Tricentis product and the intellectual property (IP) and (closed) source code is at Tricentis.

[304NotModified]

The extension is also unpublished, so it's really unsupported?

https://marketplace.visualstudio.com/items?itemName=techtalk.techtalk-specflow-plus

[304NotModified]

See also
https://github.com/orgs/reqnroll/discussions/15

No documented API, the extension is not maintained (and maybe not installable).

IMO we can't support this and we should focus on getting the new/open source stuff better.

[Moshex]

Really my question is why
iReqnrollOutputHelper.WriteLine(SpecHelperText);
iReqnrollOutputHelper.AddAttachment("attachment");
stopped working for LivingDoc in Reqroll version 3.0.0 and up, and if there is a way to work around that

[304NotModified]

Your comment isn't helping in the discussion if it's supported or not.

[304NotModified]

... databinding.LivingDoc.ExecutionDataCollector NuGet package, could you provide information about the maintainer or author of this package so I can raise the issue with them directly?

Maybe contact owners at https://www.nuget.org/packages/databinding.LivingDoc.ExecutionDataCollector/

[304NotModified]

Related

https://github.com/orgs/reqnroll/discussions/517

[Moshex]

I did see that URL, but I am unable to find a GitHub repo to submit an issue to them. Do you have more information on that?

[304NotModified]

There is a "contact owners" link on the nuget page.

[Moshex]

I have tried that and am still waiting for a response

[304NotModified]

I've moved this to a discussion as there multiple discussion topics in the starting post.

[304NotModified]

Could you please elaborate on this?

The LivingDoc format is highly valuable to me due to its simplicity and the way it presents test results.

[Moshex]

bdd-report.html
LivingDoc.html
Well, you take a look at the 2 reports and tell me which one looks better?
From my end, it's clearly the LivingDoc if you can get your report to look and act closer to the LivingDoc, I would have no issue with switching to the new report

[304NotModified]

Well, you take a look at the 2 reports and tell me which one looks better?

These are hard requirements "tell me which one looks better"...

This won't motivate me in helping with this issue.

[Moshex]

Sorry, I did not mean to be harsh in any way, and I really appreciate all the hard work you are doing. What I am trying to say is that at this current moment, per the UI and overall way the LivingDoc operates, it serves the needs of my team much better than the new report. If you are looking for requirements, I would say that in my company, if we are doing a rewrite of an existing product that we know our users love, then the requirements are the existing product. if you made your report look and function close to the current LivingDoc, that would be ideal. I have attached a fuller report for your reference if you need. Thank you for all the hard work you do. I could not do what I do without you!
LivingDoc.html

[304NotModified]

if we are doing a rewrite of an existing product

First of all, we didn't rewrite livingdoc ;)

I would be more helpful you state what you are missing. Do everything like tool X don't show any effort.

[gasparnagy]

@Moshex Have you tried the Expressium.LivingDoc plugin? That is closer to the original LivingDoc. See https://docs.reqnroll.net/latest/integrations/available-plugins.html#filter=formatter

[gasparnagy]

Also make sense to note the custom formatter option: https://docs.reqnroll.net/latest/reporting/reqnroll-formatters.html#creating-a-custom-formatter

[Moshex]

Thank you, @gasparnagy, for this information. with the "Expressium.LivingDoc" I am seeing 2 packages and the only one that seems to work is "Expressium.LivingDoc.ReqnrollPlugin". I have attached the report that gets generated. I like the report, and with more testing, I feel like this can serve our needs; however, on this report, I am still not able to get IReqnrollOutputHelper.WriteLine("SpecHelperText"); to display on the report, which is the whole issue I am having on the other report. I may have some configuration off since this is new to me. I will add my "reqnroll.json" code to see if I have things right, and if you could advise on what I need to do to get that working, that would be nice. Keep in mind that the IReqnrollOutputHelper.WriteLine("SpecHelperText"); code is getting hit within Debug and has worked on the LivingDoc for years. Thank you for all your help!

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
  "formatters": {
    "expressium": {
      "output": "LivingDoc.html",
      "FilePath": "LivingDoc.ndjson",
      "title": "WellRx.UITests",
      "testFramework": "NUnit"
    }
  }
}

reqnroll_report.html

[clrudolphi]

@Moshex Expressium is a separate project not run by the Reqnroll project team. Please contact them directly with questions on how to use and configure that project.

[MrSoerenAndersen]

Issue has been replied on Expression LivingDoc...
ExpressiumOSS/Expressium.LivingDoc#13

[Moshex]

Workaround: Restoring step-level "Show Test Output" in LivingDoc after Reqnroll migration

After migrating from SpecFlow to Reqnroll, the livingdoc CLI (v3.9.57) strips step-level Outputs from TestExecution.json when embedding it into the HTML. The React UI checks stepResults.some(s => s.outputs.length > 0) to decide whether to show the "Show Test Output" button — so it never appears.
Additionally, IReqnrollOutputHelper.WriteLine() called from [AfterStep] hooks writes to scenario-level outputs in TestExecution.json, not step-level. This means even if step outputs weren't stripped, the data isn't there at step level to begin with.

Our workaround has three parts:

Part 1: Accumulate & replay outputs at scenario level

In [AfterStep], we write error details via IReqnrollOutputHelper.WriteLine() and simultaneously accumulate copies into a ScenarioContext list. Then in [AfterScenario], we replay all accumulated lines. This populates the scenario-level Outputs in TestExecution.json — the only field the livingdoc CLI preserves.

private const string ScenarioOutputKey = "LivingDocScenarioOutput";

private static void AccumulateScenarioOutput(string text)
{
    if (GlobalContext.scenarioContext == null) return;
    if (!GlobalContext.scenarioContext.TryGetValue<List<string>>(ScenarioOutputKey, out var outputs))
    {
        outputs = new List<string>();
        GlobalContext.scenarioContext[ScenarioOutputKey] = outputs;
    }
    outputs.Add(text);
}

[AfterStep]
public static void AfterStep()
{
    // ... (only runs when TestError != null) ...
    GlobalContext.iReqnrollOutputHelper.WriteLine(errorText);
    AccumulateScenarioOutput(errorText);        // save a copy for replay
    GlobalContext.iReqnrollOutputHelper.WriteLine(screenshotUrl);
    AccumulateScenarioOutput(screenshotUrl);    // save a copy for replay
}

[AfterScenario]
public void AfterEachScenario()
{
    // Replay all accumulated output so the Reqnroll data collector
    // writes it into the scenario-level "Outputs" field in TestExecution.json.
    try
    {
        if (GlobalContext.scenarioContext != null
            && GlobalContext.scenarioContext.TryGetValue<List<string>>(ScenarioOutputKey, out var outputs)
            && outputs?.Count > 0)
        {
            foreach (var line in outputs)
            {
                GlobalContext.iReqnrollOutputHelper.WriteLine(line);
            }
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error writing scenario-level output: {ex.Message}");
    }
}

Why [AfterScenario] and not [AfterStep]? The replay must run exactly once after all steps complete. [AfterStep] fires after every step — including skipped steps after a failure (where TestError remains non-null) — so replaying there would cause duplicates.

Part 2: Post-process the LivingDoc HTML

After generating LivingDoc.html, we parse it, find the embedded window.featureData = {…}; JSON, match each scenario to TestExecution.json by a composite key, and move the scenario-level outputs onto the failed step:

private static void InjectStepOutputsIntoLivingDoc(string livingDocPath, string testExecutionJsonPath)
{
    var html = File.ReadAllText(livingDocPath);
    var teDoc = JsonDocument.Parse(File.ReadAllText(testExecutionJsonPath));
    var teResults = teDoc.RootElement.GetProperty("ExecutionResults");

    // Build lookup storing both StepResults and scenario-level Outputs per scenario
    var lookup = new Dictionary<string, List<(JsonElement StepResults, JsonElement? ScenarioOutputs)>>();
    foreach (var er in teResults.EnumerateArray())
    {
        var key = $"{er.GetProperty("FeatureTitle").GetString()}" +
                  $"|{er.GetProperty("ScenarioTitle").GetString()}" +
                  $"|{er.GetProperty("ThreadId").GetString()}" +
                  $"|{er.GetProperty("Status").GetString()}";

        if (!lookup.ContainsKey(key))
            lookup[key] = new List<(JsonElement, JsonElement?)>();

        JsonElement? scenarioOutputs = null;
        if (er.TryGetProperty("Outputs", out var outProp) &&
            outProp.ValueKind == JsonValueKind.Array && outProp.GetArrayLength() > 0)
            scenarioOutputs = outProp;

        lookup[key].Add((er.GetProperty("StepResults"), scenarioOutputs));
    }

    // Locate "window.featureData = {…};" in HTML, parse, patch, serialize back
    const string marker = "window.featureData = ";
    var startIdx = html.IndexOf(marker, StringComparison.Ordinal) + marker.Length;
    var endIdx = html.IndexOf(";\n", startIdx, StringComparison.Ordinal);
    var featureData = JsonNode.Parse(html.Substring(startIdx, endIdx - startIdx));

    int patchedCount = 0;
    PatchExecutionResults(featureData, lookup, ref patchedCount);

    html = html.Substring(0, startIdx) + featureData.ToJsonString() + html.Substring(endIdx);
    File.WriteAllText(livingDocPath, html);
}

Part 3: The patching logic — move scenario outputs to the failed step

The recursive walker tries step-level outputs first. When those are empty (the common case with Reqnroll), it falls back to injecting the scenario-level outputs onto the last failed step, then clears the scenario-level outputs so LivingDoc doesn't show duplicates:

private static void PatchExecutionResults(JsonNode? node,
    Dictionary<string, List<(JsonElement StepResults, JsonElement? ScenarioOutputs)>> lookup,
    ref int patchedCount)
{
    if (node is not JsonObject obj) { /* recurse into arrays */ return; }

    if (obj.ContainsKey("ContextType") && obj.ContainsKey("StepResults") &&
        obj.ContainsKey("FeatureTitle") && obj.ContainsKey("ScenarioTitle"))
    {
        var key = $"{obj["FeatureTitle"]?.GetValue<string>()}" +
                  $"|{obj["ScenarioTitle"]?.GetValue<string>()}" +
                  $"|{obj["ThreadId"]?.GetValue<string>()}" +
                  $"|{obj["Status"]?.GetValue<string>()}";

        if (lookup.TryGetValue(key, out var entryList) && entryList.Count > 0)
        {
            var (teSteps, scenarioOutputs) = entryList[0];
            var embeddedSteps = obj["StepResults"]?.AsArray();
            bool anyStepPatched = false;

            // Try step-level outputs first
            for (int i = 0; i < embeddedSteps.Count && i < teSteps.GetArrayLength(); i++)
            {
                var teStep = teSteps[i];
                if (teStep.TryGetProperty("Outputs", out var outputsProp) &&
                    outputsProp.ValueKind == JsonValueKind.Array && outputsProp.GetArrayLength() > 0)
                {
                    embeddedSteps[i]!.AsObject()["Outputs"] =
                        JsonNode.Parse(outputsProp.GetRawText());
                    anyStepPatched = true;
                }
            }

            // Fallback: move scenario-level outputs onto the last failed step
            if (!anyStepPatched && scenarioOutputs.HasValue)
            {
                for (int i = embeddedSteps.Count - 1; i >= 0; i--)
                {
                    var status = embeddedSteps[i]?["Status"]?.GetValue<string>();
                    if (status != null &&
                        !status.Equals("OK", StringComparison.OrdinalIgnoreCase) &&
                        !status.Equals("Skipped", StringComparison.OrdinalIgnoreCase))
                    {
                        embeddedSteps[i]!.AsObject()["Outputs"] =
                            JsonNode.Parse(scenarioOutputs.Value.GetRawText());
                        anyStepPatched = true;
                        break;
                    }
                }
            }

            if (anyStepPatched)
            {
                patchedCount++;
                // Clear scenario-level so UI doesn't show duplicate at scenario level
                if (obj.ContainsKey("Outputs"))
                    obj["Outputs"] = new JsonArray();
            }

            if (entryList.Count > 1) entryList.RemoveAt(0);
        }
    }

    // Recurse into all children
    foreach (var kvp in obj.ToList())
        PatchExecutionResults(kvp.Value, lookup, ref patchedCount);
}

Result:
The "Show Test Output" button appears under the specific failed step with the full error message, screenshot link, and triage summary — matching the behavior from the old SpecFlow LivingDoc