Make report templates resilient to typed-model null defaults#1644
Merged
mvdbeek merged 2 commits intoMay 22, 2026
Merged
Conversation
`report_markdown.tpl` iterated `test.data.output_problems` outside its
guarding `{% if %}`, relying on Jinja's silent-empty behaviour for
Undefined attribute lookups when the field was absent from the dict.
After `a0d765fd57` ("Add structured CLI and output schemas") introduced
typed Pydantic models, `merge_test_reports` round-trips reports through
`PlanemoTestReport.model_validate(...).model_dump(mode="json")`. The
default `mode="json"` dump serialises unset Optional fields as explicit
`null`, so `output_problems` is now `None` in the merged JSON instead of
absent. Jinja then raises `TypeError: 'NoneType' object is not iterable`
when rendering any test that didn't produce output problems (including
all passing tests).
Move the `{% for %}` inside the existing `{% if %}` so the loop only
runs when there are problems to render, and add a regression test that
generates a markdown report from a passing test with explicit
`output_problems: null`.
`report_junit.tpl` and `report_xunit.tpl` gated on
`{% if 'job' in testcase.data %}` to decide whether to emit a
per-job <failure>/<error> (with exit code + job dump) or fall back to
a per-test <failure>/<error> message built from `execution_problem`.
After `a0d765fd57` introduced typed Pydantic models, `merge_test_reports`
round-trips reports through `model_dump(mode="json")`, which serialises
unset Optional fields as explicit `null`. The `job` key is therefore
always present in `testcase.data` -- as `None` for tests that errored
before a job was created -- so the guard always succeeds and the
execution_problem fallback branch became unreachable. The resulting
JUnit/xUnit reports for such failures show "Tool exit code: " with no
exit code and no message, hiding the real failure reason from CI
dashboards.
Replace `'job' in testcase.data` with `testcase.data.job`, which is
truthy on a populated dict and falsy on None, restoring the intended
two-branch behaviour for both pre- and post-round-trip JSON shapes.
Collaborator
bernt-matthias
left a comment
There was a problem hiding this comment.
Looks good to me, but I do not understand how a0d765f can be the source of the problem. The commit is not yet in a release, or?
Collaborator
I see. There is "only" a tag, but no release. |
bernt-matthias
approved these changes
May 22, 2026
Collaborator
bernt-matthias
left a comment
There was a problem hiding this comment.
Thanks for the explanation
Member
|
Ouch - sorry for breaking this and thank you for the tests! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two report templates in
planemo/reports/made assumptions about the JSON test-report shape that no longer hold aftera0d765fd57("Add structured CLI and output schemas") introduced typed Pydantic models.merge_test_reportsnow round-trips reports throughPlanemoTestReport.model_validate(...).model_dump(mode=\"json\"), andmode=\"json\"serialises unset Optional fields as explicitnullinstead of omitting them. Templates that previously relied on Jinja's silent-empty behaviour for Undefined attribute lookups break when they receive real None values.This PR fixes two distinct user-visible symptoms of that shape mismatch:
report_markdown.tplcrashes rendering any report that contains a passing test.The
{% for problem in test.data.output_problems %}loop sat outside its{% if test.data.output_problems %}guard.output_problemsisOptional[list[str]] = None, so post-round-trip it isnullfor every test that didn't produce output problems (including all passing tests), and Jinja raisesTypeError: 'NoneType' object is not iterable. Symptom in the wild:planemo test_reportsafterplanemo merge_test_reportscrashes for any successful workflow test (e.g. https://github.com/galaxyproject/iwc/actions/runs/26224703416/job/77168877129). Fix: move theforinside the existingif.report_junit.tpl/report_xunit.tplsilently hide the real failure reason for tests that errored before a job was created.Both templates gated on
{% if 'job' in testcase.data %}to choose between a per-job<failure>(with exit code + job dump) and a per-test fallback<failure message=\"{{ testcase.data.execution_problem }}\">. Post-round-trip thejobkey is always present (asNonefor execution-failed tests), so the guard is always true and the fallback branch is unreachable. The resulting JUnit/xUnit reports for such failures showTool exit code:with no exit code and no message instead of the actualexecution_problem— silently broken in CI dashboards (Jenkins, GitHub Actions, etc.). Fix: replace'job' in testcase.datawithtestcase.data.job, which is truthy on a populated dict and falsy on None — correct for both pre- and post-round-trip JSON shapes.Both fixes target the templates rather than
actions.py:70/82'smodel_dump(mode=\"json\")call, because changing the dump options (e.g.exclude_none=True) would alter the on-disk JSON shape for every consumer of planemo's report files.Audit of remaining templates
For completeness I also audited the other templates under
planemo/reports/:report_text.tpl{% for %}properly nested inside{% if test.data.output_problems %}.report_markdown_minimal.tplraw_data.testsand a literal dict.xunit.tplcmd_shed_diff/cmd_shed_update.macros.tmplrender_steps(steps),render_invocation_messages(messages),render_invocation_details(details)iterate args without None-guards. Externally gated by{% if test.data.invocation_details %}, and the inner.steps/.messages/.detailscome from Galaxy's invocation API (typed elsewhere). No reproduction in the wild — leaving alone.Test plan
pytest tests/test_cmd_test_reports.py— 8 pass (6 pre-existing + 2 new regression tests).git stashof the template change → test fails with the exact symptom described above).