Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rework @test_reference messages - drop lena #118

Merged
merged 6 commits into from
Dec 28, 2023

Conversation

t-bltg
Copy link
Contributor

@t-bltg t-bltg commented Aug 10, 2022

  • rework @info in @test_reference: now includes render(...) (more structured output for the end user, see below, works well with images), this helps CI in ensuring correct message order (io flush);
  • rework tests not to show spurious messages when working with @test_reference macro unless CI env var is set to true: this was is very confusing when running ] test ReferenceTests: only the tests summary should be shown to the end user, detailed are activated in CI or by the env var CI;
  • drop lena in runtests.jl: replaced with monarch which is very similar (same format and size);
  • stop writing / deleting temporary test files in test/references: if testing is interrupted, these temporaries are leftovers, and thus can land in a commit (use mktempdir instead).

As an example, this job output (PR) is more readable than this one (master): most likely flush of stdout and stderr is de-synchronized.

PR

julia> ENV["VERBOSE"] = true  # default

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
┌ Info: Reference Test for "string1.txt" failed:- DIFF ------------------------"fintentionally wrobng to check that this message prints"-------------------------------
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_h48D23/string1.txt"
┌ Info: Reference Test for "string5.txt" failed:- DIFF ------------------------"""
│ - This is a
│ + This is an incorrect
│   multiline string that does not end with a new line."""-------------------------------
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_79MGXW/string5.txt"
[...]
     Testing ReferenceTests tests passed

julia> ENV["VERBOSE"] = false

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
Test Summary:  | Pass  Total
ReferenceTests |   49     49
     Testing ReferenceTests tests passed 

master

pkg> test ReferenceTests
[...]
┌ Info: Reference Test for "string1.txt" failed.
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_egX948/string1.txt"
- DIFF ------------------------
"fintentionally wrobng to check that this message prints"
-------------------------------
┌ Info: Reference Test for "string5.txt" failed.
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_qicFon/string5.txt"
- DIFF ------------------------
"""
- This is a
+ This is an incorrect
  multiline string that does not end with a new line."""
-------------------------------
[...]
     Testing ReferenceTests tests passed 

@t-bltg t-bltg mentioned this pull request Aug 10, 2022
@t-bltg t-bltg changed the title rework log messages (only ci) - drop lena rework @test_reference messages - test messages (only ci) - drop lena Aug 10, 2022
@t-bltg t-bltg force-pushed the ci_lena branch 2 times, most recently from c63dc5b to a39c7e9 Compare August 11, 2022 11:07
@t-bltg t-bltg changed the title rework @test_reference messages - test messages (only ci) - drop lena rework @test_reference messages - test msgs only on ci - drop lena Aug 11, 2022
test/runtests.jl Outdated Show resolved Hide resolved
@t-bltg
Copy link
Contributor Author

t-bltg commented Dec 22, 2023

@oxinabox, I have some issues with nightly CI in UnicodePlots (see e.g. corrupted paths with ANSI codes in https://github.com/JuliaPlots/UnicodePlots.jl/actions/runs/7300585284/job/19895531268#step:6:384).

This PR makes the output from ReferenceTests clearer for debugging.

Can we please revive and merge this PR and bump a new release (maybe @johnnychen94) ?

@t-bltg t-bltg changed the title rework @test_reference messages - test msgs only on ci - drop lena rework @test_reference messages - drop lena Dec 22, 2023
@t-bltg
Copy link
Contributor Author

t-bltg commented Dec 22, 2023

CI is green for all OS.

@oxinabox oxinabox merged commit 895e4be into JuliaTesting:master Dec 28, 2023
7 of 8 checks passed
@t-bltg t-bltg deleted the ci_lena branch December 28, 2023 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants