You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@nrc You mentioned "The long-term solution is going to be deeply intertwined with pluggable test runners" in #46450. Does that mean that JSON won't be stabilized until there are pluggable test runners?
Yeah I think we should stabilize something like this independently of custom test frameworks. It makes sense that the included test framework would have a programmatically readable output. Libtest's conceptual model is pretty simple and most likely won't undergo any drastic changes so I'm all for stabilizing something. I'd prefer to have a JSON Schema that describes the structure before stabilization. We'd also need to audit to ensure it's resilient to any minor changes or additions over time.
From the experience of creating naiive serde structs / enums for libtest's output, here are my thoughts from translating it exactly how it is written in libtest (rather than inventing my own schema that happens to be compatible):
For suites / tests, the data is two enums nested. For some cases, this might work well, for others, it is unnecessary nesting to deal with.
Probably the most annoying aspect is that the type field conflates suite/test being finished and why it finished. This is annoying for if you want to get information regardless of the completion status/.
No bench start even is sent
My gut is telling me that the bench event is too narrowly focused on the output of the current implementation and not on what might be wanted
Combine event and type fields. Unsure of the value of this
Split type field into type and status and define more fields as being available for all types. For example, it could be useful to programmatically report stdout for a successful test and leave it to the rendering engine to decide whether to ignore it or not. This doesn't mean all test implementations need to report all of the fields, but define it as possible.
Get someone from other bench implementations, like criterion, to review the bench schema.
Since libtest tracks test times (at minimum, reporting if its taking too long), it would be nice if that got included in the programmatic output so we can report it to the user, e.g. in JUnit. Alternatively, we'd have to timestamp the events as we read them.
If I may add a semi-random thing; https://github.com/testing-cabal/subunit is a testing protocol thats been around for a while (one of a number) - the V2 even model is very similar in nature to your proposed JSON output schema. There is a rust implementation of the binary serialisation https://github.com/mtreinish/subunit-rust and I'm interested / planning on doing a JSON variant of V2 at some point for use with web stacks.
All of which to say two things I guess - consider thinking about chunking stdout/err, allowing additional streams, tags, and possibly, if its interesting, consider adopting the same model - I think it would be a super set of libtests current needs.
Of course, doing a new schema and having folk write adapters is complete fine too; I just saw this ticket and thought I'd offer this kibbitz :)
I would also like duration output as part of the output. Furthermore when running multiple suites at the same time there's no way of knowing what test belong to what suite. It would be nice to name the suite somehow. Possibly connect it to the source file relative to the workspace root?
Are people happy with cargo test -- -Zunstable-options --format json ? Should we be looking to stabilse this argument? (I notice rustc calls the same argument --message-format)
In addition to what ehuss said, I think a good test bed for json output will be rust-lang/cargo#5609.
While its not my highest priority (finishing lints rfc, cargo-script rfc, and msrv are), I am also starting to put some attention to the problems of testing within Rust. Overall, I see this effort as (1) applying the lessons from all the different custom test harness and (2) finding the appropriate places to stablize cargo-nextest behavior. I am first experimenting with an alternative design to libtest itself that I'm hoping will offer the benefits we had hoped for from rust-lang/rfcs#2318. My hope is that both the libtest and cargo-test sides of this work (including rust-lang/cargo#5609) will lead to a solid json format.
So, I have some recent experience using this in #108659. The goal of that PR was to extract stats about the individual tests executed by bootstrap (name of the test, outcome, eventual ignore reason), and there were a few issues I noticed with the current way the JSON output works.
The minor issue is just some unclarity in the JSON format, that could likely be shuffled a bit to be more coherent (for example there is both a reason and a message when a test is ignored).
The major issue I found is that a tool wanting to extract the JSON information for further processing and wanting to show the familiar CLI UI to users needs to reimplement the whole rendering code, as there is no rendered/rendered-ansi option like rustc. This resulted in me having to write a full renderer in bootstrap just to know which tests were executed. The design of the flag will likely not be trivial, as a rendered field would also need to distinguish between the "terse" (used by bootstrap) and "verbose" (used by default by cargo) outputs.
I'm not sure I do, but the overall feeling I get from this thread is that it is at least not optimal
One flaw I see inside the current implementation is Doc Tests cargo +nightly test -- -Z unstable-options --list --format json
For Doc Tests it will print out Doc-tests heck
Then print out the doc tests in JSON format.
This string is weird if we are trying to provide a machine readable format.
Then I noticed that start line and end lines are not provided in Doc Tests.
So start lines should be in the actual JSON. And the Doc-tests message should remove and maybe a test type attribute should be added to the format. Or when the event "discovery" is printed it should have an attribute indicating the start of Doc Tests.
It would be easier to evolve the format it was versioned. This way we could also ship a suboptimal version earlier and always upgrade it with breaking changes to a more optimal version later (while keeping the old version working. It does of course still require a design for the first stable version that we can be sure won't break in the future, but doesn't require a design that's perfect and fits all use cases).
FYI the testing devex team was recently formed and we've decided that the first order of business is working to stabilize the json output. We will be writing up our plan soon for how we'll go about that effort.
Reacted by nora, Remo Senekowitsch, Weihang Lo, Daniel del Castillo, Joe Taber, Luc Georges, Squirrel, Alexandre Fiotti and 许杰友 Jieyou Xu (Joe)
Reacted by Joshua Sleeper, Abraham Egnor, Remo Senekowitsch, Squirrel and Alexandre Fiotti
Reacted by Joshua Sleeper, Abraham Egnor, Remo Senekowitsch, Daniel del Castillo, Squirrel and unrenormalizable
For CI we use the json for coverage, but humans like to look at the pretty output so we run two jobs one running the tests with the default format, one with the json format. It would be great to be able to have the json format being saved to a specified file while at the same time std out produces nice human format.
Reacted by Edvin Malinovskis, Johann Felix Soden, Bennet Bleßmann, Joshua Sleeper, Alexandre Fiotti, Donough Liu and duncanawoods
In my plan for things, the responsibility for that would be moved up into cargo test. Cargo will exclusively accept json output from libtest and then do all of the rendering. This means users will need to ask Cargo for the json output as well.
Activity
johnterickson commentedon Feb 21, 2019
@nrc You mentioned "The long-term solution is going to be deeply intertwined with pluggable test runners" in #46450. Does that mean that JSON won't be stabilized until there are pluggable test runners?
I'm trying to figure out if it makes sense for me to continue looking at #51924 right now, or if I should hold off for the time being. For example, I'm looking at adding test durations to the JSON (master...johnterickson:testjson) and converting the output to JUnit (https://github.com/johnterickson/cargo2junit/).
djrenren commentedon Feb 26, 2019
Yeah I think we should stabilize something like this independently of custom test frameworks. It makes sense that the included test framework would have a programmatically readable output. Libtest's conceptual model is pretty simple and most likely won't undergo any drastic changes so I'm all for stabilizing something. I'd prefer to have a JSON Schema that describes the structure before stabilization. We'd also need to audit to ensure it's resilient to any minor changes or additions over time.
@alexcrichton thoughts?
alexcrichton commentedon Feb 26, 2019
Seems like a plausible addition to me! So long as it's designed careful I think it'd be good to go
epage commentedon Feb 27, 2019
From the experience of creating naiive serde structs / enums for libtest's output, here are my thoughts from translating it exactly how it is written in libtest (rather than inventing my own schema that happens to be compatible):
typefield conflates suite/test being finished and why it finished. This is annoying for if you want to get information regardless of the completion status/.benchevent is too narrowly focused on the output of the current implementation and not on what might be wantedMy PR where you can see the data structure's I created: https://github.com/crate-ci/escargot/pull/24/files
Ideas to consider
eventandtypefields. Unsure of the value of thistypefield intotypeandstatusand define more fields as being available for all types. For example, it could be useful to programmatically reportstdoutfor a successful test and leave it to the rendering engine to decide whether to ignore it or not. This doesn't mean all test implementations need to report all of the fields, but define it as possible.benchimplementations, like criterion, to review thebenchschema.andoriyu commentedon Mar 3, 2019
I think it's worth adding things like package and type of test. Type of test: unit test, doc test, integration test.
The package is something like: module for unit test, resource path for doc test (crate::foo::Bar), and filename for integration test.
epage commentedon Mar 4, 2019
Since libtest tracks test times (at minimum, reporting if its taking too long), it would be nice if that got included in the programmatic output so we can report it to the user, e.g. in JUnit. Alternatively, we'd have to timestamp the events as we read them.
johnterickson commentedon Mar 6, 2019
@epage / @andoriyu Are one of you planning on proposing a better schema? I like what you're proposing.
rbtcollins commentedon Apr 24, 2019
If I may add a semi-random thing; https://github.com/testing-cabal/subunit is a testing protocol thats been around for a while (one of a number) - the V2 even model is very similar in nature to your proposed JSON output schema. There is a rust implementation of the binary serialisation https://github.com/mtreinish/subunit-rust and I'm interested / planning on doing a JSON variant of V2 at some point for use with web stacks.
All of which to say two things I guess - consider thinking about chunking stdout/err, allowing additional streams, tags, and possibly, if its interesting, consider adopting the same model - I think it would be a super set of libtests current needs.
Of course, doing a new schema and having folk write adapters is complete fine too; I just saw this ticket and thought I'd offer this kibbitz :)
Alxandr commentedon Jul 26, 2019
I'd also love to see improvements in this. Also, currently there seems to be bug in the json output: johnterickson/cargo2junit#9
janderholm commentedon Sep 19, 2019
I would also like duration output as part of the output. Furthermore when running multiple suites at the same time there's no way of knowing what test belong to what suite. It would be nice to name the suite somehow. Possibly connect it to the source file relative to the workspace root?
Here's a slimmed down example:
Since both the parent crate and the subcrate have test modules called "test" and a test called "test_true there's no way of knowing which one failed.
gilescope commentedon Sep 26, 2020
Are people happy with
cargo test -- -Zunstable-options --format json? Should we be looking to stabilse this argument? (I noticerustccalls the same argument--message-format)61 remaining items
epage commentedon Apr 26, 2023
In addition to what ehuss said, I think a good test bed for json output will be rust-lang/cargo#5609.
While its not my highest priority (finishing lints rfc, cargo-script rfc, and msrv are), I am also starting to put some attention to the problems of testing within Rust. Overall, I see this effort as (1) applying the lessons from all the different custom test harness and (2) finding the appropriate places to stablize cargo-nextest behavior. I am first experimenting with an alternative design to libtest itself that I'm hoping will offer the benefits we had hoped for from rust-lang/rfcs#2318. My hope is that both the libtest and cargo-test sides of this work (including rust-lang/cargo#5609) will lead to a solid json format.
nrc commentedon May 2, 2023
I'm not sure I do, but the overall feeling I get from this thread is that it is at least not optimal
emilyalbini commentedon May 2, 2023
So, I have some recent experience using this in #108659. The goal of that PR was to extract stats about the individual tests executed by bootstrap (name of the test, outcome, eventual ignore reason), and there were a few issues I noticed with the current way the JSON output works.
The minor issue is just some unclarity in the JSON format, that could likely be shuffled a bit to be more coherent (for example there is both a
reasonand amessagewhen a test is ignored).The major issue I found is that a tool wanting to extract the JSON information for further processing and wanting to show the familiar CLI UI to users needs to reimplement the whole rendering code, as there is no
rendered/rendered-ansioption like rustc. This resulted in me having to write a full renderer in bootstrap just to know which tests were executed. The design of the flag will likely not be trivial, as arenderedfield would also need to distinguish between the "terse" (used by bootstrap) and "verbose" (used by default by cargo) outputs.Disable test tool window on toolchain version >= 1.70-beta
wyatt-herkamp commentedon Nov 19, 2023
One flaw I see inside the current implementation is Doc Tests
cargo +nightly test -- -Z unstable-options --list --format jsonFor Doc Tests it will print out
Doc-tests heckThen print out the doc tests in JSON format.
This string is weird if we are trying to provide a machine readable format.
Then I noticed that start line and end lines are not provided in Doc Tests.
So start lines should be in the actual JSON. And the
Doc-testsmessage should remove and maybe a test type attribute should be added to the format. Or when the event "discovery" is printed it should have an attribute indicating the start of Doc Tests.Noratrieb commentedon Jan 9, 2024
It would be easier to evolve the format it was versioned. This way we could also ship a suboptimal version earlier and always upgrade it with breaking changes to a more optimal version later (while keeping the old version working. It does of course still require a design for the first stable version that we can be sure won't break in the future, but doesn't require a design that's perfect and fits all use cases).
epage commentedon Jan 9, 2024
FYI the testing devex team was recently formed and we've decided that the first order of business is working to stabilize the json output. We will be writing up our plan soon for how we'll go about that effort.
epage commentedon Jan 9, 2024
I've posted a draft of my plan at https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163
epage commentedon Jan 19, 2024
I've posted an eRFC for the stabilization plan: rust-lang/rfcs#3558
gilescope commentedon May 16, 2024
For CI we use the json for coverage, but humans like to look at the pretty output so we run two jobs one running the tests with the default format, one with the json format. It would be great to be able to have the json format being saved to a specified file while at the same time std out produces nice human format.
epage commentedon May 16, 2024
In my plan for things, the responsibility for that would be moved up into
cargo test. Cargo will exclusively accept json output from libtest and then do all of the rendering. This means users will need to ask Cargo for the json output as well.