Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-8436] xctest should have an option to emit parsable output #361

Open
aciidb0mb3r opened this issue Aug 1, 2018 · 8 comments
Open

[SR-8436] xctest should have an option to emit parsable output #361

aciidb0mb3r opened this issue Aug 1, 2018 · 8 comments

Comments

@aciidb0mb3r
Copy link
Member

@aciidb0mb3r aciidb0mb3r commented Aug 1, 2018

Previous ID SR-8436
Radar rdar://problem/42828674
Original Reporter @aciidb0mb3r
Type Bug
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: 5c1c8b6daff9759845d17ee54c42f4a4

Issue Description:

Similar to the Swift compiler, xctest should provide a way to emit parsable output so the clients can control and format the results.

@aciidb0mb3r
Copy link
Member Author

@aciidb0mb3r aciidb0mb3r commented Aug 1, 2018

@swift-ci create

@keith
Copy link
Contributor

@keith keith commented Aug 1, 2018

If you have thoughts on the approach here I might be interested in implementing this.

@aciidb0mb3r
Copy link
Member Author

@aciidb0mb3r aciidb0mb3r commented Aug 1, 2018

That would be awesome! Have you seen Swift's parsable output? It would be great if we can make something similar in XCTest.

/cc @briancroom @ddunbar

@keith
Copy link
Contributor

@keith keith commented Aug 1, 2018

Yes I have. So we'd be willing to entirely transform the output coming from Darwin's XCTest?

@aciidb0mb3r
Copy link
Member Author

@aciidb0mb3r aciidb0mb3r commented Aug 1, 2018

We basically need two implementations here:

1. corelibs-xctest for Linux. This should be somewhat straightforward. We can add a -parseable-output option to the test binaries.
2. I am not clear on what should happen for Darwin. SwiftPM uses the xctest command-line tool to run the tests. A -parseable-output option on xctest command-line tool would make sense to me but that tool is not open-sourced. It looks like we can use XCTestObservation APIs to generate the required output but that would mean we can't use the xctest tool.

Lets wait for @briancroom's thoughts on this.

@briancroom
Copy link
Collaborator

@briancroom briancroom commented Aug 7, 2018

This is indeed a long-standing request for Darwin XCTest as well. If we can come up with a format that we're comfortable with, I think we could provide an implementation there as well, via an opt-in command line argument.

I assume you would like this to be output in a streaming fashion as the tests are run?

One complication to any attempt at parsing test results via stdout/stderr is that, since arbitrary user code is running in the process, XCTest's output can end up interleaved with output from the tests themselves. There's not really anything we can do about that besides moving to a model of using a separate pipe for result reporting, which is what Xcode does when running tests through the tool. The structured format being discussed here could potentially be fed either to stderr or to an alternate sink (e.g. a named pipe) to provide a solution here.

@aciidb0mb3r
Copy link
Member Author

@aciidb0mb3r aciidb0mb3r commented Aug 7, 2018

Yep, a streaming output on stderr/separate pipe sounds perfect!

@MaxDesiatov
Copy link
Contributor

@MaxDesiatov MaxDesiatov commented Jun 14, 2020

In the meantime, here's a PR that at least allows making test observations (i.e. reports) configurable in situations where you can call XCTMain directly (i.e. LinuxMain.swift) #306

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants