New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[XCTestObservation] Add XCTestSuite announcements #84

Merged
merged 1 commit into from Apr 3, 2016

Conversation

Projects
None yet
3 participants
@modocache
Collaborator

modocache commented Mar 27, 2016

What's in this pull request?

Apple XCTest's XCTestObservation protocol includes methods that announce when an XCTestSuite will begin executing, or has finished executing. Add these same announcements to swift-corelibs-xctest.

swift-corelibs-xctest did not already defined XCTestSuite, so add it.

Apple XCTest allows XCTestObservation listeners to determine the number of passing and failing tests by accessing an XCTestSuite's testRun property, but this is not yet supported in swift-corelibs-xctest.

Why merge this pull request?

  • This makes the swift-corelibs-xctest and Apple XCTest versions of XCTestObservation identical, bringing us closer to our Swift 3 goal of API parity.

What are the downsides of merging this pull request?

  • This adds XCTestSuite to the codebase, but not XCTestRun. As a result, the most interesting aspect of test announcements (how many tests passed/failed) is still unimplemented. Still, I feel like this is a positive step forward, and one that keeps this pull request small enough to actually review.
  • swift-corelibs-xctest doesn't use XCTestSuite to represent the collection of test cases it executes. As a result, the quality of implementation here leaves much to be desired. On the other hand, refactoring the codebase to seamless integrate XCTestSuite would involve more widespread changes, which would make this pull request more difficult to review. If you ask me, thanks to the functional test cases, we could merge this, then easily refactor in a subsequent pull request.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Mar 28, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.
@mike-ferris

This comment has been minimized.

Show comment
Hide comment
@mike-ferris

mike-ferris Mar 29, 2016

Member

This seems like a reasonable first step. I agree generally with your assessment in the second "downside" you mention. The weirdness around having the "All" suite and the bundle suite be peers in a list of root suites instead of having a containment relationship arises from this, I guess. I would definitely like to see continued movement towards having test suites constructed and then used to control the running of the tests instead of just being constructed in order to send the observations. But I am OK with taking this in incremental stages.

Member

mike-ferris commented Mar 29, 2016

This seems like a reasonable first step. I agree generally with your assessment in the second "downside" you mention. The weirdness around having the "All" suite and the bundle suite be peers in a list of root suites instead of having a containment relationship arises from this, I guess. I would definitely like to see continued movement towards having test suites constructed and then used to control the running of the tests instead of just being constructed in order to send the observations. But I am OK with taking this in incremental stages.

@mike-ferris

This comment has been minimized.

Show comment
Hide comment
@mike-ferris

mike-ferris Mar 29, 2016

Member

@swift-ci please test

Member

mike-ferris commented Mar 29, 2016

@swift-ci please test

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Mar 30, 2016

Collaborator

Here's the error from Linux CI:

/usr/bin/ld: /home/buildnode/jenkins/workspace/swift-corelibs-xctest-PR-Linux/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/test-Package: hidden symbol `_TFC6XCTest10XCTestCasemP33_3BE257A46ADB477C7BF2D39968B39F9D5_nameSS' isn't defined
/usr/bin/ld: final link failed: Bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I think this is the error @briancroom mentioned in #69 (is there a JIRA issue for this problem?). I'll try to address this in the coming days.

Collaborator

modocache commented Mar 30, 2016

Here's the error from Linux CI:

/usr/bin/ld: /home/buildnode/jenkins/workspace/swift-corelibs-xctest-PR-Linux/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/test-Package: hidden symbol `_TFC6XCTest10XCTestCasemP33_3BE257A46ADB477C7BF2D39968B39F9D5_nameSS' isn't defined
/usr/bin/ld: final link failed: Bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I think this is the error @briancroom mentioned in #69 (is there a JIRA issue for this problem?). I'll try to address this in the coming days.

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Apr 2, 2016

Collaborator

@mike-ferris-apple I've resolved the Linux build issue. Could you ask @swift-ci to please test? If the tests pass, I'll merge this to achieve the "incremental stages" you describe (or you can request "@swift-ci please test and merge").

#86 is the realization of the "continued movement towards having test suites constructed and then used to control the running of the tests" you describe.

Collaborator

modocache commented Apr 2, 2016

@mike-ferris-apple I've resolved the Linux build issue. Could you ask @swift-ci to please test? If the tests pass, I'll merge this to achieve the "incremental stages" you describe (or you can request "@swift-ci please test and merge").

#86 is the realization of the "continued movement towards having test suites constructed and then used to control the running of the tests" you describe.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 2, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 2, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 2, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.
@parkera

This comment has been minimized.

Show comment
Hide comment
@parkera

parkera Apr 2, 2016

Member

@swift-ci please test

Member

parkera commented Apr 2, 2016

@swift-ci please test

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 3, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.
[XCTestObservation] Add XCTestSuite announcements
Apple XCTest's `XCTestObservation` protocol includes methods that
announce when an `XCTestSuite` will begin executing, or has finished
executing. Add these same announcements to swift-corelibs-xctest.

swift-corelibs-xctest did not already defined `XCTestSuite`, so add it.

Apple XCTest allows `XCTestObservation` listeners to determine the
number of passing and failing tests by accessing an `XCTestSuite`'s
`testRun` property, but this is not yet supported in
swift-corelibs-xctest.

@modocache modocache merged commit a431376 into apple:master Apr 3, 2016

@modocache modocache deleted the modocache:observation-test-suite branch Apr 3, 2016

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 3, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache pushed a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Brian Gesiak Brian Gesiak
Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 6, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.

modocache added a commit to modocache/swift-corelibs-xctest that referenced this pull request Apr 7, 2016

Use XCTestRun to execute tests and report results
A major goal for swift-corelibs-xctest is API parity with Apple XCTest.
This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`.

In Apple XCTest, `XCTestRun` is responsible for keeping track of the
result of a test run. It's an integral part of how Apple XCTest works.
swift-corelibs-xctest, on the other hand, used a global array of
`XCTRun` structs to keep track of how many tests passed/failed.

While it may have been possible to tack on `XCTestRun` to the
swift-corelibs-xctest mechanism for failure reporting, this commit
instead fully integrates it. As a result, the changes are widespread:
gone is `XCTFailureHandler`, `XCTRun`, and other internal structures.
In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract
class, `XCTestRun`, and its subclasses `XCTestCaseRun` and
`XCTestSuiteRun`.

In conjunction with the new `XCTestSuite`-related observation methods
from apple#84,
test reporting is now done exclusively through `XCTestObservation`.
As a result, test output is now nearly identical to Apple XCTest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment