Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-9222] XCTest performance tests are always much slower on the first measurement #358

Open
swift-ci opened this issue Nov 11, 2018 · 5 comments

Comments

@swift-ci
Copy link

@swift-ci swift-ci commented Nov 11, 2018

Previous ID SR-9222
Radar None
Original Reporter bnut (JIRA User)
Type Bug

Attachment: Download

Environment
Xcode 10.1 (10B61)
MacBook Pro (15-inch, 2016)
2.9 GHz Intel Core i7
16 GB 2133 MHz LPDDR3
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: d535dadca1b8fdfb15ae346fea01370a

Issue Description:

Using the measure block in XCTest always results in a significant overhead on the first measurement.

import XCTest
@testable import xctperf_test


final class xctperf_testTests: XCTestCase {
    func testExample() {
        measure {
            for _ in 0 ... 10_000 {
                XCTAssertEqual("abc", "abc")
            }
        }
    }


    static var allTests = [
        ("testExample", testExample),
    ]
}

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Nov 11, 2018

Comment by Andrew Bennett (JIRA)

@briancroom this might be something you're interested in, based on other similar tickets.

@briancroom
Copy link
Collaborator

@briancroom briancroom commented Nov 12, 2018

Hi bnut (JIRA User), based on the screenshot, it looks like you are seeing this problem when using Xcode’s XCTest, rather than swift-corelibs-xctest. Could you confirm?

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Nov 12, 2018

Comment by Andrew Bennett (JIRA)

Yes, that's correct @briancroom - sorry if you were the wrong person to ping.

I created the target with SwiftPM using swift package init which adds some linux support, and then using swift package generate-xcodeproj. I don't think this would lead it to use swift-corelibs-xctest though.

I've verified this by looking at the `XCTest` module, which contains:

public var XCT_UI_TESTING_AVAILABLE: Int32 { get }
//
//  Copyright (c) 2014-2015 Apple Inc. All rights reserved.
//

Which doesn't seem to be in the corelibs version.

@briancroom
Copy link
Collaborator

@briancroom briancroom commented Nov 18, 2018

Hey bnut (JIRA User), sorry for the delay. No worries, I'm a fine person to ask about this!

Yeah, swift-corelibs-xctest is generally only used on non-Apple platforms, so this would be Xcode's version of XCTest that you're using here. I haven't seen this particular behavior before, but it does seem like a bug. Would you be willing to file a Radar about this on https://bugreport.apple.com and attach an Xcode project which reproduces the problem?

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Nov 18, 2018

Comment by Andrew Bennett (JIRA)

Thanks for the feedback @briancroom, I've created a report at https://bugreport.apple.com/web/?problemID=46153989

Please let me know if you need more information.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants