improves CI Testing

[dnadoba]

improve CI Testing

Motivation:

Make sure RSocket does work on iOS & Linux too.

Modifications:

• build & run tests on Ubuntu
• run tests on iOS
• enable thread sanitizer for iOS & macOS
• build in parallel
• run tests in parallel (except performance tests)
• use xcodebuild to build & run tests on macOS

Result:

Tests now run on iOS and Linux too and should be slightly faster because of parallel building and testing

Discussion

Thread Sanitizer on Linux

On Linux, the test fail with thread sanitizer enabled. Thread santizer support on linux is new for swift and I think there is still a bug in it.
For Example, RequestResponseTerminationBehaviourTests.testRequesterSendsCancel fails but it just calls a pure function and does not mutate any state. I don't have a working linux environment to debug this issue further so I would just leave thread sanitizer disabled for now. It is still enabled on macOS.

Thread Sanitizer Error for RequestResponseTerminationBehaviourTests.testRequesterSendsCancel

Test Suite 'Selected tests' started at 2021-03-18 18:21:43.116
584
Test Suite 'RequestResponseTerminationBehaviourTests' started at 2021-03-18 18:21:43.129
585
Test Case 'RequestResponseTerminationBehaviourTests.testRequesterSendsCancel' started at 2021-03-18 18:21:43.129
586
ThreadSanitizer:DEADLYSIGNAL
587
==2562==ERROR: ThreadSanitizer: SEGV on unknown address 0x07fda4cff260 (pc 0x55ff68f291a5 bp 0x7c0000000000 sp 0x7ffc48ae00b8 T2562)
588
==2562==The signal is caused by a WRITE memory access.
589
    #0 __tsan::MemoryAccess(__tsan::ThreadState*, unsigned long, unsigned long, int, bool, bool) /home/build-user/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_atomic_clang_x86.h:81 (RSocketPackageTests.xctest+0x1071a5)
590
    #1 StoreShadow /home/build-user/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:613 (RSocketPackageTests.xctest+0x1071a5)
591
    #2 MemoryAccessImpl1 /home/build-user/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:701 (RSocketPackageTests.xctest+0x1071a5)
592
    #3 MemoryAccess /home/build-user/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:878 (RSocketPackageTests.xctest+0x1071a5)
593
    #4 __tsan::ExternalAccess(void*, void*, void*, void (*)(__tsan::ThreadState*, unsigned long, unsigned long, int)) /home/build-user/llvm-project/compiler-rt/lib/tsan/rtl/tsan_external.cpp:67 (RSocketPackageTests.xctest+0xa01dc)
594
    #5 $s16RSocketCoreTests035RequestResponseTerminationBehaviourC0C24testRequesterSendsCancelyyFSbyKXEfu_ /home/runner/work/rsocket-swift/rsocket-swift/<compiler-generated>:? (RSocketPackageTests.xctest+0x51dc27)
595
    #6 $s16RSocketCoreTests035RequestResponseTerminationBehaviourC0C24testRequesterSendsCancelyyFSbyKXEfu_TA TerminationBehaviourTests.swift.o:? (RSocketPackageTests.xctest+0x51dc97)
596
    #7 $s6XCTest13XCTAssertTrue__4file4lineySbyKXK_SSyXKs12StaticStringVSutF ??:? (libXCTest.so+0x3d8a5)
597
    #8 $s16RSocketCoreTests035RequestResponseTerminationBehaviourC0C24testRequesterSendsCancelyyF /home/runner/work/rsocket-swift/rsocket-swift/Tests/RSocketCoreTests/TerminationBehaviourTests.swift:27 (RSocketPackageTests.xctest+0x51dbbf)
598
    #9 $s16RSocketCoreTests035RequestResponseTerminationBehaviourC0C0a7PackageC0E05__allc2__defgC033_A8597678E06260891431B4EF4ED59470LLSaySS_yycACctGvpZfiyycACcfu_yycfu0_ /home/runner/work/rsocket-swift/rsocket-swift/.build/x86_64-unknown-linux-gnu/debug/RSocketPackageTestsTestlist.derived/RSocketCoreTests.swift:33 (RSocketPackageTests.xctest+0x526ccd)
599
    #10 $s16RSocketCoreTests035RequestResponseTerminationBehaviourC0C0a7PackageC0E05__allc2__defgC033_A8597678E06260891431B4EF4ED59470LLSaySS_yycACctGvpZfiyycACcfu_yycfu0_TA RSocketCoreTests.swift.o:? (RSocketPackageTests.xctest+0x52b9cd)
600
    #11 $sIeg_ytIegr_TR crtstuff.c:? (RSocketPackageTests.xctest+0x33d6b7)
601
    #12 $sIeg_ytIegr_TRTA.109 RSocketCoreTests.swift.o:? (RSocketPackageTests.xctest+0x52b98d)
602
    #13 $sytIegr_Ieg_TRTA XCTestCase.swift.o:? (libXCTest.so+0x37a20)
603
    #14 $s6XCTest4test33_3BE257A46ADB477C7BF2D39968B39F9DLLyyAA0A4CaseCKcyyKcxcAERbzlFyAEKcfU_TA XCTestCase.swift.o:? (libXCTest.so+0x37a8b)
604
    #15 $s6XCTest0A4CaseCs5Error_pIeggzo_ACytsAD_pIegnrzo_TRTA XCTestCase.swift.o:? (libXCTest.so+0x37a03)
605
    #16 $s6XCTest0A4CaseCyts5Error_pIegnrzo_ACsAD_pIeggzo_TRTA XCTestCaseSuite.swift.o:? (libXCTest.so+0x28576)
606
    #17 $s6XCTest0A4CaseC10invokeTestyyF ??:? (libXCTest.so+0x363b6)
607
    #18 $s6XCTest0A4CaseC7performyyAA0A3RunCF ??:? (libXCTest.so+0x36105)
608
    #19 $s6XCTestAAC3runyyF ??:? (libXCTest.so+0x39de5)
609
    #20 $s6XCTest0A5SuiteC7performyyAA0A3RunCF ??:? (libXCTest.so+0x3803f)
610
    #21 $s6XCTestAAC3runyyF ??:? (libXCTest.so+0x39de5)
611
    #22 $s6XCTest0A5SuiteC7performyyAA0A3RunCF ??:? (libXCTest.so+0x3803f)
612
    #23 $s6XCTestAAC3runyyF ??:? (libXCTest.so+0x39de5)
613
    #24 $s6XCTest7XCTMain_9argumentss5NeverOSayAA0A4CaseCm04testE5Class_SaySS_yAGKctG8allTeststG_SaySSGtF ??:? (libXCTest.so+0x34b19)
614
    #25 $s6XCTest7XCTMainys5NeverOSayAA0A4CaseCm04testD5Class_SaySS_yAFKctG8allTeststGF ??:? (libXCTest.so+0x34548)
615
    #26 main /home/runner/work/rsocket-swift/rsocket-swift/.build/x86_64-unknown-linux-gnu/debug/RSocketPackageTestsTestlist.derived/main.swift:8 (RSocketPackageTests.xctest+0x52c9f4)
616
    #27 __libc_start_main ??:? (libc.so.6+0x270b2)
617
    #28 _start ??:? (RSocketPackageTests.xctest+0x7cd5d)
618

619
ThreadSanitizer can not provide additional info.
620
SUMMARY: ThreadSanitizer: SEGV /home/build-user/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_atomic_clang_x86.h:81 in atomic_store<__sanitizer::atomic_uint64_t>
621
==2562==ABORTING

xcodebuild vs. SwiftPM

• xcodebuild is not available on Linux
• SwiftPM does not yet support running tests on iOS. We need to use xcodebuild for that.
• SwiftPM and xcodebuild do not share any caches and will refetch and rebuild all dependencies which should be avoided.
• Performance Tests are tricky:
  • Both SwiftPM and xcodebuild do support a filter option which lets us run only a subset of the tests.
  • SwiftPM only builds test targets which are needed
  • xcodebuild builds everything (or tries to)
    • normal tests in debug mode are no problem
    • performance tests need to be build in release configuration. In release mode, @testable import is not supported but used in most tests targets and all but RSocketCorePerformanceTests fail to build

Because of the reasons above, I have decided to use xcodebuild to build & run tests for macOS & iOS but run performance tests using SwiftPM.
On Linux there was really no choice but to use SwiftPM.

[ciychodianda]

Just curious ,how will this run on Linux since we use Foundation which is just for Apple platforms?

[dnadoba]

Foundation is available on Linux too: https://github.com/apple/swift-corelibs-foundation

[ciychodianda]

Thanks, the documentation that I read was stale then.

[dnadoba]

Foundation on Linux is not as good as the one on Apple Platforms. Some API's are not implemented or crash at runtime but we are only using Data which should be fine.

[OlegDokuka]

LGTM