Discard swift.ld in favor of portable solution and support gold linker in Swift #1157

Merged
merged 2 commits into from Feb 18, 2016

Conversation

Projects
None yet
10 participants
@hpux735
Contributor

hpux735 commented Feb 2, 2016

This pull request addresses the conformances and metadata sections differently than the former Linux solution. Unlike the former solution, the size of the sections are not stored as part of the sections themselves. Rather, section start and end markers are stored in the binary, and are used to compute the size of these sections at load time.

This solution is more portable and allows the use of the gold linker, which is far more stable than BFD on ARM targets. Specifically, the gold linker does not generate R_ARM_COPY relocations, which break protocol conformances, though BFD does. Although gold does address some of these issues, -Bsymbolic is still required because swift still generates R_ARM_REL32 relocations, which are illegal. Finally, because the JIT images also contain the illegal relocations, RuntimeDyld refuses to process them, and as such the interpreter tests are disabled on ARM..

In further support of the use of the gold linker on arm-linux, we added a switch to swift for the selection of linker, and under arm, the gold linker is selected as the default.

This work was the topic of an RFC on the swift-dev mailing list:
https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160111/000822.html

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 2, 2016

Collaborator

I think the FreeBSD port is also using the linker script, as does the Cygwin port.

/cc @dcci @tinysun212 What do you think?

Collaborator

gribozavr commented Feb 2, 2016

I think the FreeBSD port is also using the linker script, as does the Cygwin port.

/cc @dcci @tinysun212 What do you think?

test/DebugInfo/ASTSection_ObjC.swift
@@ -12,6 +12,9 @@
// REQUIRES: executable_test
// REQUIRES: objc_interop
+// UNSUPPORTED: OS=linux-gnu
+// UNSUPPORTED: OS=linux-gnueabihf
+// UNSUPPORTED: OS=freebsd

This comment has been minimized.

@modocache

modocache Feb 2, 2016

Collaborator

Are these OS's not already covered by the REQUIRES: objc_interop requirement above? I believe #1032 disabled this test for all Linux variants.

@modocache

modocache Feb 2, 2016

Collaborator

Are these OS's not already covered by the REQUIRES: objc_interop requirement above? I believe #1032 disabled this test for all Linux variants.

This comment has been minimized.

@hpux735

hpux735 Feb 2, 2016

Contributor

Yep, you're right. Removed.

@hpux735

hpux735 Feb 2, 2016

Contributor

Yep, you're right. Removed.

test/Misc/expression_too_complex.swift
@@ -1,4 +1,4 @@
-// RUN: %target-parse-verify-swift -solver-memory-threshold 5000
+// RUN: %target-parse-verify-swift -solver-memory-threshold 4000

This comment has been minimized.

@modocache

modocache Feb 2, 2016

Collaborator

What's the intention behind making this change? Does using gold have an effect on %target-parse-verify-swift, or was this included by accident?

@modocache

modocache Feb 2, 2016

Collaborator

What's the intention behind making this change? Does using gold have an effect on %target-parse-verify-swift, or was this included by accident?

This comment has been minimized.

@hpux735

hpux735 Feb 2, 2016

Contributor

The reasoning is that the 5000 threshold was sufficient memory to process the expression. Reducing it to 4000 allows the test to pass on ARM, and it still passes on x86_64.

@hpux735

hpux735 Feb 2, 2016

Contributor

The reasoning is that the 5000 threshold was sufficient memory to process the expression. Reducing it to 4000 allows the test to pass on ARM, and it still passes on x86_64.

This comment has been minimized.

@modocache

modocache Feb 2, 2016

Collaborator

Neat! 👌

@modocache

modocache Feb 2, 2016

Collaborator

Neat! 👌

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 2, 2016

Contributor

By the way, the failing test count in arm-linux is down to 5 (from the 40-or-so prior to this patch)


Testing Time: 601.26s


Failing Tests (5):
Swift :: 1_stdlib/VarArgs.swift
Swift :: Driver/Dependencies/one-way-external.swift
Swift :: Driver/sdk.swift
Swift :: IRGen/c_layout.sil
Swift :: IRGen/class_resilience.swift

Expected Passes : 1699
Expected Failures : 67
Unsupported Tests : 676
Unexpected Failures: 5

Contributor

hpux735 commented Feb 2, 2016

By the way, the failing test count in arm-linux is down to 5 (from the 40-or-so prior to this patch)


Testing Time: 601.26s


Failing Tests (5):
Swift :: 1_stdlib/VarArgs.swift
Swift :: Driver/Dependencies/one-way-external.swift
Swift :: Driver/sdk.swift
Swift :: IRGen/c_layout.sil
Swift :: IRGen/class_resilience.swift

Expected Passes : 1699
Expected Failures : 67
Unsupported Tests : 676
Unexpected Failures: 5

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Feb 2, 2016

Collaborator

@hpux735 That's great! Kudos to you and @tienex, this is fantastic work. I'm pulling it down now to give it a whirl.

I've looked into the c_layout.sil failure and wrote up what I found in #1103. To fix the failure, you'll probably need to modify the IRGen/c_layout.sil test for the default char-signedness of your platform--which might be unsigned in your case.

Collaborator

modocache commented Feb 2, 2016

@hpux735 That's great! Kudos to you and @tienex, this is fantastic work. I'm pulling it down now to give it a whirl.

I've looked into the c_layout.sil failure and wrote up what I found in #1103. To fix the failure, you'll probably need to modify the IRGen/c_layout.sil test for the default char-signedness of your platform--which might be unsigned in your case.

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@modocache Yes, that's correct, 'char' is unsigned on Linux/arm, the very same issue you have, I see you did already some work on it! Cool. The other failure that can be fixed quickly is 'Driver/Dependencies/one-way-external.swift', it is failing because it tries to run touch with year 3000, which fails on Linux/arm because of 32-bits time_t, changing it to 2037 makes it pass. SDK seems just a matter of CHECK order. This leaves VarArgs and class_resiliance to be fixed.

Contributor

tienex commented Feb 2, 2016

@modocache Yes, that's correct, 'char' is unsigned on Linux/arm, the very same issue you have, I see you did already some work on it! Cool. The other failure that can be fixed quickly is 'Driver/Dependencies/one-way-external.swift', it is failing because it tries to run touch with year 3000, which fails on Linux/arm because of 32-bits time_t, changing it to 2037 makes it pass. SDK seems just a matter of CHECK order. This leaves VarArgs and class_resiliance to be fixed.

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Feb 2, 2016

Collaborator

The following changes get IRGen/class_resilience.swift passing on Android armv7. I'm not sure why the index passed to getelementptr would be offset by three here--perhaps you have some insight here, @tienex?

diff --git a/test/IRGen/class_resilience.swift b/test/IRGen/class_resilience.swift
index 6868574..b9782fa 100644
--- a/test/IRGen/class_resilience.swift
+++ b/test/IRGen/class_resilience.swift
@@ -232,11 +232,11 @@ public class MyResilientChild : MyResilientParent {
 // CHECK:             [[SIZE_METADATA:%.*]] = call %swift.type* @_TMaV16resilient_struct4Size()
 // CHECK:             call void @swift_initClassMetadata_UniversalStrategy(
 // CHECK-native:      [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 12
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 15
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience26ClassWithResilientProperty1sV16resilient_struct4Size
 // CHECK-native-NEXT: [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 13
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 16
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience26ClassWithResilientProperty5colorVs5Int32
 // CHECK:             ret %swift.type* [[METADATA]]
@@ -249,11 +249,11 @@ public class MyResilientChild : MyResilientParent {
 // CHECK:             [[RECTANGLE_METADATA:%.*]] = call %swift.type* @_TMaV16resilient_struct9Rectangle()
 // CHECK:             call void @swift_initClassMetadata_UniversalStrategy(
 // CHECK-native:      [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 11
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 14
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience33ClassWithResilientlySizedProperty1rV16resilient_struct9Rectangle
 // CHECK-native-NEXT: [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 12
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 15
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience33ClassWithResilientlySizedProperty5colorVs5Int32
 // CHECK:             ret %swift.type* [[METADATA]]
Collaborator

modocache commented Feb 2, 2016

The following changes get IRGen/class_resilience.swift passing on Android armv7. I'm not sure why the index passed to getelementptr would be offset by three here--perhaps you have some insight here, @tienex?

diff --git a/test/IRGen/class_resilience.swift b/test/IRGen/class_resilience.swift
index 6868574..b9782fa 100644
--- a/test/IRGen/class_resilience.swift
+++ b/test/IRGen/class_resilience.swift
@@ -232,11 +232,11 @@ public class MyResilientChild : MyResilientParent {
 // CHECK:             [[SIZE_METADATA:%.*]] = call %swift.type* @_TMaV16resilient_struct4Size()
 // CHECK:             call void @swift_initClassMetadata_UniversalStrategy(
 // CHECK-native:      [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 12
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 15
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience26ClassWithResilientProperty1sV16resilient_struct4Size
 // CHECK-native-NEXT: [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 13
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 16
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience26ClassWithResilientProperty5colorVs5Int32
 // CHECK:             ret %swift.type* [[METADATA]]
@@ -249,11 +249,11 @@ public class MyResilientChild : MyResilientParent {
 // CHECK:             [[RECTANGLE_METADATA:%.*]] = call %swift.type* @_TMaV16resilient_struct9Rectangle()
 // CHECK:             call void @swift_initClassMetadata_UniversalStrategy(
 // CHECK-native:      [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 11
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 14
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience33ClassWithResilientlySizedProperty1rV16resilient_struct9Rectangle
 // CHECK-native-NEXT: [[METADATA_PTR:%.*]] = bitcast %swift.type* [[METADATA]] to [[INT]]*
-// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 12
+// CHECK-native-NEXT: [[FIELD_OFFSET_PTR:%.*]] = getelementptr inbounds [[INT]], [[INT]]* [[METADATA_PTR]], [[INT]] 15
 // CHECK-native-NEXT: [[FIELD_OFFSET:%.*]] = load [[INT]], [[INT]]* [[FIELD_OFFSET_PTR]]
 // CHECK-native-NEXT: store [[INT]] [[FIELD_OFFSET]], [[INT]]* @_TWvdvC16class_resilience33ClassWithResilientlySizedProperty5colorVs5Int32
 // CHECK:             ret %swift.type* [[METADATA]]
@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 2, 2016

Collaborator

CC @slavapestov for the class_resilience.swift test.

Collaborator

gribozavr commented Feb 2, 2016

CC @slavapestov for the class_resilience.swift test.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

(Sorry, missed that you are William Dillon.) We can avoid runtime cost by using .s or .ll files instead of C++ files to define these symbols: https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160118/000914.html Did that not work?

Member

jckarter commented Feb 2, 2016

(Sorry, missed that you are William Dillon.) We can avoid runtime cost by using .s or .ll files instead of C++ files to define these symbols: https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160118/000914.html Did that not work?

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter We tried, but the .ll approach resulted in a too complex expression, while the .s variant has to special case arm because of assembly syntax. In the end, the major problem is that using subtraction in the assembly produces PC-relative relocations, they work on ARM with -Bsymbolic, but will fail with gold on x86-64 because the relocation is unsupported in PIC model or results in being evaluated to zero using BFD linker.

Contributor

tienex commented Feb 2, 2016

@jckarter We tried, but the .ll approach resulted in a too complex expression, while the .s variant has to special case arm because of assembly syntax. In the end, the major problem is that using subtraction in the assembly produces PC-relative relocations, they work on ARM with -Bsymbolic, but will fail with gold on x86-64 because the relocation is unsupported in PIC model or results in being evaluated to zero using BFD linker.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

Which relocation is it that gold doesn't support—was it R_X86_64_PC64? Was just running into this in #1154 while testing on Linux. Is that something that can be fixed in gold?

Member

jckarter commented Feb 2, 2016

Which relocation is it that gold doesn't support—was it R_X86_64_PC64? Was just running into this in #1154 while testing on Linux. Is that something that can be fixed in gold?

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter Exactly, it produces an R_X86_64_PC64 relocation, @compnerd may help here, he has some good knowledge of gold.

Contributor

tienex commented Feb 2, 2016

@jckarter Exactly, it produces an R_X86_64_PC64 relocation, @compnerd may help here, he has some good knowledge of gold.

@slavapestov

This comment has been minimized.

Show comment
Hide comment
@slavapestov

slavapestov Feb 2, 2016

Member

@modocache @tienex Perhaps the failing class_resilience test is caused by differences in alignment? Was it failing on Linux/ARM before this patch too?

Member

slavapestov commented Feb 2, 2016

@modocache @tienex Perhaps the failing class_resilience test is caused by differences in alignment? Was it failing on Linux/ARM before this patch too?

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

@tienex The R_X86_64_PC64 relocation comes from the .end - .start calculation. We already make assumptions that Swift images will be small code model (<2GB), and .end should always be ahead of .start, so it should be possible to assume it's positive and 32-bit, and pad it out to 64 bits with a literal zero:

.align 8
.long (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
.long 0

which should produce a PC32 relocation instead.

Member

jckarter commented Feb 2, 2016

@tienex The R_X86_64_PC64 relocation comes from the .end - .start calculation. We already make assumptions that Swift images will be small code model (<2GB), and .end should always be ahead of .start, so it should be possible to assume it's positive and 32-bit, and pad it out to 64 bits with a literal zero:

.align 8
.long (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
.long 0

which should produce a PC32 relocation instead.

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Feb 2, 2016

Collaborator

@slavapestov Yes, the failure occurs prior to this patch as well.

Collaborator

modocache commented Feb 2, 2016

@slavapestov Yes, the failure occurs prior to this patch as well.

@slavapestov

This comment has been minimized.

Show comment
Hide comment
@slavapestov

slavapestov Feb 2, 2016

Member

@modocache Oh, the problem is obvious.

Those CHECK lines only run when no ObjC runtime is present. We have CI for 32-bit and 64-bit Darwin platforms, but on Linux we're only testing 64-bit. So I think you can safely replace those offsets with {{12|15}}, etc.

Member

slavapestov commented Feb 2, 2016

@modocache Oh, the problem is obvious.

Those CHECK lines only run when no ObjC runtime is present. We have CI for 32-bit and 64-bit Darwin platforms, but on Linux we're only testing 64-bit. So I think you can safely replace those offsets with {{12|15}}, etc.

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter I'll perform more tests tomorrow, I'm on the GMT+1 timezone, no offense :-) We wanted to avoid to clutter the file with ifdefs supporting arm and little/big endian configurations, also we would need to change the top-level cmake file to enable support for assembly files, by default cmake doesn't know how to handle them, and we found some issues with cmake 2.8 which seems to be the minimum version supported by swift, @hpux735 can give you more information regarding this issue.

Contributor

tienex commented Feb 2, 2016

@jckarter I'll perform more tests tomorrow, I'm on the GMT+1 timezone, no offense :-) We wanted to avoid to clutter the file with ifdefs supporting arm and little/big endian configurations, also we would need to change the top-level cmake file to enable support for assembly files, by default cmake doesn't know how to handle them, and we found some issues with cmake 2.8 which seems to be the minimum version supported by swift, @hpux735 can give you more information regarding this issue.

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 2, 2016

Contributor

@jckarter We were able to eventually get CMake to play nice with .S files, though the solution was a little hackish. CMake versions subsequent to 2.8 handle it more cleanly.

Contributor

hpux735 commented Feb 2, 2016

@jckarter We were able to eventually get CMake to play nice with .S files, though the solution was a little hackish. CMake versions subsequent to 2.8 handle it more cleanly.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

I wouldn't worry too much about big endian for now; there are far deeper endianness sins in IRGen and the runtime.

Member

jckarter commented Feb 2, 2016

I wouldn't worry too much about big endian for now; there are far deeper endianness sins in IRGen and the runtime.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

…though that's another thing an .ll file could abstract away. A zeroext of an i32 relative reference should get assembled as a zero-padded 32-bit relocation too.

Member

jckarter commented Feb 2, 2016

…though that's another thing an .ll file could abstract away. A zeroext of an i32 relative reference should get assembled as a zero-padded 32-bit relocation too.

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 2, 2016

Contributor

@slavapestov I applied your fix for class_resilience. Thanks!

Contributor

hpux735 commented Feb 2, 2016

@slavapestov I applied your fix for class_resilience. Thanks!

@tinysun212

This comment has been minimized.

Show comment
Hide comment
@tinysun212

tinysun212 Feb 2, 2016

Contributor

Cygwin port is not using the linker script swift.ld. It is reading the size of the sections directly from the COFF section header structure which is loaded in memory.

Contributor

tinysun212 commented Feb 2, 2016

Cygwin port is not using the linker script swift.ld. It is reading the size of the sections directly from the COFF section header structure which is loaded in memory.

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter I performed some tests, even in the case of R_X86_64_PC32 I get gold complain about it, unless I use -Bsymbolic, I could use .quad instead of .long and I would have the same result. BFD accepts it, but like R_X86_64_PC64 produces a zero value, should I assume -Bsymbolic is the default now? If so, we should make it also the default for non-ARM platforms when invoking clang++ from swift during linking shared objects, in this PR we specify it only for ARM/Thumb targets.

Contributor

tienex commented Feb 2, 2016

@jckarter I performed some tests, even in the case of R_X86_64_PC32 I get gold complain about it, unless I use -Bsymbolic, I could use .quad instead of .long and I would have the same result. BFD accepts it, but like R_X86_64_PC64 produces a zero value, should I assume -Bsymbolic is the default now? If so, we should make it also the default for non-ARM platforms when invoking clang++ from swift during linking shared objects, in this PR we specify it only for ARM/Thumb targets.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

Yeah, -Bsymbolic was supposed to be the default when building for all ELF, but that got accidentally reverted when we merged the arm-linux branch.

Member

jckarter commented Feb 2, 2016

Yeah, -Bsymbolic was supposed to be the default when building for all ELF, but that got accidentally reverted when we merged the arm-linux branch.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

I had added it back in #1154, and everything seems to work with BFD using PC64 relocations from metadata to nominal type descriptors, though gold still crashes when linking apps.

Member

jckarter commented Feb 2, 2016

I had added it back in #1154, and everything seems to work with BFD using PC64 relocations from metadata to nominal type descriptors, though gold still crashes when linking apps.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

Longer-term, it might be a good idea to investigate alternatives to -Bsymbolic; if we can mark only Swift-generated symbols to be eagerly resolved, that would be a little bit more hygienic for mixed-language projects. AIUI some awful C++ and Fortran features depend on load-time fixup to work as expected under ELF.

Member

jckarter commented Feb 2, 2016

Longer-term, it might be a good idea to investigate alternatives to -Bsymbolic; if we can mark only Swift-generated symbols to be eagerly resolved, that would be a little bit more hygienic for mixed-language projects. AIUI some awful C++ and Fortran features depend on load-time fixup to work as expected under ELF.

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter that's why we ended up with the non-assembly version, it worked independently of -Bsymbolic

Contributor

tienex commented Feb 2, 2016

@jckarter that's why we ended up with the non-assembly version, it worked independently of -Bsymbolic

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter Feb 2, 2016

Member

@tienex Sure, but the behavior of -Bsymbolic is the right thing for Swift-generated symbols.

Member

jckarter commented Feb 2, 2016

@tienex Sure, but the behavior of -Bsymbolic is the right thing for Swift-generated symbols.

@tienex

This comment has been minimized.

Show comment
Hide comment
@tienex

tienex Feb 2, 2016

Contributor

@jckarter We should be ready to switch back to the assembly version in a few mins, time to perform tests on x86-64, ARM works.

Contributor

tienex commented Feb 2, 2016

@jckarter We should be ready to switch back to the assembly version in a few mins, time to perform tests on x86-64, ARM works.

@hpux735 hpux735 referenced this pull request in apple/swift-corelibs-foundation Feb 2, 2016

Merged

Companion PR to Swift #1157 and 32-bit bug fixes #249

@slavapestov

This comment has been minimized.

Show comment
Hide comment
@slavapestov

slavapestov Feb 2, 2016

Member

@hpux735 This commit looks like an unrelated change snuck in -- also it would be great if you could add a bit more detail to the commit message:

hpux735@3a0be39

Thanks!

Member

slavapestov commented Feb 2, 2016

@hpux735 This commit looks like an unrelated change snuck in -- also it would be great if you could add a bit more detail to the commit message:

hpux735@3a0be39

Thanks!

@dcci

This comment has been minimized.

Show comment
Hide comment
@dcci

dcci Feb 2, 2016

Member

I'd be very happy to see the linker script dependency going away on FreeBSD. This way we can use the old FreeBSD ld in base (or gold from binutils). I won't be in a position to test this in the next few days, but I'll take care of FreeBSD breakages if they occur after the patch one goes in.

Thanks for taking care of this!

Member

dcci commented Feb 2, 2016

I'd be very happy to see the linker script dependency going away on FreeBSD. This way we can use the old FreeBSD ld in base (or gold from binutils). I won't be in a position to test this in the next few days, but I'll take care of FreeBSD breakages if they occur after the patch one goes in.

Thanks for taking care of this!

cmake/modules/AddSwift.cmake
+ if("${libkind}" STREQUAL "SHARED")
+ set(arch_subdir "${SWIFTLIB_DIR}/${SWIFTLIB_SINGLE_SUBDIR}")
+
+ add_custom_command(

This comment has been minimized.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Please use add_custom_command_target. add_custom_command is not safe in certain (subtle) scenarios in parallel builds.

    "Do not list the output in more than one independent target that
     may build in parallel or the two instances of the rule may
     conflict (instead use add_custom_target to drive the command and
     make the other targets depend on that one)."
@gribozavr

gribozavr Feb 2, 2016

Collaborator

Please use add_custom_command_target. add_custom_command is not safe in certain (subtle) scenarios in parallel builds.

    "Do not list the output in more than one independent target that
     may build in parallel or the two instances of the rule may
     conflict (instead use add_custom_target to drive the command and
     make the other targets depend on that one)."
cmake/modules/AddSwift.cmake
+
+ add_custom_command(
+ OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/swift_begin.o"
+ COMMAND "${CMAKE_COMMAND}" -E copy "${arch_subdir}/swift_begin.o" "${CMAKE_CURRENT_BINARY_DIR}/swift_begin.o"

This comment has been minimized.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Why do we need to copy object files around? Is this working around a limitation of CMake?

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Why do we need to copy object files around? Is this working around a limitation of CMake?

This comment has been minimized.

@tienex

tienex Feb 2, 2016

Contributor

This exactly work around the issue of creating a target, this creates effectively a dependency on the output of the add_custom_command. The problem is that there is no way to create just objects in cmake.

@tienex

tienex Feb 2, 2016

Contributor

This exactly work around the issue of creating a target, this creates effectively a dependency on the output of the add_custom_command. The problem is that there is no way to create just objects in cmake.

@@ -1131,16 +1144,6 @@ function(_add_swift_library_single target name)
set(PLIST_INFO_BUILD_VERSION)
endif()
- # On Linux and FreeBSD add the linker script that coalesces protocol
- # conformance sections. This wouldn't be necessary if the link was done by
- # the swift binary: rdar://problem/19007002

This comment has been minimized.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

This comment still applies. We wouldn't need to specify the object files manually if we linked the sources using swift.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

This comment still applies. We wouldn't need to specify the object files manually if we linked the sources using swift.

This comment has been minimized.

@hpux735

hpux735 Feb 3, 2016

Contributor

We kept the comment in spirit, but moved it up to where the begin/end objects are processed in AddSwift.cmake. See line 851.

@hpux735

hpux735 Feb 3, 2016

Contributor

We kept the comment in spirit, but moved it up to where the begin/end objects are processed in AddSwift.cmake. See line 851.

This comment has been minimized.

@gribozavr

gribozavr Feb 3, 2016

Collaborator

I see. Somehow I missed it.

@gribozavr

gribozavr Feb 3, 2016

Collaborator

I see. Somehow I missed it.

@@ -875,8 +894,10 @@ function(_add_swift_library_single target name)
INSTALL_IN_COMPONENT "${SWIFTLIB_INSTALL_IN_COMPONENT}")
add_library("${target}" ${libkind}

This comment has been minimized.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Would this archive these object files into static archive versions of the libraries? It seems like we don't want that.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Would this archive these object files into static archive versions of the libraries? It seems like we don't want that.

This comment has been minimized.

@tienex

tienex Feb 2, 2016

Contributor

The value of SWIFT_OBJECT_{BEGIN,END} is assigned only when libkind is SHARED, so they should not be present when generating static libraries targets.

@tienex

tienex Feb 2, 2016

Contributor

The value of SWIFT_OBJECT_{BEGIN,END} is assigned only when libkind is SHARED, so they should not be present when generating static libraries targets.

stdlib/public/runtime/CMakeLists.txt
+ add_library(${conformance_magic_end_name} STATIC swift_sections.cpp)
+ set_target_properties(${conformance_magic_end_name} PROPERTIES COMPILE_FLAGS "-DSWIFT_END")
+
+ add_custom_command(

This comment has been minimized.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Please use add_custom_command_target everywhere here.

@gribozavr

gribozavr Feb 2, 2016

Collaborator

Please use add_custom_command_target everywhere here.

@modocache

This comment has been minimized.

Show comment
Hide comment
@modocache

modocache Feb 9, 2016

Collaborator

Thanks, @hpux735! Looking forward to this.

I've been looking into the 1_stdlib/VarArgs.swift test failure above. To keep some tangential chatter off this thread, I've written up what I've found so far here: SwiftAndroid#15

If anyone familiar with how the ClangImporter works with va_list could comment on that issue, I'd very much appreciate it.

Collaborator

modocache commented Feb 9, 2016

Thanks, @hpux735! Looking forward to this.

I've been looking into the 1_stdlib/VarArgs.swift test failure above. To keep some tangential chatter off this thread, I've written up what I've found so far here: SwiftAndroid#15

If anyone familiar with how the ClangImporter works with va_list could comment on that issue, I'd very much appreciate it.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 10, 2016

Collaborator

@hpux735 Is add_custom_command_target() not covering this usecase? If you could use it, then there would be no need to create commands to extract .os for every library, we could just depend on the target properly.

Collaborator

gribozavr commented Feb 10, 2016

@hpux735 Is add_custom_command_target() not covering this usecase? If you could use it, then there would be no need to create commands to extract .os for every library, we could just depend on the target properly.

lib/Frontend/CompilerInvocation.cpp
+ // the ArchName in the RuntimeLibraryPath.
+ // By explicitly setting the architecture to the subtype that it aliases to,
+ // we remove that extra character while not introducing other side effects.
+ if (Triple.getOS() == llvm::Triple::Linux) {

This comment has been minimized.

@jrose-apple

jrose-apple Feb 10, 2016

Member

I'm a little concerned about this function mutating Triple. It should probably be passed const.

LLVM does have logic to canonicalize a triple. If you do that early on in the Driver, does everything work?

@jrose-apple

jrose-apple Feb 10, 2016

Member

I'm a little concerned about this function mutating Triple. It should probably be passed const.

LLVM does have logic to canonicalize a triple. If you do that early on in the Driver, does everything work?

test/Driver/sdk.swift
+// LINUX: bin/{{.+}} {{.*}}swift_begin.o
+// LINUX: {{-syslibroot|--sysroot}} {{.*}}/Inputs/clang-importer-sdk
+// LINUX: {{.*}}swift_end.o
+// LINUX-ENV: bin/swift

This comment has been minimized.

@jrose-apple

jrose-apple Feb 10, 2016

Member

Please use the same check lines for the -sdk and env tests. The main point of this test is that they should have identical behavior.

@jrose-apple

jrose-apple Feb 10, 2016

Member

Please use the same check lines for the -sdk and env tests. The main point of this test is that they should have identical behavior.

This comment has been minimized.

@hpux735

hpux735 Feb 14, 2016

Contributor

@jrose-apple I think I've finally got this test right. Let me know if you have any remaining concerns about it.

@hpux735

hpux735 Feb 14, 2016

Contributor

@jrose-apple I think I've finally got this test right. Let me know if you have any remaining concerns about it.

test/lit.cfg
@@ -473,7 +473,7 @@ if platform.system() == 'Darwin' and (run_os == 'macosx' or run_os == 'darwin'):
# <rdar://problem/16678410> Adding more libraries with -lfoo to REPL is broken
config.available_features.add('swift_repl')
config.available_features.add('swift_interpreter')
-elif platform.system() == 'Linux':
+elif platform.system() == 'Linux':

This comment has been minimized.

@jrose-apple

jrose-apple Feb 10, 2016

Member

Spurious whitespace change here?

@jrose-apple

jrose-apple Feb 10, 2016

Member

Spurious whitespace change here?

+ // libraries. LLVM matches these SubArchTypes to the generic
+ // ARMSubArch_v7 (for example) type. If that is the case,
+ // use the base of the architecture type in the library path.
+ if (Triple.isOSLinux()) {

This comment has been minimized.

@hpux735

hpux735 Feb 10, 2016

Contributor

@jrose-apple Your comment got lost again in the I've pasted it here from my email:

I'm a little concerned about this function mutating Triple. It should probably be passed const.

LLVM does have logic to canonicalize a triple. If you do that early on in the Driver, does everything work?

And I think that's a good point. I refactored it a bit, and I think that this solution really gets at the essence of what I'm trying to achieve here. The goal is to get the library path using the architecture type. In many other places in the project, "armv7" and "armv6" are used for this path according to the architecture type (not the least of which are the build and install scripts). It probably makes sense to decouple the assumption that llvm uses the same string to achieve this, or the unnecessary setting of llvm::Triple's string to match. I think it makes more sense to ask llvm for the architecture we're using, then use that to select the explicit string that we're really looking for, and is hard-coded elsewhere.

With regard to the normalization logic, I did some experiments with that. It preserves the 'l' provided by linux, so it won't do what we want here.

@hpux735

hpux735 Feb 10, 2016

Contributor

@jrose-apple Your comment got lost again in the I've pasted it here from my email:

I'm a little concerned about this function mutating Triple. It should probably be passed const.

LLVM does have logic to canonicalize a triple. If you do that early on in the Driver, does everything work?

And I think that's a good point. I refactored it a bit, and I think that this solution really gets at the essence of what I'm trying to achieve here. The goal is to get the library path using the architecture type. In many other places in the project, "armv7" and "armv6" are used for this path according to the architecture type (not the least of which are the build and install scripts). It probably makes sense to decouple the assumption that llvm uses the same string to achieve this, or the unnecessary setting of llvm::Triple's string to match. I think it makes more sense to ask llvm for the architecture we're using, then use that to select the explicit string that we're really looking for, and is hard-coded elsewhere.

With regard to the normalization logic, I did some experiments with that. It preserves the 'l' provided by linux, so it won't do what we want here.

This comment has been minimized.

@jrose-apple

jrose-apple Feb 10, 2016

Member

Thanks for looking into this further. This version looks good to me.

@jrose-apple

jrose-apple Feb 10, 2016

Member

Thanks for looking into this further. This version looks good to me.

This comment has been minimized.

@jrose-apple

jrose-apple Feb 10, 2016

Member

…although another possible answer would be to actually use armv7l as the name of the standard library, in case you ever wanted it to coexist with an armv7b.

@jrose-apple

jrose-apple Feb 10, 2016

Member

…although another possible answer would be to actually use armv7l as the name of the standard library, in case you ever wanted it to coexist with an armv7b.

This comment has been minimized.

@hpux735

hpux735 Feb 10, 2016

Contributor

Hmmm... interesting point. Thoughts @tienex ?

@hpux735

hpux735 Feb 10, 2016

Contributor

Hmmm... interesting point. Thoughts @tienex ?

This comment has been minimized.

@tienex

tienex Feb 10, 2016

Contributor

I still think we'll need to normalize somehow... assuming we want it to behave like clang, we should know how to translate armv7- (with no endian notion) to armv7([bl])-, the same happens for thumbv7([bl])?- which should use armv7 libraries (which effectively should be the default on linux/armv7 being the preferred ISA).

@tienex

tienex Feb 10, 2016

Contributor

I still think we'll need to normalize somehow... assuming we want it to behave like clang, we should know how to translate armv7- (with no endian notion) to armv7([bl])-, the same happens for thumbv7([bl])?- which should use armv7 libraries (which effectively should be the default on linux/armv7 being the preferred ISA).

This comment has been minimized.

@hpux735

hpux735 Feb 10, 2016

Contributor

Because the fact that there are still many showstopper issues with big endian (which were brought up in the discussion of the c-based implementation of the conformance objects) it may not make sense to put the cart before the horse on this one. I doubt this would pose undue complication to a future effort supporting big-endian arm cores.

@hpux735

hpux735 Feb 10, 2016

Contributor

Because the fact that there are still many showstopper issues with big endian (which were brought up in the discussion of the c-based implementation of the conformance objects) it may not make sense to put the cart before the horse on this one. I doubt this would pose undue complication to a future effort supporting big-endian arm cores.

This comment has been minimized.

cmake/modules/AddSwift.cmake
+ if("${libkind}" STREQUAL "SHARED")
+ set(arch_subdir "${SWIFTLIB_DIR}/${SWIFTLIB_SINGLE_SUBDIR}")
+
+ add_custom_command_target(conformance_object_begin

This comment has been minimized.

@hpux735

hpux735 Feb 12, 2016

Contributor

@gribozavr I think this addresses your comment about add_custom_command_target. Is this what you had in mind?

@hpux735

hpux735 Feb 12, 2016

Contributor

@gribozavr I think this addresses your comment about add_custom_command_target. Is this what you had in mind?

This comment has been minimized.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

Thank you, it partially does. This will definitely improve robustness, but we can do better. Now that you have a proper CMake target that you can make other targets depend on, you don't need to execute this command (adding a target) for every library, you can do it once at the toplevel, and depend on conformance_object_begin everywhere.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

Thank you, it partially does. This will definitely improve robustness, but we can do better. Now that you have a proper CMake target that you can make other targets depend on, you don't need to execute this command (adding a target) for every library, you can do it once at the toplevel, and depend on conformance_object_begin everywhere.

This comment has been minimized.

@hpux735

hpux735 Feb 12, 2016

Contributor

I think that would hurt the generality of this approach. Because add_swift_library_single accepts a triple, it should be assumed that the triple could change between invocations. If this logic was factored out of the function, it's possible that the conformance objects from one target would be linked against a shared library from another.

@hpux735

hpux735 Feb 12, 2016

Contributor

I think that would hurt the generality of this approach. Because add_swift_library_single accepts a triple, it should be assumed that the triple could change between invocations. If this logic was factored out of the function, it's possible that the conformance objects from one target would be linked against a shared library from another.

This comment has been minimized.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

You're right, we should have one for every triple. But not for every library.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

You're right, we should have one for every triple. But not for every library.

This comment has been minimized.

@tienex

tienex Feb 12, 2016

Contributor

I have a proposal, but I am not sure it will work but we can give it a try, the issue is currently this, from what I could gather reading the cmake scripts:

  • there's a cycle dependency at source level; this file is being included after AddSwift, cmake can't the objects as such we need to work around it adding the extra add_custom_command to copy the objects.
  • there's a dependency inversion, because the objects are effectively built before the add_swift_library is invoked.

The idea is to move the generation of the objects in a separate cmake file, since it is self contained, it could be included in the top level cmake file before AddSwift, at that point cmake should be able to match the dependencies and we can remove the extra copy step. Any thoughts?

@tienex

tienex Feb 12, 2016

Contributor

I have a proposal, but I am not sure it will work but we can give it a try, the issue is currently this, from what I could gather reading the cmake scripts:

  • there's a cycle dependency at source level; this file is being included after AddSwift, cmake can't the objects as such we need to work around it adding the extra add_custom_command to copy the objects.
  • there's a dependency inversion, because the objects are effectively built before the add_swift_library is invoked.

The idea is to move the generation of the objects in a separate cmake file, since it is self contained, it could be included in the top level cmake file before AddSwift, at that point cmake should be able to match the dependencies and we can remove the extra copy step. Any thoughts?

This comment has been minimized.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

Sounds good to me.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

Sounds good to me.

stdlib/public/runtime/CMakeLists.txt
+ add_custom_command(
+ OUTPUT "${SWIFTLIB_DIR}/${arch_subdir}/swift_end.o"
+ COMMAND "${CMAKE_COMMAND}" -E copy "${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${conformance_magic_end_name}.dir/swift_sections.S${CMAKE_C_OUTPUT_EXTENSION}" "${SWIFTLIB_DIR}/${arch_subdir}/swift_end.o"
+ DEPENDS ${conformance_magic_end_name})

This comment has been minimized.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

And, once you have a global target you can depend on, you can remove these extra custom commands, too, and just depend on that global target.

@gribozavr

gribozavr Feb 12, 2016

Collaborator

And, once you have a global target you can depend on, you can remove these extra custom commands, too, and just depend on that global target.

This comment has been minimized.

@hpux735

hpux735 Feb 14, 2016

Contributor

@gribozavr I think the current solution will be much more acceptable. Let me know if you have any more concerns. :)

@hpux735

hpux735 Feb 14, 2016

Contributor

@gribozavr I think the current solution will be much more acceptable. Let me know if you have any more concerns. :)

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 14, 2016

Collaborator

@hpux735 The new CMake code is much nicer and cleaner, thank you!

Collaborator

gribozavr commented Feb 14, 2016

@hpux735 The new CMake code is much nicer and cleaner, thank you!

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 14, 2016

Collaborator

@swift-ci Please test

Collaborator

gribozavr commented Feb 14, 2016

@swift-ci Please test

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 14, 2016

Contributor

@gribozavr Thanks! There's a companion PR for foundation apple/swift-corelibs-foundation#249

Contributor

hpux735 commented Feb 14, 2016

@gribozavr Thanks! There's a companion PR for foundation apple/swift-corelibs-foundation#249

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 14, 2016

Contributor

I reproduced the failure on Darwin, and fixed it.

Contributor

hpux735 commented Feb 14, 2016

I reproduced the failure on Darwin, and fixed it.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 14, 2016

Collaborator

@swift-ci Please test

Collaborator

gribozavr commented Feb 14, 2016

@swift-ci Please test

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 14, 2016

Collaborator

@jrose-apple Does everything look good from your side?

Collaborator

gribozavr commented Feb 14, 2016

@jrose-apple Does everything look good from your side?

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 14, 2016

Collaborator

@hpux735 I have ran the tests on Linux/x86_64, and it looks like we would also need a companion PR at least for xctest, and possibly the package manager and possibly lldb (for the JIT'ed code).

You can reproduce the buildbot configuration with:

./utils/build-script --preset=buildbot_linux_1404 install_destdir=/tmp/install installable_package=/tmp/swift.tar.gz
Collaborator

gribozavr commented Feb 14, 2016

@hpux735 I have ran the tests on Linux/x86_64, and it looks like we would also need a companion PR at least for xctest, and possibly the package manager and possibly lldb (for the JIT'ed code).

You can reproduce the buildbot configuration with:

./utils/build-script --preset=buildbot_linux_1404 install_destdir=/tmp/install installable_package=/tmp/swift.tar.gz
@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 15, 2016

Contributor

Thanks for spending the time looking into it, @gribozavr . I tried to reproduce your issues with both my branch and the apple/master branch. The errors pasted below occur on the apple master branches as well as mine. Here were my steps to verify (my branch is the only provided example, the apple branch is just using update-checkout --clone as is):

  477  rm -rf *
  478  git clone https://github.com/hpux735/swift.git
  479  cd swift/
  480  ./utils/update-checkout --clone
  481  cd ..
  482  rm -rf swift-corelibs-foundation/
  483  git clone https://github.com/hpux735/swift-corelibs-foundation.git
  484  cd swift-corelibs-foundation/
  485  git checkout gold
  486  cd ..
  487  cd swift
  488  git checkout gold
  489  ./utils/build-script --preset=buildbot_linux_1404 install_destdir=/tmp/install installable_package=/tmp/swift.tar.gz
  490  history

The results from testing are these two errors which seem to be lit being braindead, and have been present for quite a long time.:

******************** TEST 'Swift :: Driver/environment.swift' FAILED ********************
Script:
--
env SDKROOT= '/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheBaTruO' -target x86_64-unknown-gnu-linux -L/foo/ -driver-use-frontend-path /home/wdillon/swift-build/swift/test/Driver/Inputs/print-var.sh /home/wdillon/swift-build/swift/test/Driver/environment.swift LD_LIBRARY_PATH | FileCheck /home/wdillon/swift-build/swift/test/Driver/environment.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/wdillon/swift-build/swift/test/Driver/environment.swift:5:11: error: expected string not found in input
// CHECK: {{^/foo/:[^:]+/lib/swift/linux$}}
          ^
<stdin>:1:1: note: scanning from here
/foo/:/home/wdillon/swift-build/swift/test/Driver/lib/swift/linux:/usr/local/cuda-7.0/lib64:
^
<stdin>:1:36: note: possible intended match here
/foo/:/home/wdillon/swift-build/swift/test/Driver/lib/swift/linux:/usr/local/cuda-7.0/lib64:
********************
FAIL: Swift :: Driver/options-interpreter.swift (463 of 7952)
******************** TEST 'Swift :: Driver/options-interpreter.swift' FAILED ********************
Script:
--
not env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -deprecated-integrated-repl -emit-module 2>&1 | FileCheck -check-prefix=IMMEDIATE_NO_MODULE /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
not env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -emit-module 2>&1 | FileCheck -check-prefix=IMMEDIATE_NO_MODULE /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix INTERPRET /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift a b c | FileCheck -check-prefix ARGS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -parse-stdlib /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix PARSE_STDLIB /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -parse-stdlib | FileCheck -check-prefix PARSE_STDLIB /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -resource-dir /RSRC/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-RESOURCE-DIR-ONLY /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-unknown-linux-gnu -resource-dir /RSRC/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-RESOURCE-DIR-ONLY-LINUX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -L/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L2 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_LIBRARY_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L2-ENV /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-NO-FRAMEWORKS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-NO-FRAMEWORKS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -F/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F2 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F2-ENV /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ -L/foo2/ -L/bar2/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-COMPLEX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-unknown-linux-gnu -L/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L-LINUX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env LD_LIBRARY_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-unknown-linux-gnu -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-LINUX-COMPLEX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift:21:35: error: expected string not found in input
// CHECK-RESOURCE-DIR-ONLY-LINUX: # LD_LIBRARY_PATH=/RSRC/linux{{$}}
                                  ^
<stdin>:1:1: note: scanning from here
/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift -frontend -interpret /home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -module-cache-path /tmp/swift-testsuite-clang-module-cacheBaTruO -resource-dir /RSRC/ -module-name main # LD_LIBRARY_PATH=/RSRC/linux:/usr/local/cuda-7.0/lib64:
^
<stdin>:1:327: note: possible intended match here
/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift -frontend -interpret /home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -module-cache-path /tmp/swift-testsuite-clang-module-cacheBaTruO -resource-dir /RSRC/ -module-name main # LD_LIBRARY_PATH=/RSRC/linux:/usr/local/cuda-7.0/lib64:

Looking at them with my (admittedly very permissive) expression parser, I don't understand why these tests fail in the first place.

Contributor

hpux735 commented Feb 15, 2016

Thanks for spending the time looking into it, @gribozavr . I tried to reproduce your issues with both my branch and the apple/master branch. The errors pasted below occur on the apple master branches as well as mine. Here were my steps to verify (my branch is the only provided example, the apple branch is just using update-checkout --clone as is):

  477  rm -rf *
  478  git clone https://github.com/hpux735/swift.git
  479  cd swift/
  480  ./utils/update-checkout --clone
  481  cd ..
  482  rm -rf swift-corelibs-foundation/
  483  git clone https://github.com/hpux735/swift-corelibs-foundation.git
  484  cd swift-corelibs-foundation/
  485  git checkout gold
  486  cd ..
  487  cd swift
  488  git checkout gold
  489  ./utils/build-script --preset=buildbot_linux_1404 install_destdir=/tmp/install installable_package=/tmp/swift.tar.gz
  490  history

The results from testing are these two errors which seem to be lit being braindead, and have been present for quite a long time.:

******************** TEST 'Swift :: Driver/environment.swift' FAILED ********************
Script:
--
env SDKROOT= '/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheBaTruO' -target x86_64-unknown-gnu-linux -L/foo/ -driver-use-frontend-path /home/wdillon/swift-build/swift/test/Driver/Inputs/print-var.sh /home/wdillon/swift-build/swift/test/Driver/environment.swift LD_LIBRARY_PATH | FileCheck /home/wdillon/swift-build/swift/test/Driver/environment.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/wdillon/swift-build/swift/test/Driver/environment.swift:5:11: error: expected string not found in input
// CHECK: {{^/foo/:[^:]+/lib/swift/linux$}}
          ^
<stdin>:1:1: note: scanning from here
/foo/:/home/wdillon/swift-build/swift/test/Driver/lib/swift/linux:/usr/local/cuda-7.0/lib64:
^
<stdin>:1:36: note: possible intended match here
/foo/:/home/wdillon/swift-build/swift/test/Driver/lib/swift/linux:/usr/local/cuda-7.0/lib64:
********************
FAIL: Swift :: Driver/options-interpreter.swift (463 of 7952)
******************** TEST 'Swift :: Driver/options-interpreter.swift' FAILED ********************
Script:
--
not env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -deprecated-integrated-repl -emit-module 2>&1 | FileCheck -check-prefix=IMMEDIATE_NO_MODULE /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
not env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -emit-module 2>&1 | FileCheck -check-prefix=IMMEDIATE_NO_MODULE /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix INTERPRET /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift a b c | FileCheck -check-prefix ARGS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -parse-stdlib /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix PARSE_STDLIB /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -parse-stdlib | FileCheck -check-prefix PARSE_STDLIB /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -resource-dir /RSRC/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-RESOURCE-DIR-ONLY /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-unknown-linux-gnu -resource-dir /RSRC/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-RESOURCE-DIR-ONLY-LINUX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -L/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L2 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_LIBRARY_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L2-ENV /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-NO-FRAMEWORKS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-NO-FRAMEWORKS /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -F/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F2 /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-F2-ENV /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env DYLD_FRAMEWORK_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-apple-macosx10.9 -F/foo/ -F/bar/ -L/foo2/ -L/bar2/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-COMPLEX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env SDKROOT= '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -module-cache-path '/tmp/swift-testsuite-clang-module-cacheAaHAPZ' -### -target x86_64-unknown-linux-gnu -L/foo/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-L-LINUX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
env LD_LIBRARY_PATH=/abc/ '/home/wdillon/swift-generic/build/buildbot_linux/swift-linux-x86_64/bin/swift' -### -target x86_64-unknown-linux-gnu -L/foo/ -L/bar/ /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift | FileCheck -check-prefix=CHECK-LINUX-COMPLEX /home/wdillon/swift-generic/swift/test/Driver/options-interpreter.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift:21:35: error: expected string not found in input
// CHECK-RESOURCE-DIR-ONLY-LINUX: # LD_LIBRARY_PATH=/RSRC/linux{{$}}
                                  ^
<stdin>:1:1: note: scanning from here
/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift -frontend -interpret /home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -module-cache-path /tmp/swift-testsuite-clang-module-cacheBaTruO -resource-dir /RSRC/ -module-name main # LD_LIBRARY_PATH=/RSRC/linux:/usr/local/cuda-7.0/lib64:
^
<stdin>:1:327: note: possible intended match here
/home/wdillon/swift-build/build/buildbot_linux/swift-linux-x86_64/bin/swift -frontend -interpret /home/wdillon/swift-build/swift/test/Driver/options-interpreter.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -module-cache-path /tmp/swift-testsuite-clang-module-cacheBaTruO -resource-dir /RSRC/ -module-name main # LD_LIBRARY_PATH=/RSRC/linux:/usr/local/cuda-7.0/lib64:

Looking at them with my (admittedly very permissive) expression parser, I don't understand why these tests fail in the first place.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 15, 2016

Collaborator

@hpux735 I think these issues are related to your local configuration. Something is adding /usr/local/cuda-7.0/lib64 to your paths.

Collaborator

gribozavr commented Feb 15, 2016

@hpux735 I think these issues are related to your local configuration. Something is adding /usr/local/cuda-7.0/lib64 to your paths.

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 15, 2016

Contributor

Thanks @gribozavr . I was able to reproduce what I think was the error you're referencing:

Test Case 'ManifestTests.testManifestLoading' started.
fatal error: Unexpected TOMLItem: file /home/wdillon/swift-gold/swiftpm/Sources/dep/Manifest.swift, line 78
Current stack trace:
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(swift_reportError+0x8b) [0x7f7f12a3f58b]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(_swift_stdlib_reportFatalErrorInFile+0x67) [0x7f7f12a50277]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(+0x27296c) [0x7f7f129a096c]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(+0x272cab) [0x7f7f129a0cab]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(function signature specialization <Arg[0] = Exploded, Arg[1] = Exploded, Arg[2] = Exploded> of Swift._assertionFailed (Swift.StaticString, Swift.String, Swift.StaticString, Swift.UInt, flags : Swift.UInt32) -> ()+0x75) [0x7f7f129a0df5]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a883a]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a5572]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a9b6c]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f300]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f589]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x45130a]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x40d698]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f179]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x16083) [0x7f7f12522083]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x16c2e) [0x7f7f12522c2e]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x154b5) [0x7f7f125214b5]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x15f9f) [0x7f7f12521f9f]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so((extension in XCTest):XCTest.XCTestCase.invokeTest () -> ()+0x150) [0x7f7f12520780]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x180a7) [0x7f7f125240a7]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(XCTest.XCTMain (Swift.Array<XCTest.XCTestCase>) -> ()+0x5b) [0x7f7f125236db]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x495c7c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f11323ec5]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x40c979]
bootstrap: error: tests failed with exit status -4
./utils/build-script: command terminated with a non-zero exit status 1, aborting
./utils/build-script: command terminated with a non-zero exit status 1, aborting

I'll look into it.

With regard to the other two tests that were failing, it's probably not great that environment-defined variables can break them.

Contributor

hpux735 commented Feb 15, 2016

Thanks @gribozavr . I was able to reproduce what I think was the error you're referencing:

Test Case 'ManifestTests.testManifestLoading' started.
fatal error: Unexpected TOMLItem: file /home/wdillon/swift-gold/swiftpm/Sources/dep/Manifest.swift, line 78
Current stack trace:
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(swift_reportError+0x8b) [0x7f7f12a3f58b]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(_swift_stdlib_reportFatalErrorInFile+0x67) [0x7f7f12a50277]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(+0x27296c) [0x7f7f129a096c]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(+0x272cab) [0x7f7f129a0cab]
/home/wdillon/swift-gold/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftCore.so(function signature specialization <Arg[0] = Exploded, Arg[1] = Exploded, Arg[2] = Exploded> of Swift._assertionFailed (Swift.StaticString, Swift.String, Swift.StaticString, Swift.UInt, flags : Swift.UInt32) -> ()+0x75) [0x7f7f129a0df5]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a883a]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a5572]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x4a9b6c]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f300]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f589]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x45130a]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x40d698]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x44f179]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x16083) [0x7f7f12522083]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x16c2e) [0x7f7f12522c2e]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x154b5) [0x7f7f125214b5]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x15f9f) [0x7f7f12521f9f]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so((extension in XCTest):XCTest.XCTestCase.invokeTest () -> ()+0x150) [0x7f7f12520780]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x180a7) [0x7f7f125240a7]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(+0x14cba) [0x7f7f12520cba]
/home/wdillon/swift-gold/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so(XCTest.XCTMain (Swift.Array<XCTest.XCTestCase>) -> ()+0x5b) [0x7f7f125236db]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x495c7c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f11323ec5]
/home/wdillon/swift-gold/build/buildbot_linux/swiftpm-linux-x86_64/.bootstrap/bin/dep-tests() [0x40c979]
bootstrap: error: tests failed with exit status -4
./utils/build-script: command terminated with a non-zero exit status 1, aborting
./utils/build-script: command terminated with a non-zero exit status 1, aborting

I'll look into it.

With regard to the other two tests that were failing, it's probably not great that environment-defined variables can break them.

@jrose-apple

This comment has been minimized.

Show comment
Hide comment
@jrose-apple

jrose-apple Feb 15, 2016

Member

@gribozavr My comments were all addressed, LGTM. :-)

Member

jrose-apple commented Feb 15, 2016

@gribozavr My comments were all addressed, LGTM. :-)

@jrose-apple

This comment has been minimized.

Show comment
Hide comment
@jrose-apple

jrose-apple Feb 15, 2016

Member

@hpux735 Lit filters out most environment variables but it can't filter out PATH because it expects to be able to use things like "cp" and "ls" unqualified. I'm not sure what to do about LD_LIBRARY_PATH because it's possible some systems depend on it just to run things. We could either:

  • take it out of the environment variable whitelist, or
  • update this test to use env -i on every line.

But either way we can do that in a separate PR.

Member

jrose-apple commented Feb 15, 2016

@hpux735 Lit filters out most environment variables but it can't filter out PATH because it expects to be able to use things like "cp" and "ls" unqualified. I'm not sure what to do about LD_LIBRARY_PATH because it's possible some systems depend on it just to run things. We could either:

  • take it out of the environment variable whitelist, or
  • update this test to use env -i on every line.

But either way we can do that in a separate PR.

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 15, 2016

Contributor

@jrose-apple Thanks! I agree that it's inappropriate for this PR 👍

Contributor

hpux735 commented Feb 15, 2016

@jrose-apple Thanks! I agree that it's inappropriate for this PR 👍

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 18, 2016

Contributor

@gribozavr would you mind taking a look at the changes in the other PRs and the current version of this. I think we've addressed your comments, but there is some new weirdness with the Evolution validation tests. These are new since the last time we exchanged about this, and I don't know if they're supposed to fail. Either way, they're preventing the buildbot from finishing. In my testing, I deleted the tests and everything went though fine.

Contributor

hpux735 commented Feb 18, 2016

@gribozavr would you mind taking a look at the changes in the other PRs and the current version of this. I think we've addressed your comments, but there is some new weirdness with the Evolution validation tests. These are new since the last time we exchanged about this, and I don't know if they're supposed to fail. Either way, they're preventing the buildbot from finishing. In my testing, I deleted the tests and everything went though fine.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 18, 2016

Collaborator

CC @slavapestov for Evolution tests.

Collaborator

gribozavr commented Feb 18, 2016

CC @slavapestov for Evolution tests.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 18, 2016

Collaborator

I'm running tests on Ubuntu 14.04.

Collaborator

gribozavr commented Feb 18, 2016

I'm running tests on Ubuntu 14.04.

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 18, 2016

Contributor

Oops! Sorry @slavapestov I just saw that I was a couple hundred commits behind master. I rebased and ran it again and the evolution tests pass. Also, @gribozavr , the patch to swift package manager was just merged in (I works with or without this PR).

Anyway, I just re-tested with your build script, and all the tests pass. The only exception is swift-package-tests :: repl/test-repl-glibc.py post-install.

Contributor

hpux735 commented Feb 18, 2016

Oops! Sorry @slavapestov I just saw that I was a couple hundred commits behind master. I rebased and ran it again and the evolution tests pass. Also, @gribozavr , the patch to swift package manager was just merged in (I works with or without this PR).

Anyway, I just re-tested with your build script, and all the tests pass. The only exception is swift-package-tests :: repl/test-repl-glibc.py post-install.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 18, 2016

Collaborator

I tested a combination of:

on Ubuntu 14.04 x86_64 and found no issues. I couldn't reproduce swift-package-tests :: repl/test-repl-glibc.py.

Merging.

Collaborator

gribozavr commented Feb 18, 2016

I tested a combination of:

on Ubuntu 14.04 x86_64 and found no issues. I couldn't reproduce swift-package-tests :: repl/test-repl-glibc.py.

Merging.

gribozavr added a commit that referenced this pull request Feb 18, 2016

Merge pull request #1157 from hpux735/gold
Discard swift.ld in favor of portable solution and support gold linker in Swift

@gribozavr gribozavr merged commit 220a2b9 into apple:master Feb 18, 2016

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 18, 2016

Contributor

Wow. I want to sincerely thank @tienex @gribozavr @compnerd @modocache @jckarter and @jrose-apple for all your help! This was one heck of a PR. As far as I can tell, it has the most comments of any non-joke PR.

Contributor

hpux735 commented Feb 18, 2016

Wow. I want to sincerely thank @tienex @gribozavr @compnerd @modocache @jckarter and @jrose-apple for all your help! This was one heck of a PR. As far as I can tell, it has the most comments of any non-joke PR.

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 18, 2016

Collaborator

@hpux735 This is huge, thank you! Could you add an item to the changelog?

Collaborator

gribozavr commented Feb 18, 2016

@hpux735 This is huge, thank you! Could you add an item to the changelog?

@hpux735

This comment has been minimized.

Show comment
Hide comment
@hpux735

hpux735 Feb 18, 2016

Contributor

@gribozavr Of course, under the Swift3 header?

Contributor

hpux735 commented Feb 18, 2016

@gribozavr Of course, under the Swift3 header?

@gribozavr

This comment has been minimized.

Show comment
Hide comment
@gribozavr

gribozavr Feb 18, 2016

Collaborator

@hpux735 Yes.

Collaborator

gribozavr commented Feb 18, 2016

@hpux735 Yes.

@hpux735 hpux735 deleted the hpux735:gold branch Feb 18, 2016

@modocache modocache referenced this pull request Mar 7, 2016

Closed

Enable cross-compiling #1398

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment