Skip to content

Conversation

marcrasi
Copy link

Having Array.DifferentiableView only defined when Element : Differentiable makes the compiler quite confused, leading to issues like TF-489, TF-934.

This PR works around the issue by making Array.DifferentiableView unconstrained.

I reduced the problem to something that reproduces on master, and filed an issue for that: SR-11685. Once that issue is resolved, we can try reconstraining it.

@marcrasi marcrasi requested a review from dan-zheng October 30, 2019 21:30
@marcrasi
Copy link
Author

@swift-ci please test tensorflow

Copy link
Contributor

@dan-zheng dan-zheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

EDIT: It seems Array.DifferentiableView extensions in tensorflow/swift-apis also require where Element : Differentiable constraints.

@marcrasi
Copy link
Author

Those test failures remind me that this is a breaking API change for clients: any code assuming that Array.DifferentiableView implies Element: Differentiable will no longer compile.

I think it's an acceptable API breakage because:

  1. It's the only way to get around build breakage.
  2. It's simple to fix in client code (just add an explicit Element: Differentiable constraint).

I'll run our full toolchain build and downstream client tests on this to make sure we're not unexpectedly breaking anything.

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

6 similar comments
@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

This latest compile error should be simple to fix, but first I'll reproduce it on my local machine so that I'm not just going back and forth for a long time with CI. I expect to send a PR to swift-apis fixing it.

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

3 similar comments
@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow

@marcrasi
Copy link
Author

@swift-ci please test tensorflow macos

@marcrasi
Copy link
Author

Odd, we've got some test failures in seemingly-unrelated tests: SourceKit/InterfaceGen.gen_swift_type.swift & IDE.print_type_interface.swift.

Also a test failure in ParseableInterface.verify_all_overlays.py, but that's not very surprising because I touched that one.

And my full toolchain build reveals that this also causes a swift-apis compile crash:

Step 17/30 : RUN /swift-tensorflow-toolchain/usr/bin/swift test
 ---> Running in 1a511890753d
[1/57] Compiling TensorFlow Optimizer.swift
[2/57] Compiling TensorFlow TensorShape.swift
[3/57] Compiling TensorFlow Threading.swift
[4/57] Compiling TensorFlow TensorHandle.swift
[5/57] Compiling TensorFlow TensorProtocol.swift
[6/57] Compiling TensorFlow Execution.swift
[7/57] Compiling TensorFlow LazyTensorContext.swift
[8/57] Compiling TensorFlow StdlibExtensions.swift
[9/57] Compiling TensorFlow ShapedArray.swift
[10/57] Compiling TensorFlow StringTensor.swift
[11/57] Compiling TensorFlow Runtime.swift
[12/57] Compiling TensorFlow Serialization.swift
[13/57] Compiling TensorFlow SGD.swift
[14/57] Compiling TensorFlow LazyTensorTFFunctionBuilder.swift
[15/57] Compiling TensorFlow LazyTensorTrace.swift
[16/57] Compiling TensorFlow Context.swift
[17/57] Compiling TensorFlow ArrayOps.swift
[18/57] Compiling TensorFlow Random.swift
[19/57] Compiling TensorFlow Comparison.swift
[20/57] Compiling TensorFlow BackwardsCompatibility.swift
[21/58] Compiling TensorFlow Dataset.swift
[22/58] Merging module Experimental
[23/60] Wrapping AST for Experimental for debugging
[24/60] Compiling ExperimentalTests ComplexTests.swift
[25/60] Compiling ExperimentalTests XCTestManifests.swift
[26/60] Compiling TensorFlow DataTypes.swift
[27/60] Compiling TensorFlow DifferentialOperators.swift
[28/60] Compiling TensorFlow LazyTensorTraceCache.swift
[29/60] Compiling TensorFlow PythonConversion.swift
[30/60] Compiling TensorFlow Sequential.swift
[31/60] Compiling TensorFlow Tensor.swift
[32/60] Compiling TensorFlow TensorGroup.swift
[33/60] Compiling TensorFlow LazyTensorOperation.swift
[34/60] Compiling TensorFlow LazyTensorShapeInference.swift
[37/61] Merging module ExperimentalTests
[38/61] Wrapping AST for ExperimentalTests for debugging
[39/61] Compiling TensorFlow NN.swift
[40/61] Compiling TensorFlow Upsampling.swift
[41/61] Compiling TensorFlow MomentumBased.swift
[42/61] Compiling TensorFlow Utilities.swift
[43/61] Compiling TensorFlow Initializers.swift
[44/61] Compiling TensorFlow Initialization.swift
[45/61] Compiling TensorFlow Normalization.swift
[46/61] Compiling TensorFlow Recurrent.swift
[47/61] Compiling TensorFlow Basic.swift
[48/61] Compiling TensorFlow RawOpsGenerated.swift
[49/61] Compiling TensorFlow TFTensorOperation.swift
[50/61] Compiling TensorFlow Core.swift
[51/61] Compiling TensorFlow Embedding.swift
[52/61] Compiling TensorFlow Loss.swift
[53/61] Compiling TensorFlow Pooling.swift
[54/61] Compiling TensorFlow Layer.swift
[55/61] Compiling TensorFlow Convolutional.swift
[56/61] Compiling TensorFlow Math.swift
swift: /swift-base/swift/lib/Serialization/Serialization.cpp:2313: void swift::serialization::Serializer::DeclSerializer::writeDeclAttribute(const swift::DeclAttribute *): Assertion `attr->getOriginalFunction() && "@differentiable attribute must have original function resolved"' failed.
Stack dump:
0.	Program arguments: /swift-tensorflow-toolchain/usr/bin/swift -frontend -merge-modules -emit-module /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/BackwardsCompatibility~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Bindings/EagerExecution~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Bindings/RawOpsGenerated~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Bindings/TFTensorOperation~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Context~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/ArrayOps~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/DataTypes~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/DifferentialOperators~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Execution~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorContext~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorOperation~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorShapeInference~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorTFFunctionBuilder~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorTrace~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/LazyTensorTraceCache~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/PythonConversion~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Runtime~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Serialization~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/ShapedArray~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/StringTensor~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Tensor~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/TensorGroup~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/TensorHandle~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/TensorProtocol~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/TensorShape~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Threading~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Core/Utilities~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Initializers~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layer~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Convolutional~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Core~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Embedding~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Initialization~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Normalization~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Pooling~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Recurrent~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Sequential~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Layers/Upsampling~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Loss~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Operators/Basic~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Operators/Comparison~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Operators/Dataset~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Operators/Math~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Operators/NN~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Optimizers/MomentumBased~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Optimizers/Optimizer~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Optimizers/SGD~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/Random~partial.swiftmodule /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.build/StdlibExtensions~partial.swiftmodule -parse-as-library -sil-merge-partial-modules -disable-diagnostic-passes -disable-sil-perf-optzns -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift-apis/.build/x86_64-unknown-linux-gnu/debug -enable-testing -g -module-cache-path /swift-apis/.build/x86_64-unknown-linux-gnu/debug/ModuleCache -swift-version 4.2 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -emit-module-doc-path /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.swiftdoc -emit-module-source-info-path /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.swiftsourceinfo -module-name TensorFlow -o /swift-apis/.build/x86_64-unknown-linux-gnu/debug/TensorFlow.swiftmodule
1.	Swift version 5.1.1-dev (LLVM 6e04008c7f, Swift 4df03077b6)
2.	While running pass #6054 SILModuleTransform "SerializeSILPass".
3.	While serializing 'layer1' (in module 'TensorFlow')
/swift-tensorflow-toolchain/usr/bin/swift[0x4c27934]
/swift-tensorflow-toolchain/usr/bin/swift[0x4c2554e]
/swift-tensorflow-toolchain/usr/bin/swift[0x4c27be8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f7fc3df0890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f7fc2253e97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f7fc2255801]
/lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7f7fc224539a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7f7fc2245412]
/swift-tensorflow-toolchain/usr/bin/swift[0x11e10f7]
/swift-tensorflow-toolchain/usr/bin/swift[0x11d3a5b]
/swift-tensorflow-toolchain/usr/bin/swift[0x11d4cab]
/swift-tensorflow-toolchain/usr/bin/swift[0x11d60d8]
/swift-tensorflow-toolchain/usr/bin/swift[0x11dd9b8]
/swift-tensorflow-toolchain/usr/bin/swift[0x1232a9b]
/swift-tensorflow-toolchain/usr/bin/swift[0x4abeae]
/swift-tensorflow-toolchain/usr/bin/swift[0x178ee11]
/swift-tensorflow-toolchain/usr/bin/swift[0x4a8856]
/swift-tensorflow-toolchain/usr/bin/swift[0x11df1f3]
/swift-tensorflow-toolchain/usr/bin/swift[0x4e1679]
/swift-tensorflow-toolchain/usr/bin/swift[0xebb1a4]
/swift-tensorflow-toolchain/usr/bin/swift[0xac114d]
/swift-tensorflow-toolchain/usr/bin/swift[0x980bef]
/swift-tensorflow-toolchain/usr/bin/swift[0x9815e1]
/swift-tensorflow-toolchain/usr/bin/swift[0x555298]
/swift-tensorflow-toolchain/usr/bin/swift[0x9899c5]
/swift-tensorflow-toolchain/usr/bin/swift[0x7579ab]
/swift-tensorflow-toolchain/usr/bin/swift[0x4e0c2e]
/swift-tensorflow-toolchain/usr/bin/swift[0x4d68bd]
/swift-tensorflow-toolchain/usr/bin/swift[0x4d395d]
/swift-tensorflow-toolchain/usr/bin/swift[0x474070]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7fc2236b97]
/swift-tensorflow-toolchain/usr/bin/swift[0x473cba]
The command '/bin/sh -c /swift-tensorflow-toolchain/usr/bin/swift test' returned a non-zero code: 1

I'll investigate all these problems today.

@dan-zheng
Copy link
Contributor

swift: /swift-base/swift/lib/Serialization/Serialization.cpp:2313: void swift::serialization::Serializer::DeclSerializer::writeDeclAttribute(const swift::DeclAttribute *): Assertion 'attr->getOriginalFunction() && "@differentiable attribute must have original function resolved"' failed.

I believe this assertion was introduced by #27985, which has not yet been merged. Perhaps this is a red herring?

@marcrasi
Copy link
Author

I believe this assertion was introduced by #27985, which has not yet been merged. Perhaps this is a red herring?

Oh yes good point, I made a mistake. I was looking at the results from your build of that PR, not the results of my build of this PR. This PR actually passes toolchain build and tests :)

Marc Rasi and others added 3 commits October 31, 2019 15:15
…med superclass type

It is possible for getSuperclassDecl() to return a non-null type, while
getSuperclass() returns an ErrorType. In this case, getContextSubstitutions()
could crash because the walk of the superclass chain via getSuperclass()
might not find the context class.

Instead of crashing in asserts builds, let getContextSubstitutions()
silently build an invalid substitution map here, just as it does in no-asserts
builds.
@marcrasi
Copy link
Author

marcrasi commented Nov 1, 2019

I believe that #27857 will fix these test failures. I'll cherry-pick that in, run CI, and hopefully finally be able to merge this!

@marcrasi
Copy link
Author

marcrasi commented Nov 1, 2019

@swift-ci please test tensorflow

1 similar comment
@marcrasi
Copy link
Author

marcrasi commented Nov 1, 2019

@swift-ci please test tensorflow

@marcrasi marcrasi merged commit d6ff384 into tensorflow Nov 1, 2019
@marcrasi marcrasi deleted the tf-489 branch November 1, 2019 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants