Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrow 0.7.x causes builds to fail on Android projects #807

Closed
jrgonzalezg opened this issue Apr 19, 2018 · 33 comments · Fixed by #926
Closed

Arrow 0.7.x causes builds to fail on Android projects #807

jrgonzalezg opened this issue Apr 19, 2018 · 33 comments · Fixed by #926
Assignees

Comments

@jrgonzalezg
Copy link
Member

jrgonzalezg commented Apr 19, 2018

Just adding the Arrow dependencies on an Android project with Java 8 features enabled causes the builds to fail due to a missing traverse related class / lambda during the desugar task.

Sample repo here: https://github.com/jrgonzalezg/arrow-android-failing

@jrgonzalezg jrgonzalezg changed the title Arrow 0.7.x makes builds fail on Android Arrow 0.7.x causes builds to fail on Android projects Apr 19, 2018
@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

Did a bit more research and it seems this also affected other projects (not using Arrow) even with fairly standard lambdas like this one: https://stackoverflow.com/questions/49866403/gradle-build-failure-couldnt-find-outer-class-xxxx So it maybe a more broad issue with lambdas that Arrow just triggered but not any issue in Arrow itself.

After that I checked again with Kotlin 1.2.41 and Android Gradle Plugin 3.2.0-alpha12 and the build succeeded. So maybe it was a bug either in Kotlin compiler fixed on 1.2.41 or the desugar process which may be fixed in canary versions of Android Studio / Gradle plugin. So while this means Arrow can't be used on current stable version of Android tools, it may work on the next one, so maybe this issue could be closed although it would be nice to know why it happened and still I haven't figured that out 😞

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

Checked separately and it seems to be the Android Gradle Plugin and not the Kotlin version which makes the build work. Still, it is now failing on generating release versions in the ProGuard stage with similar errors:

Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2

So I guess the problem about the class not being there still occurs, but now it compiles and fails in ProGuard instead of desugar. I am not sure if it will be safe to make ProGuard ignore all that issues (and it may require quite a bit of ProGuard rules or a broad / clever one). Could it be that the Kotlin compiler is omitting things in the class files that it shouldn't so the class files do not fully conform to the specification?. Maybe someone from Kotlin or with good knowledge of the class file requirements could figure out what is wrong here.

@jrgonzalezg
Copy link
Member Author

I also see that there is a PR updating Kotlin: #820 Maybe that one could be further updated from 1.2.40 to 1.2.41 and we could check this issue again once a new release of Arrow compiled with a more recent version of Kotlin is deployed. If it was a bug on the compiler it may now be fixed and new artifacts could work again. What do you think @raulraja ?

@raulraja
Copy link
Member

@jrgonzalezg asked @jereksel to bump up to latest so we can test it in the upcoming release. I still have no idea why this is showing up. I suspect this is some kind of inlining compiler bug by the error message but would be good to javap those files to see if arrow.data.ListK$traverse$1$2 is actually there.

@jrgonzalezg
Copy link
Member Author

@raulraja How would that be done? What I can see is that if you download the arrow-data-0.7.1.jar from http://search.maven.org/remotecontent?filepath=io/arrow-kt/arrow-data/0.7.1/arrow-data-0.7.1.jar and unzip it, you get a arrow/data/ListK$traverse$1$2$1.class but not a arrow/data/ListK$traverse$1$2.class. The full list of ListK classes on that artifact is:

ForListK.class
Higherkind_arrow_data_ListKKt.class
ListK$ap$1.class
ListK.class
ListK$Companion.class
ListK$foldRight$1$1.class
ListK$foldRight$1.class
ListK$foldRight$2.class
ListKKt.class
ListK$map2$1$1.class
ListK$map2$1.class
ListK$mapFilter$1.class
ListK$traverse$1$1.class
ListK$traverse$1$2$1.class
ListK$traverse$$inlined$run$lambda$1.class

Using javap *.class for all the classes on that artifact I don't see any reference to ListK$traverse$1$2 anywhere, not sure why the message mentions a reference or if the $ alone already implies subclass / inner class reference and thats why it tries to find it.

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

I have also tried compiling @jereksel branch to try to see if the artifacts there do not have this problem but I can't get that one to compile, I keep getting:

e: [kapt] An exception occurred: java.lang.IllegalStateException: currentFile.absoluteFile.parentFile must not be null
        at arrow.common.utils.ProcessorUtilsKt.recurseFilesUpwards(ProcessorUtils.kt:174)

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

@jereksel @raulraja Seems builds are also broken in master. Last commit that works is 3237249 and first that fails is 40baca8 by @JorgeCastilloPrz On builds after that one I get an automatic deletion of infographic/arrow-infographic.txt on "clean" task (./gradlew clean assemble) which could be the file that causes the error. But even if I restore that file and just do ./gradlew assemble (removing the clean) it still fails. Maybe Jorge knows what could be wrong with that commit.

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

Also tested against a rebased master removing some of the commits that cause issues but increasing Kotlin version to 1.2.41 and the contents of arrow-data-0.7.2-SNAPSHOT.jar are the same I reported above for the currently published version, so maybe that update will not help for the issue here 😞

@JorgeCastilloPrz
Copy link
Member

Deletion of the infographic.txt file is intended on clean. That's the expected behavior. It's interesting, because I've pulled master right now and run ./gradlew clean build on the root arrow dir and it's a successful build. Travis is also green on master branch.

@andrzejressel
Copy link
Contributor

@jrgonzalezg Arrow source code must be in "arrow" directory: 40baca8#diff-ea6113863a28ffe3f15442a287e31dadR103

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Apr 28, 2018

Thanks @jereksel I noticed that this afternoon and even commented it on Slack but it still failed when I renamed my repo root directory from arrow-jrgonzalezg to arrow. But I tried again now and it worked, so I'm not sure if I made a typo on the first renaming attempt or what happened.

Anyway, couldn't the root directory be determined from gradle or something similar instead of hardcoding it? git allows you to clone a repo into a directory with any name and it does not seem very reliable that the project does not work unless the directory name is fixed. Also the reason is very difficult to find.

The current behavior also makes it difficult to have several arrow repos cloned as I have had for a while (arrow-arrow-kt, arrow-jrgonzalez, ...) cause now only one named arrow will work. Is this a limitation of the annotation processors that can not be passed the root folder? If it is and they only get a limited context about location of files, could it be possible to make it recurse to a known directory with fixed name inside the repo instead of the containing root folder? For instance something like recurse up to "modules" and use an extra directory parent from that. In that way it could probably work in all cases despite were the repo is cloned. @JorgeCastilloPrz ?

@andrzejressel
Copy link
Contributor

@jrgonzalezg I'm sure this can be done another way (i like idea with parent of modules). I've created another issue for that problem: #824

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented May 1, 2018

I have updated the sample repository with another branch to show builds failing also when Java compatibility is set to Java 7. For that new case it fails on the ProGuard task instead of the desugar one. I have also updated the sample repo's README.md with instructions for triggering the build problem in both cases. See https://github.com/jrgonzalezg/arrow-android-failing

@JorgeCastilloPrz
Copy link
Member

JorgeCastilloPrz commented Jul 3, 2018

I've tried to add Arrow to a repo with plain Android and targeting java 8. This is a very standard configuration for Android projects:

android {
  //...
  compileOptions {
      sourceCompatibility JavaVersion.VERSION_1_8
      targetCompatibility JavaVersion.VERSION_1_8
  }
}

And as @jrgonzalezg said many times, you get a compile time error because of desugaring:

Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/data/ListK$traverse$1$2 of arrow/data/ListK$traverse$1$2$1
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1079)
        at com.google.devtools.build.android.desugar.ClassVsInterface.isOuterInterface(ClassVsInterface.java:56)
        at com.google.devtools.build.android.desugar.InterfaceDesugaring.visitOuterClass(InterfaceDesugaring.java:246)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:638)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:500)
        at com.google.devtools.build.android.desugar.Desugar.desugarClassesInInput(Desugar.java:477)
        at com.google.devtools.build.android.desugar.Desugar.desugarOneInput(Desugar.java:361)
        at com.google.devtools.build.android.desugar.Desugar.desugar(Desugar.java:314)
        at com.google.devtools.build.android.desugar.Desugar.main(Desugar.java:711)

The artifacts used are:

def arrow_version = '0.7.2'

implementation "io.arrow-kt:arrow-core:$arrow_version"
implementation "io.arrow-kt:arrow-syntax:$arrow_version"
implementation "io.arrow-kt:arrow-data:$arrow_version"

So yeah, there's a problem on this, since Arrow can't be used on Android like this. (People does not target jdk7 anymore if they can avoid it, since Android supports desugaring from some given java 8 features out of the box). I find it quite important. cc @raulraja @jereksel @pakoito.

@andrzejressel
Copy link
Contributor

andrzejressel commented Jul 3, 2018

Google bug report: https://issuetracker.google.com/issues/72750890 points to this bug report: https://youtrack.jetbrains.com/issue/KT-16084 which is "in progress".

Only thing we can do now is to test workaround and, if it works, post it here.

@pakoito
Copy link
Member

pakoito commented Jul 3, 2018

We can rewrite the classes not to use that pattern, right?

@pakoito
Copy link
Member

pakoito commented Jul 3, 2018

https://github.com/arrow-kt/arrow/blob/master/modules/core/arrow-data/src/main/kotlin/arrow/data/ListK.kt#L27

fun <G, B> traverse(GA: Applicative<G>, f: (A) -> Kind<G, B>): Kind<G, ListK<B>> =
    foldRight(Eval.always { GA.just(emptyList<B>().k()) }) { a, eval ->
      GA.run { f(a).map2Eval(eval) { (listOf(it.a) + it.b).k() } }
    }.value()
  }

@pakoito
Copy link
Member

pakoito commented Jul 3, 2018

If that fails we can make a local lambda or local function or capture a function reference to map2Eval, until it works.

@andrzejressel
Copy link
Contributor

Yeah, I think for now it'll do - we should comment old code out and restore it after KT-16084 is fixed.

@pakoito
Copy link
Member

pakoito commented Jul 5, 2018

Waiting to see how many more instances of the issue pop up.

@pakoito
Copy link
Member

pakoito commented Jul 13, 2018

Posted in gitter:

Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2
Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2

@jrgonzalezg
Copy link
Member Author

Just found the way to update the sample repo to Arrow 0.7.3-SNAPSHOT: https://github.com/jrgonzalezg/arrow-android-failing

@jrgonzalezg
Copy link
Member Author

This is for Arrow 0.7.3-SNAPSHOT on the repo above java7 branch:

Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.instances.EithertKt$traverse$1$1$1$1: can't find referenced class arrow.instances.EithertKt$traverse$1$1
Warning: arrow.instances.EithertKt$traverse$1$1$1$1: can't find referenced class arrow.instances.EithertKt$traverse$1$1
Warning: arrow.instances.OptiontKt$traverse$1$1$1$1: can't find referenced class arrow.instances.OptiontKt$traverse$1$1
Warning: arrow.instances.OptiontKt$traverse$1$1$1$1: can't find referenced class arrow.instances.OptiontKt$traverse$1$1
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2

@jrgonzalezg
Copy link
Member Author

jrgonzalezg commented Jul 13, 2018

So it seems the fix for ListK may have worked, but we got a few extra occurrences too 😅

@jrgonzalezg
Copy link
Member Author

Seems java7 reporting the issues in ProGuard gives more info than the master java 8 branch which fails in desugar with just these reported:

Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/data/StateT$Companion$tailRecM$1$1$1 of arrow/data/StateT$Companion$tailRecM$1$1$1$1
Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/instances/EithertKt$traverse$1$1 of arrow/instances/EithertKt$traverse$1$1$1$1
Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/typeclasses/Foldable$traverse_$1$2 of arrow/typeclasses/Foldable$traverse_$1$2$1

but I suspect it may be giving up before reporting all the errors like ProGuard.

@pakoito
Copy link
Member

pakoito commented Jul 28, 2018

Segun mis ultimos tests el fix de #955 lo ha arreglado todo. Puedes comprobarlo, @jrgonzalezg ??

@erickok
Copy link

erickok commented Aug 1, 2018

@pakoito This fixed by Proguard-enabled Android project release build (tested using JitPack build of master branch).

@jrgonzalezg
Copy link
Member Author

Seems this is fixed with Arrow 0.7.3, so we can finally close this 🎉

@m0skit0
Copy link

m0skit0 commented Oct 16, 2019

Looks like it is happening again. Kotlin 1.3.41, Arrow 0.9.0.

Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1

@pakoito
Copy link
Member

pakoito commented Oct 16, 2019

Yeah, that bug has been haunting us for a while. It's fixed in 0.10.1

@aballano
Copy link
Member

@m0skit0 @pakoito To be precise this is fixed in the current 0.10.2 version as 0.10.1 is failing in API < 24 atm

@jrgonzalezg
Copy link
Member Author

@aballano there is no 0.10.2, right? is it going to be generated soon?

@JorgeCastilloPrz
Copy link
Member

The snapshot is available, we're aiming for a release during next week :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants