Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARGC exceptions when translating #1471

Open
tomekfab opened this issue Sep 15, 2020 · 8 comments
Open

ARGC exceptions when translating #1471

tomekfab opened this issue Sep 15, 2020 · 8 comments

Comments

@tomekfab
Copy link

tomekfab commented Sep 15, 2020

We are trying to use garbage collection approach in our project to see if it helps with some of the memory issues we're facing, but we've been struggling with exceptions that arise during translation. We've built j2objc from master-ARGC branch according to the instructions (we've successfully did this with previous releases). The issues we're having arise when translating dependencies that we use, specifically Joda Convert (v1.8) and jtry.

error: internal error translating "/.../build/j2objc/java/source/org/joda/convert/AbstractTypeStringConverter.java"
com.google.devtools.j2objc.ast.TreeVisitorError: /.../build/j2objc/java/source/org/joda/convert/AbstractTypeStringConverter.java:36: NullPointerException: null
	at com.google.devtools.j2objc.util.TypeUtil.asTypeElement(TypeUtil.java:238)
	at com.google.devtools.j2objc.ast.TypeDeclaration.acceptInner(TypeDeclaration.java:95)
	at com.google.devtools.j2objc.ast.TreeNode.accept(TreeNode.java:82)
	at com.google.devtools.j2objc.ast.ChildLink.accept(ChildLink.java:76)
	at com.google.devtools.j2objc.ast.ChildList.accept(ChildList.java:90)
	at com.google.devtools.j2objc.ast.CompilationUnit.acceptInner(CompilationUnit.java:194)
	at com.google.devtools.j2objc.ast.TreeNode.accept(TreeNode.java:82)
	at com.google.devtools.j2objc.ast.UnitTreeVisitor.run(UnitTreeVisitor.java:50)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.applyMutations(TranslationProcessor.java:152)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.processConvertedTree(TranslationProcessor.java:107)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processCompiledSource(FileProcessor.java:169)
	at com.google.devtools.j2objc.pipeline.FileProcessor.access$000(FileProcessor.java:43)
	at com.google.devtools.j2objc.pipeline.FileProcessor$1.handleParsedUnit(FileProcessor.java:141)
	at com.google.devtools.j2objc.javac.JavacParser.parseFiles(JavacParser.java:299)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processBatch(FileProcessor.java:146)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processInputs(FileProcessor.java:71)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.processInputs(TranslationProcessor.java:83)
	at com.google.devtools.j2objc.J2ObjC.runEx(J2ObjC.java:138)
	at com.google.devtools.j2objc.J2ObjC.main(J2ObjC.java:200)

We have this exception for ~50 files. Stacktrace is exactly the same for all of them. We're using Java 1.8.0_221. None of these exceptions arise when translating with 2.4, 2.5 or 2.6

I'd greatly appreciate any suggestions to make this work for us 🙏

@tomball
Copy link
Collaborator

tomball commented Sep 15, 2020

Try addressing the TypeUtil.asTypeElement() NPE (that method hasn't changed in the ARGC branch) with:

  public DeclaredType getSuperclass(TypeMirror t) {
    List<? extends TypeMirror> supertypes = directSupertypes(t);
    if (supertypes.isEmpty()) {
      return null;
    }
    TypeMirror first = supertypes.get(0);
+   ElementKind kind = getDeclaredTypeKind(first);
>   if (kind != null && kind.isClass()) {
      return (DeclaredType) first;
    }
    return null;
  }

This isn't likely to fix the underlying problem, but hopefully you'll get more descriptive errors.

What version of Joda Convert are you transpiling? The current version doesn't have an AbstractTypeStringConverter class, so I cannot recreate this issue locally. Also, what j2objc flags do you set?

@tomekfab
Copy link
Author

tomekfab commented Sep 16, 2020

Thanks for response, much appreciated

What version of Joda Convert are you transpiling?

We're using 1.8 but I've also tried with 1.9.2

Also, what j2objc flags do you set?

            def classPath = "${System.env.J2OBJC_HOME}/lib/jre_emul.jar:" +
                    "${System.env.J2OBJC_HOME}/lib/guava-25.1-jre.jar:" +
                    "${System.env.J2OBJC_HOME}/lib/jsr305-3.0.0.jar:" +
                    "${System.env.J2OBJC_HOME}/lib/javax.inject-1.jar:" +
                    "${System.env.J2OBJC_HOME}/lib/jre_emul.jar:" +
                    "${System.env.J2OBJC_HOME}/lib/j2objc.jar:" +
                    "$projectDir/src/main/java/:" +
                    "$projectDir/build/j2objc/java/source:" +
                    "$projectDir/build/generated/source/apt/main"

            executable "${System.env.J2OBJC_HOME}/j2objc"
            args "-source", "1.8"
            args "-classpath", classPath
            args "-encoding", "utf-8"
            args "--prefixes", "$projectDir/prefixes.properties"
            args "--doc-comments"
            args "--static-accessor-methods"
            args "--nullability"
            args "--generate-deprecated"
            args "-g" //debug flag

            args "-sourcepath", "$projectDir/src/main/java/:$projectDir/build/j2objc/java/source:$projectDir/build/generated/source/apt/main"
            args "-d", "$projectDir/build/j2objc/objc/source"
            args "--dead-code-report", "$projectDir/build/proguard/usage.log"

Try addressing the TypeUtil.asTypeElement() NPE

I've rebuilt with the change you proposed but the error description remains the same:

error: internal error translating "/.../build/j2objc/java/source/org/joda/convert/ToStringConverter.java"
com.google.devtools.j2objc.ast.TreeVisitorError: /.../build/j2objc/java/source/org/joda/convert/ToStringConverter.java:26: NullPointerException: null
	at com.google.devtools.j2objc.util.TypeUtil.asTypeElement(TypeUtil.java:238)
	at com.google.devtools.j2objc.ast.TypeDeclaration.acceptInner(TypeDeclaration.java:95)
	at com.google.devtools.j2objc.ast.TreeNode.accept(TreeNode.java:82)
	at com.google.devtools.j2objc.ast.ChildLink.accept(ChildLink.java:76)
	at com.google.devtools.j2objc.ast.ChildList.accept(ChildList.java:90)
	at com.google.devtools.j2objc.ast.CompilationUnit.acceptInner(CompilationUnit.java:194)
	at com.google.devtools.j2objc.ast.TreeNode.accept(TreeNode.java:82)
	at com.google.devtools.j2objc.ast.UnitTreeVisitor.run(UnitTreeVisitor.java:50)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.applyMutations(TranslationProcessor.java:152)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.processConvertedTree(TranslationProcessor.java:107)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processCompiledSource(FileProcessor.java:169)
	at com.google.devtools.j2objc.pipeline.FileProcessor.access$000(FileProcessor.java:43)
	at com.google.devtools.j2objc.pipeline.FileProcessor$1.handleParsedUnit(FileProcessor.java:141)
	at com.google.devtools.j2objc.javac.JavacParser.parseFiles(JavacParser.java:299)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processBatch(FileProcessor.java:146)
	at com.google.devtools.j2objc.pipeline.FileProcessor.processInputs(FileProcessor.java:71)
	at com.google.devtools.j2objc.pipeline.TranslationProcessor.processInputs(TranslationProcessor.java:83)
	at com.google.devtools.j2objc.J2ObjC.runEx(J2ObjC.java:138)
	at com.google.devtools.j2objc.J2ObjC.main(J2ObjC.java:200)

@tomball
Copy link
Collaborator

tomball commented Sep 16, 2020 via email

@tomekfab
Copy link
Author

tomekfab commented Sep 18, 2020

Thanks for suggestion. I tried that, result was the same. Another thing was suggested though to remove args "--dead-code-report", "$projectDir/build/proguard/usage.log" which did help. j2objc completed successfully however the created Xcode project does not compile and has ~6000 warnings vs ~15 which we get from j2objc 2.4 - not sure if that's expected given the experimental nature of this approach. The only line that prevents compilation is this:

Screenshot_2020-09-17_at_14 21 52

which produces Prefix attribute must be followed by an interface or protocol error.

I've tried translating without dead-code-report on j2objc 2.4 to make sure that this is not caused by that and I can confirm that these issues only appear for the ARGC build.

@tomball
Copy link
Collaborator

tomball commented Sep 18, 2020

The ARGC branch is experimental from an external contributor, so it hasn't gone through any rigorous code review yet (that's our next task, now that 2.7 has released). This change looks to be caused by the ARGC compiler generating code to support java.lang.String.intern(), by moving Objective C string constants into a local table and then in the initialize method interning them. So it should be harmless, though that design will probably need improvement.

I can certainly believe that dead-code stripping may be broken; although the ARGC compiler succeeds for me with a simple dead-code-report file, your file is probably stripping something which causes the breakage. Would you mind attaching the joda-convert (org/joda/convert) related entries from your dead-code-report? Since that library is open-source, those entries shouldn't accidentally expose any proprietary API in your app.

@tomball
Copy link
Collaborator

tomball commented Sep 18, 2020

FYI, you can get 99% of the benefits of the ARGC compiler without using it, since it's design uses LLVM's ARC for most of the memory management, with a lightweight garbage collector just to clean up any remaining leaks. ARC will help even if the memory issues you are facing are leaks, and both the cycle_finder tool and Xcode Instruments can find leaks that can be addressed using J2ObjC's memory management annotations. It's more work to do that, but hopefully less work than wrestling with an experimental compiler. :-)

To update a project using j2objc to use ARC:

  • Use the -use-arc with the j2objc compiler,
  • In Xcode's target Build Settings:
    • set "Objective-C Automatic Reference Counting" to "Yes",
    • set "Enable Objective-C Exceptions" to "Yes".

If you embedded Objective-C native code in any Java sources, the cleanup involves:

  • replace retain messages with RETAIN_() macros,
  • replace release messages with RELEASE_() macros,
  • replace autorelease messages with AUTORELEASE() macros.

The macros are declared here, as well as other useful memory management macros. Although those messages can be deleted in ARC sources, these macros allow your code to be built correctly using either ARC or manual resource counting.

The final cleanup is to conditionally call [super dealloc] in any native dealloc methods, like this:

- (void)dealloc {
  // clean up statements

#if !__has_feature(objc_arc)
  [super dealloc];
#endif
}

@tomball
Copy link
Collaborator

tomball commented Sep 18, 2020

One last point: ARC and manual reference counting can co-exist, so your project can have native code that does reference counting, and transpiled Java that uses ARC.

@tomekfab
Copy link
Author

tomekfab commented Sep 21, 2020

Would you mind attaching the joda-convert (org/joda/convert) related entries from your dead-code-report?

Here are the rules that we use:

-injars ../build/libs/core.jar
-libraryjars <java.home>/lib/rt.jar

-dontoptimize
-dontobfuscate
-dontpreverify
-printusage
-dontnote

-keepclassmembers class * {
    static final % *;
    static final java.lang.String *;
}
-keep class org.joda.time.** { *; }

-dontwarn com.google.j2objc.**
-dontwarn com.google.common.**
-dontwarn javax.**
-dontwarn org.json.**
-dontwarn com.google.auto.value.**

Besides those we just have a few more keep for our classes and some other dependencies.

And here are the joda convert related logs:

org.joda.convert.AbstractTypeStringConverter
org.joda.convert.AnnotationStringConverterFactory
org.joda.convert.EnumStringConverterFactory
org.joda.convert.EnumStringConverterFactory$EnumStringConverter
org.joda.convert.FromString
org.joda.convert.FromStringConverter
org.joda.convert.FromStringFactory
org.joda.convert.JDKStringConverter
org.joda.convert.JDKStringConverter$1
org.joda.convert.JDKStringConverter$10
org.joda.convert.JDKStringConverter$11
org.joda.convert.JDKStringConverter$12
org.joda.convert.JDKStringConverter$13
org.joda.convert.JDKStringConverter$14
org.joda.convert.JDKStringConverter$15
org.joda.convert.JDKStringConverter$16
org.joda.convert.JDKStringConverter$17
org.joda.convert.JDKStringConverter$18
org.joda.convert.JDKStringConverter$19
org.joda.convert.JDKStringConverter$2
org.joda.convert.JDKStringConverter$20
org.joda.convert.JDKStringConverter$21
org.joda.convert.JDKStringConverter$22
org.joda.convert.JDKStringConverter$23
org.joda.convert.JDKStringConverter$24
org.joda.convert.JDKStringConverter$25
org.joda.convert.JDKStringConverter$26
org.joda.convert.JDKStringConverter$27
org.joda.convert.JDKStringConverter$28
org.joda.convert.JDKStringConverter$29
org.joda.convert.JDKStringConverter$3
org.joda.convert.JDKStringConverter$30
org.joda.convert.JDKStringConverter$31
org.joda.convert.JDKStringConverter$4
org.joda.convert.JDKStringConverter$5
org.joda.convert.JDKStringConverter$6
org.joda.convert.JDKStringConverter$7
org.joda.convert.JDKStringConverter$8
org.joda.convert.JDKStringConverter$9
org.joda.convert.MethodConstructorStringConverter
org.joda.convert.MethodsStringConverter
org.joda.convert.OptionalDoubleStringConverter
org.joda.convert.OptionalIntStringConverter
org.joda.convert.OptionalLongStringConverter
org.joda.convert.ReflectionStringConverter
org.joda.convert.RenameHandler
org.joda.convert.StringConvert
org.joda.convert.StringConvert$1
org.joda.convert.StringConvert$2
org.joda.convert.StringConverter
org.joda.convert.StringConverterFactory
org.joda.convert.ToString
org.joda.convert.ToStringConverter
org.joda.convert.TypeStringConverter
org.joda.convert.TypeTokenStringConverter
org.joda.convert.TypedAdapter
org.joda.convert.TypedStringConverter
org.joda.convert.factory.BooleanArrayStringConverterFactory
org.joda.convert.factory.BooleanArrayStringConverterFactory$1
org.joda.convert.factory.BooleanArrayStringConverterFactory$BooleanArrayStringConverter
org.joda.convert.factory.BooleanArrayStringConverterFactory$BooleanArrayStringConverter$1
org.joda.convert.factory.BooleanObjectArrayStringConverterFactory
org.joda.convert.factory.BooleanObjectArrayStringConverterFactory$1
org.joda.convert.factory.BooleanObjectArrayStringConverterFactory$BooleanArrayStringConverter
org.joda.convert.factory.BooleanObjectArrayStringConverterFactory$BooleanArrayStringConverter$1
org.joda.convert.factory.ByteObjectArrayStringConverterFactory
org.joda.convert.factory.ByteObjectArrayStringConverterFactory$1
org.joda.convert.factory.ByteObjectArrayStringConverterFactory$ByteArrayStringConverter
org.joda.convert.factory.ByteObjectArrayStringConverterFactory$ByteArrayStringConverter$1
org.joda.convert.factory.CharObjectArrayStringConverterFactory
org.joda.convert.factory.CharObjectArrayStringConverterFactory$1
org.joda.convert.factory.CharObjectArrayStringConverterFactory$CharecterArrayStringConverter
org.joda.convert.factory.CharObjectArrayStringConverterFactory$CharecterArrayStringConverter$1
org.joda.convert.factory.NumericArrayStringConverterFactory
org.joda.convert.factory.NumericArrayStringConverterFactory$1
org.joda.convert.factory.NumericArrayStringConverterFactory$DoubleArrayStringConverter
org.joda.convert.factory.NumericArrayStringConverterFactory$DoubleArrayStringConverter$1
org.joda.convert.factory.NumericArrayStringConverterFactory$FloatArrayStringConverter
org.joda.convert.factory.NumericArrayStringConverterFactory$FloatArrayStringConverter$1
org.joda.convert.factory.NumericArrayStringConverterFactory$IntArrayStringConverter
org.joda.convert.factory.NumericArrayStringConverterFactory$IntArrayStringConverter$1
org.joda.convert.factory.NumericArrayStringConverterFactory$LongArrayStringConverter
org.joda.convert.factory.NumericArrayStringConverterFactory$LongArrayStringConverter$1
org.joda.convert.factory.NumericArrayStringConverterFactory$ShortArrayStringConverter
org.joda.convert.factory.NumericArrayStringConverterFactory$ShortArrayStringConverter$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$DoubleArrayStringConverter
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$DoubleArrayStringConverter$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$FloatArrayStringConverter
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$FloatArrayStringConverter$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$IntArrayStringConverter
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$IntArrayStringConverter$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$LongArrayStringConverter
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$LongArrayStringConverter$1
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$ShortArrayStringConverter
org.joda.convert.factory.NumericObjectArrayStringConverterFactory$ShortArrayStringConverter$1

It's more work to do that, but hopefully less work than wrestling with an experimental compiler. :-)

That's exactly our problem, we've neglected task of taking care of leaks for quite a long time and now wrestling with an experimental compiler has become a chance for an easy way out :) Our hope was this garbage collector would help us resolve those leaks to some extent at least but I suppose it's time to pull up the sleeves and start working on those ourselves :) We did try ARC in the past and it didn't change much in terms of the memory-related issues we're facing.

Anyway thanks for all the help! Given how we rely on j2objc I suppose we'll give the garbage collector another go once it's more stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants