Initial code generator support draft

[mahesh-hegde]

This PR adds initial support for code generator.

There are 2 parts;

* ApiSummarizer: Java component that generates JSON representation of Java APIs.
* jni_gen: takes JSON and generates code based on it.

test package is in jni_gen_test, which tests generating code for a very simple java class.

Locally I am able to generate code for some PDFBox classes and some packages from android libraries as well. Still contemplating how to add these to CI. PDFBox requires 5-6 jar dependencies to be downloaded from apache and bouncy castle sites. Looking for a way to automate it with maven or something. Similarly android-stubs-src.jar from SDK-28 is the one working with Android libraries.

TODO: (Before merging this PR)

• How to add a LICENSE to ApiSummarizer? It depends on some JDK APIs, should it be GPL?

[coveralls]

Pull Request Test Coverage Report for Build 2831694928

• -301 of 349 (13.75%) changed or added relevant lines in 8 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-23.1%) to 34.918%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
jni/lib/src/jni_exceptions.dart	0	1	0.0%
jni_gen/lib/src/util/name_utils.dart	4	12	33.33%
jni/lib/src/jni.dart	6	15	40.0%
jni_gen/lib/src/util/rename_conflict.dart	0	10	0.0%
jni_gen/lib/src/bindings/symbol_resolver.dart	36	47	76.6%
jni/lib/src/jl_object.dart	0	21	0.0%
jni_gen/lib/src/elements/elements.dart	2	71	2.82%
jni_gen/lib/src/elements/elements.g.dart	0	172	0.0%

Totals
Change from base Build 2723959478:	-23.1%
Covered Lines:	235
Relevant Lines:	673

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 2920439553

• 754 of 985 (76.55%) changed or added relevant lines in 20 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+20.8%) to 78.903%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
jni/lib/src/jni_exceptions.dart	0	1	0.0%
jni_gen/lib/src/bindings/symbol_resolver.dart	49	51	96.08%
jni_gen/lib/src/bindings/preprocessor.dart	38	41	92.68%
jni_gen/lib/src/writers/callback_writer.dart	0	3	0.0%
jni_gen/lib/src/writers/files_writer.dart	65	69	94.2%
jni_gen/lib/src/bindings/c_bindings.dart	161	166	96.99%
jni_gen/lib/src/bindings/common.dart	21	26	80.77%
jni_gen/lib/src/config/summary_source.dart	21	26	80.77%
jni_gen/lib/src/bindings/dart_bindings.dart	177	183	96.72%
jni_gen/lib/src/util/name_utils.dart	5	11	45.45%

Totals
Change from base Build 2723959478:	20.8%
Covered Lines:	748
Relevant Lines:	948

---

💛 - Coveralls

[dcharkes]

• How to add a LICENSE to ApiSummarizer? It depends on some JDK APIs, should it be GPL?

I believe this is the case. See https://www.gnu.org/licenses/gpl-faq.en.html#IfLibraryIsGPL. This is why LGPL was invented, it allows linking/importing a LGPL library in non-LGPL code. If the APIs are GPL, we are out of luck and have to use GPL. cc @mit-mit.
(This is hairy, if we load a .jar in a Java VM that we spun up with dart:ffi & JNI, is that considered linking as well? In that case the GPL would spill to this whole package. Maybe then it is a good thing that the API summarizer is Java and standalone. Then the Dart program that takes the summary doesn't have to link to that, and that can then be released under the Dart license.)

@devoncarew we're adding Java code to a Dart package. Do we have a practise for what Java package name to use? Something like dev.pub.jni_gen.<...>? Or com.github.dart_lang.jni_gen.<...> (In the Flutter plugins we have com.example.<package-name> which gives no guidance to the user.)

[dcharkes]

First round of comments. I'll take a look at the Java code later.

[dcharkes]

Super cool stuff! 🔥 🎸 🚀

[mahesh-hegde]

The doclet API has same license as rest of the java standard library, for example compare java.io.FileOutputStream and jdk.javadoc.doclet.Doclet.

On surface, we are using an API with classpath exception. However, the API calls the javadoc tool itself, I am not sure if such use is covered under classpath exception or not.

It's possible I am being overly cautious. There are some uses similar to ours covered under permissive licenses.

Selenium
Scala SBT

[dcharkes]

Super nice to play around with the summarizer! 🚀

More comments 😄

[mahesh-hegde]

Hi, I think I implemented most of the suggestions. What do we do about the license?

[dcharkes]

Hi, I think I implemented most of the suggestions. What do we do about the license?

Java code GPL. And it has to stay in third_party.

The generator can be the Dart license. It is very handy that we have the summarizer as a separate program now. Because we don't link it from the Dart, we just invoke it with Process.run or some other script.

[dcharkes]

Is there any google repo with GPL as a reference for how to add GPL license to ApiSummarizer files? The FSF website makes it quite long and confusing.

I'm not aware of any Google repo.

But, GitHub has a nifty search tool. To find repos that have Java and the GPL license: https://github.com/search?q=stars%3A%3E10000+language%3AJava+license%3Agpl&type=Repositories&ref=advsearch&l=Java&l=

• GPL 2 example: https://github.com/openjdk/jdk/blob/master/LICENSE
• GPL 3 example: https://github.com/java-decompiler/jd-gui

Probably:

• LICENSE file in jni_gen/third_party/ApiSummarizer/LICENSE
• And then in every source file:

/*
 * Copyright 2022, the Dart project authors.
 * This project is distributed under the GPLv3 license.
 * This is a Copyleft license that gives the user the right to use, 
 * copy and modify the code freely for non-commercial purposes.
 */

cc @mit-mit

[dcharkes]

@mahesh-hegde

I asked the experts about the licensing.

Our obligation to make the complete corresponding source code of a program available to end users as open source only triggers if we are distributing compiled software. If we are just publishing source code and are not distributing anything compiled, there should be no copyleft virality concerns.

Currently we are only distributing source, but we might want to distribute precompiled binaries at some point. So, GPLv2 applies.

The BSD is recognized by GNU/FSF/SFC as a GPLv2-compatible license. This is because compliance with the GPLv2 automatically satisfies compliance with the BSD-3 license, so if I distribute an application that depends on a GPLv2 licensed library, and I distribute the complete corresponding source code for that application, applying the GPLv2 license to the third-party GPLv2 files that I used, and applying the BSD license to all of the other code, this gives end users/the public all of the permissions they need to rebuild that application and to distribute it as a whole under the terms of the GPLv2 license.

So, we can release the Java code under the Dart BSD license. Which also means it doesn't have to live in the third_party folder.

Sorry for asking to change it again, 😅 we're all learning here! 😸

While you're at updating the folder and license, please also change the java package name to com.github.dart_lang.jni_gen.apisummarizer (Note dart-lang becomes dart_lang as per https://docs.oracle.com/javase/tutorial/java/package/namingpkgs.html)

Looks good to me, as long as all the TODOs have github issues.

Please also update the TODOs to point to GitHub issues.

[mahesh-hegde]

Re: package name in java

In the Android package:jni we need a java plugin class to init the JNI. There I used dev.dart.jni as package name since dart.dev seems to be official website. Should I update that too?

[dcharkes]

Re: package name in java

In the Android package:jni we need a java plugin class to init the JNI. There I used dev.dart.jni as package name since dart.dev seems to be official website. Should I update that too?

Yes, lets keep it consistent.

And yes, this is a dart-lang package, but it is not part of the Dart SDK. So I'm not sure if dev.dart makes sense.

[mahesh-hegde]

Added copyright notices to files in jni_gen/.

I am leaving the jni/ folder currently because I have another branch with some changes and don't want extra merge / rebase conflicts.

[dcharkes]

@mahesh-hegde please address these comments first so that we can merge this PR (instead of tagging new features on to this PR).

As mentioned before:

• Please file github issues for all open TODOs.
• All generated files require a copyright/license from the source file.

[mahesh-hegde]

Yes I am planning to put the examples in follow up PR. But can the disasm java code stay? Mahesh

…

On Tue, 23 Aug, 2022, 5:36 pm Daco Harkes, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In jni_gen/java/src/main/java/com/github/dart_lang/jni_gen/apisummarizer/disasm/AsmAnnotatedElementVisitor.java <#19 (comment)>: > @@ -0,0 +1,24 @@ +// Copyright (c) 2022, the Dart project authors. Please see the AUTHORS file Can we merge it in a follow up PR with those examples together? That makes this easier to review. — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALAKLFVQR75RN23NO5ENZX3V2S5EPANCNFSM56DP7VIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dcharkes]

Yes I am planning to put the examples in follow up PR. But can the disasm java code stay?

I think it is bad practise to land code that is not used. What is the problem with adding it in the next PR?

[mahesh-hegde]

The --backend asm option calls it. I think it's extra work to remove that option and corresponding logic in this PR and add back in next. It actually works right now. I can add a test for that. Mahesh

…

On Tue, 23 Aug, 2022, 6:07 pm Daco Harkes, ***@***.***> wrote: Yes I am planning to put the examples in follow up PR. But can the disasm java code stay? I think it is bad practise to land code that is not used. What is the problem with adding it in the next PR? — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALAKLFRAPUKOSYQ5GZE7R5TV2TA2FANCNFSM56DP7VIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dcharkes]

A ton of small comments, but no larger concerns anymore.