Native libraries licensing

[karllessard]

(note: this is an extract from a previous pull request, which is now made as a separate issue)

In actual TF Java artifacts (i.e. those issued from the main TF repository), the native artifacts (e.g. libtensorflow_jni.jar) contains a LICENSE file (and a new THIRD_PARTY_TF_JNI_LICENSES since 1.15, I think). These licenses list copyright notices from different libraries used by TF core runtime.

When we migrated to our Java repo, I took a snapshot of the LICENSE file and copy it under tensorflow-core-api/src/main/resources. But that was not very useful since the licenses must be attached to the same artifact/folder than the native libraries we distribute (i.e. that jars with the os-arch classifier) and now it was ending in our Java jar (without classifier).

So I moved it to tensorflow-core-api directly for now and I guess that we need to add some Maven rules to include it in our classified/native jars. Nonetheless, some details remain obscure to me:

Where do come from the LICENSE file (and the new THIRD_PARTY_TF_JNI_LICENCES) found in the TF Java 1.15 artifacts?
How do we stay in sync with changes in those files?
Are the licenses the same for all supported OS?

CC: @sjamesr

[karllessard]

From @sjamesr : actual licenses are generated from this rule: https://github.com/tensorflow/tensorflow/blob/1b5876658957d91287d63b0fffb28b1510ac7fab/tensorflow/tools/lib_package/BUILD#L214

[saudet]

It basically just calls concat_licenses.sh, so it's something we can call ourselves in build.sh. That doesn't answer the question about how that list of files is generated though. Are we supposed to maintain it ourselves? That doesn't sound practical...

[karllessard]

I've added the jnilicenses_generate rule to the Bazel command line in our script and it does generate the licence file we need. It make sense that the list of third-party C++ dependencies is maintain in TF core itself but we should probably move the genrule on our side. I don't know how we can do this though, we'll probably need to make changes to the main repo. But for now, we can live by calling the @org_tensorflow genrule directly.

So to do the same as in 1.15, we need to move the following files into our native target folder (target/native/org/tensorflow/internal/c_api/{classifier}/) :

• bazel-bin/external/org_tensorflow/tensorflow/tools/lib_package/THIRD_PARTY_TF_JNI_LICENSES
• ../../LICENSE (i.e. at the root of tensorflow-java repo)

@saudet , since all libraries found in the native target folder are already copied by JavaCPP, is there any additional properties we can set to copy those licence files as well?

[karllessard]

Never mind, I think I've found out. I'm almost done, I'll create a PR so you can review it.

[saudet]

The problem as I understand it is that no one is maintaining the jnilicenses_generate rule anymore and that it's already out-of-date.

[karllessard]

Is it? Maybe that list could be maintained by our fellows at Google @sjamesr?

[karllessard]

This aggregate third-party license is quite big as well, 400K. So for a core platform, which includes three native jars, it sums up to an additional 1M, which is half the size of the Java core API artifact. But compared to the size of a native jar (~50M), this become suddenly acceptable :)

[saudet]

Is it? Maybe that list could be maintained by our fellows at Google @sjamesr?

Well, anything named "JNI" rings bells about not being maintained anymore, so we should be careful about it being meaningful. There should be some sort of equivalent for the C API...

[karllessard]

I guess the list should already match what the C API is using since there is no third party library that is called directly from the JNI code, afaik.

[karllessard]

But is the list the third party dependencies the same for 1.15 (which is probably the snapshot we see now) and 2.0? There is a lot of assumptions here, we definitely need the point of view from someone at Google.

[karllessard]

This has been resolved a long time ago with #17 , closing it