[Bug] META-INF issues in paimon-s3-impl-0.6.0-incubating.jar

[pjfanning]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.6.0-rc2

Compute Engine

n/a

Minimal reproduce step

unzip https://repository.apache.org/content/repositories/orgapachepaimon-1031/org/apache/paimon/paimon-s3-impl/0.6.0-incubating/paimon-s3-impl-0.6.0-incubating.jar

you'll find a nested META-INF dir

    0  09-12-2018 06:28   META-INF/versions/11/META-INF/
 2345  09-12-2018 06:28   META-INF/versions/11/META-INF/MANIFEST.MF
    0  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/
    0  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/9/
    0  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/9/javax/
    0  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/9/javax/xml/
    0  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/9/javax/xml/bind/
 4403  09-12-2018 06:28   META-INF/versions/11/META-INF/versions/9/javax/xml/bind/ModuleUtil.class
36551  09-12-2018 06:28   META-INF/versions/11/META-INF/LICENSE.txt

What doesn't meet your expectations?

We shouldn't include the nested META-INF dir

Anything else?

I think all of your bundle jars should be reviewed again. I have reported similar problems before.

#2424 and #2429 are incomplete fixes

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

Hi @pjfanning , Very thanks to your review~

Sorry for this, I thought the biggest problem was that the root directory didn't have all the META-INF content, so I followed this approach to fix it.

Let me explain to you the current mechanism here. Currently, in order to isolate classes, we have extracted the jar into a subdirectory, which can avoid loading classes that are prone to conflicts in the default classloader. Then we use the new classloader during runtime to load the contents of the subdirectory.

The entire jar has been decompressed to a subdirectory, and a new classloader will access the entire subdirectory. This requires that the contents of the subdirectory be complete files, including META-INF. Why? For example, classes for JDk11 will be included, which cannot be deleted.

[pjfanning]

META-INF/versions/11/META-INF/versions/9/javax/xml/bind/ModuleUtil.class should be META-INF/versions/9/javax/xml/bind/ModuleUtil.class

these files should be excluded:
META-INF/versions/11/META-INF/MANIFEST.MF
META-INF/versions/11/META-INF/LICENSE.txt

[JingsongLi]

@pjfanning Thanks!

We will try to exclude MANIFEST.MF and LICENSE.txt and test it.

Now, only paimon-s3 and paimon-oss have the plugin mechanism, we will fix them.