Open API: Remove runtime Jar from build and deploy

[rdblue]

This removes the Open API runtime Jar from build and deploy. This is an alternative to recent PRs that are attempting to fix the LICENSE and NOTICE documentation, #15872 and #16143.

This Jar was added by #11279. Looking at that PR, the original runtime was about 50 MB, with a 100+ MB version that was rejected for having too much bundled. The Jar is now 4x larger: 200+ MB.

I'm not confident that the original LICENSE and NOTICE were handled correctly; they were auto-generated from license-report, but had to be merged with the root and had a LGPL dependency at some point. I also don't think that the LICENSE was ever thoroughly checked. One comment says "I did some checking and didn't find any red flags" but it doesn't look like there was a thorough review.

Rather than minimizing dependencies and reviewing the LICENSE and NOTICE content, I think that this artifact should be removed from the build.

If I understand correctly, it was created in order to build a Docker image (#11283), but a Docker image doesn't require a runtime Jar. It just needs a lib/ directory with the necessary dependencies, which is a better way to distribute transitive dependencies with their own license documentation. If there is no need to have a runtime Jar, we should not distribute one.

I also don't think that there is a reasonable use for this runtime Jar. Runtime Jars are a substantial amount of work -- to minimize dependencies and document them -- and we produce runtime Jars when it is a big help to downstream users. For example, adding Iceberg to Spark is much, much easier. We do not distribute runtime Jars for downstream projects that use build systems.

Because this is integration test code, we can reasonably assume that the downstream consumers are using a build system and are able to use that to maintain their own version-managed classpath. This is also test code because we decided not to distribute an example REST service implementation in the project.

I think that this runtime Jar should be removed. Projects using it for testing can use their builds to integrate it and produce the classpath. We still want a Docker image for cross-project testing within the Iceberg community, but that should be built differently.

[rdblue]

I've also disabled the Docker image workflow that is producing artifacts with incorrect LICENSE documentation.

[stevenzwu]

LGTM.

Regarding the docker image, I guess we need to fix the license and publish it. other language repos (like python, rust, go) depends the iceberg-rest-fixture docker image for integration test using docker compose.

https://github.com/apache/iceberg-python/blob/main/dev/docker-compose-integration.yml

[RussellSpitzer]

From another discussion, I'm +1 with the plan to

1. In this pr remove the deploy directive to stop publishing the fat jar

2. In a follow up replace the fatjar task with something similar to how kafka connect bundles up a little zip file of dependencies

3. Go through that list and remove anything that is strictly required

4. Update License and Notice

5. Re-enable docker build using the new tar / untar approach of the test classpath instead of a fat jar

[rdblue]

@danielcweeks's feedback was to leave the shadowJar task and just remove it from deploy.gradle so it is not published. I think that should still work so I updated this PR.

[RussellSpitzer]

Merged, I think we have consensus here and a plan forward.

Thanks @rdblue for the pr and everyone who reviewed / chimed in

@nastra , @stevenzwu , @huaxingao , @singhpk234 , @danielcweeks , @aihuaxu , @amogh-jahagirdar

[RussellSpitzer]

Quick test for publishing

rm -rf ~/.m2/repository/org/apache/iceberg/iceberg-open-api
./gradlew :iceberg-open-api:publishToMavenLocal 2>&1
find ~/.m2/repository/org/apache/iceberg/iceberg-open-api -type f | sort

/Users/rspitzer/.m2/repository/org/apache/iceberg/iceberg-open-api/3f14731/iceberg-open-api-3f14731-test-fixtures.jar
/Users/rspitzer/.m2/repository/org/apache/iceberg/iceberg-open-api/3f14731/iceberg-open-api-3f14731-tests.jar
/Users/rspitzer/.m2/repository/org/apache/iceberg/iceberg-open-api/3f14731/iceberg-open-api-3f14731.pom
/Users/rspitzer/.m2/repository/org/apache/iceberg/iceberg-open-api/maven-metadata-local.xml