Remove remaining checked-in binaries (ZIPs, JARs) used in CI/CD

[dblock]

Is your feature request related to a problem? Please describe

Because of lack of maven, support for "latest" URIs, or for need of quickly replacing third-party dependencies, projects have checked in ZIPs and sometimes JARs that are either used in CI/CD or occasionally released with the product.

• Checked-in ZIPs (e.g. a plugin that is used for integration tests) need to be constantly updated, or tests are run against obsolete versions of dependencies.
• Hand-built JARs are often not released versions, therefore can contain unfixed bugs.
• JARs built on developer machines cannot be trusted, the environment where these are built is unknown and cannot be reproduced with confidence.
• For bwc tests we don't check in entire distributions of OpenSearch, it shouldn't be different for some plugins either.

Describe the solution you'd like

Replace any checked-in ZIPs or JARs with dynamic dependencies obtained from reliable sources.

• [BUG] Plugin contains checked in JARs with unclear provenance anomaly-detection#433
• [BUG] Checked in versions of job-scheduler-*-SNAPSHOT.zip  anomaly-detection#17
• [BUG] Replace checked-in ZIPs with dynamic dependencies  index-management#305
• Replace opendistro-alerting-1.13.1.0.zip with an external download alerting#350
• [BUG] Replace checked in reports-scheduler/src/test/resources/job-scheduler/opensearch-job-scheduler-1.2.3.0-*.zip with a dynamic dependency reporting#263
• Replace checked-in .zip-s with dynamic downloads k-NN#320
• Replace checked-in SNAPSHOT zip with a dynamic dependency observability#581
• Replace checked-in ZIPs with dynamic dependencies cross-cluster-replication#335
• Replace checked in ZIP with a dynamic dependency performance-analyzer#167
• Replace checked-in ZIP with a dynamic dependency sql#491
• Replace checked-in ml-commons ZIP with a dynamic dependency sql#495
• Replace checked-in ZIPs with downloads of official releases security#1767
• Replace checked-in ZIPs with dynamic dependencies asynchronous-search#132
• Replace checked-in ZIPs with dynamic dependencies notifications#415

Describe alternatives you've considered

No response

Additional context

No response

[dblock]

As of 1.3, a quick search in plugins.

$ find . -name *.zip
./alerting/alerting/src/test/resources/bwc/alerting/1.13.1.0/opendistro-alerting-1.13.1.0.zip
./index-management/src/test/resources/bwc/job-scheduler/1.13.0.0/opendistro-job-scheduler-1.13.0.0.zip
./index-management/src/test/resources/bwc/indexmanagement/1.13.2.0/opendistro-indexmanagement-1.13.2.0.zip
./index-management/src/test/resources/job-scheduler/opensearch-job-scheduler-1.3.0.0-SNAPSHOT.zip
./index-management/src/test/resources/lang-mustache/lang-mustache-1.0.0.zip
./anomaly-detection/src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.13.0.0/opendistro-anomaly-detection-1.13.0.0.zip
./anomaly-detection/src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT/opensearch-anomaly-detection-1.1.0.0-SNAPSHOT.zip
./anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.13.0.0/opendistro-job-scheduler-1.13.0.0.zip
./anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.2.0.0-SNAPSHOT/opensearch-job-scheduler-1.2.0.0-SNAPSHOT.zip
./anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.3.0.0-SNAPSHOT/opensearch-job-scheduler-1.3.0.0-SNAPSHOT.zip
./anomaly-detection/src/test/resources/job-scheduler/opensearch-job-scheduler-1.3.0.0-SNAPSHOT.zip
./dashboards-reports/reports-scheduler/src/test/resources/bwc/job-scheduler/1.13.0.0/opendistro-job-scheduler-1.13.0.0.zip
./dashboards-reports/reports-scheduler/src/test/resources/bwc/job-scheduler/1.3.0.0-SNAPSHOT/opensearch-job-scheduler-1.3.0.0-SNAPSHOT.zip
./dashboards-reports/reports-scheduler/src/test/resources/bwc/reports-scheduler/1.13.0.0/opendistro-reports-scheduler-1.13.0.0.zip
./dashboards-reports/reports-scheduler/src/test/resources/job-scheduler/opensearch-job-scheduler-1.3.0.0-SNAPSHOT.zip
./k-nn/src/test/resources/org/opensearch/knn/bwc/1.1.0.0-SNAPSHOT/opensearch-knn-1.1.0.0.zip
./k-nn/src/test/resources/org/opensearch/knn/bwc/1.0.0.0-SNAPSHOT/opensearch-knn-1.0.0.0.zip
./k-nn/src/test/resources/org/opensearch/knn/bwc/1.2.0.0-SNAPSHOT/opensearch-knn-1.2.0.0-linux-x64.zip
./observability/opensearch-observability/src/test/kotlin/org/opensearch/observability/resources/bwc/1.1.0-SNAPSHOT/opensearch-notebooks-1.1.0.0-SNAPSHOT.zip
./cross-cluster-replication/src/test/resources/replication/opensearch-cross-cluster-replication-1.1.0.0.zip
./performance-analyzer/src/test/resources/org/opensearch/performanceanalyzer/bwc/1.13.0.0/opendistro-performance-analyzer-1.13.0.0.zip
./sql/integ-test/src/test/resources/bwc/1.13.2.0/opendistro-sql-1.13.2.0.zip

$ find . -name *.jar
./anomaly-detection/lib/randomcutforest-parkservices-2.0.1.jar
./anomaly-detection/lib/protostuff-runtime-1.8.0-SNAPSHOT.jar
./anomaly-detection/lib/randomcutforest-core-2.0.1.jar
./anomaly-detection/lib/protostuff-api-1.8.0-SNAPSHOT.jar
./anomaly-detection/lib/protostuff-core-1.8.0-SNAPSHOT.jar
./anomaly-detection/lib/protostuff-collectionschema-1.8.0-SNAPSHOT.jar

[jmazanec15]

@dblock I was trying to figure out how to take dependencies from https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/. (Specifically for "lang-painless" module). However, one issue is that the plugins do not have "plugin-descriptor.properties" leading to clis to fail

| Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for module directory [lang-painless]
|       at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:452)
|       at org.opensearch.plugins.PluginsService.findBundles(PluginsService.java:437)
|       at org.opensearch.plugins.PluginsService.getModuleBundles(PluginsService.java:425)
|       at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:849)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.Command.main(Command.java:101)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
| Caused by: java.nio.file.NoSuchFileException: .../k-NN-1/build/testclusters/integTest-0/distro/2.0.0-INTEG_TEST/modules/lang-painless/plugin-descriptor.properties
|       at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
|       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
|       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
|       at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
|       at java.base/java.nio.file.Files.newByteChannel(Files.java:375)
|       at java.base/java.nio.file.Files.newByteChannel(Files.java:426)
|       at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
|       at java.base/java.nio.file.Files.newInputStream(Files.java:160)
|       at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:199)
|       at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:449)

Do you know why they do not contain those?

[dblock]

@jmazanec15 No idea, I would open an issue to track in the repo that contains that code

[jmazanec15]

@dblock For backwards compatibility tests, I think we are blocked by #716.

We need to be able to install the old plugins on the integ test cluster to run bwc. To do that, we need to retrieve the zip from somewhere. Correct me if I am wrong, but I dont think there is anywhere to get these zips directly.

[dblock]

@dblock For backwards compatibility tests, I think we are blocked by #716.

We need to be able to install the old plugins on the integ test cluster to run bwc. To do that, we need to retrieve the zip from somewhere. Correct me if I am wrong, but I dont think there is anywhere to get these zips directly.

Is there such a thing for modules? In the distribution (e.g. 1.3.0) we include this:

modules/lang-painless
modules/lang-painless/lang-painless-1.3.0.jar
modules/lang-painless/plugin-descriptor.properties
...

[jmazanec15]

@dblock Sorry, for this I was talking about the knn plugin zip versions, not painless. For BWC, we use ARCHIVE distributions, which come with the lang-painless module installed (but don't work on Mac).

For the plugin bwc, we pass the location of the zip to get installed on the test cluster. In order to remove those checked in artifacts, we need the ability to fetch, say opensearch-knn-1.3.0.zip.

I guess we could get the release distribution tar and extract the plugin from that and either zip it or do some copy and paste in a gradle task, but I think it would be easier and less error prone to be able to grab the plugin zip from somewhere and install it on the cluster and avoid that gradle logic.

[dblock]

I think we're conflating two asks: one is to publish to maven, the other is to publish ZIPs at all. We already publish "native" plugins, so maybe we should just do that for all the other ones? I opened #1823 where we could probably easily modify https://github.com/opensearch-project/opensearch-build/blob/main/vars/promoteArtifacts.groovy#L61 to accomplish what you want.

I also included the current alternatives. For existing releases I think you could go with the "latest" approach?

[jmazanec15]

@dblock I think for existing releases a hardcode of the build number thats used for release for that release would be even better. Ill try to get that working.

[dblock]

@dblock I think for existing releases a hardcode of the build number thats used for release for that release would be even better. Ill try to get that working.

The only drawback is that those aren't "official" aka signed bits, but practically I think that doesn't make a difference.

[bbarani]

@davidlago @praveensameneni Can you please provide an ETA to close out this campaign?

[davidlago]

The PR has 2 approvals already, so merging any minute now (cc @cliu123 )

[bbarani]

Closing this issue as all the plugins have removed the checked-in binaries. Feel free to re-open if needed.