Package `elastic_integration` plugin. #15769

mashhurs · 2024-01-10T01:19:09Z

Release notes

[rn:skip]

What does this PR do?

Introduces a feature (based on oss-skip flag of the plugin metadata) to exclude plugins in OSS distributions
Sets elastic_integration plugin as a default plugin

History

CI fails were due to elastic/logstash-filter-elastic_integration#114

We face error if make the plugin as a default now because the TarWriter (in assembleOssTarDistribution gradle task) complains on the path which cannot be longer than 256. The workaround is to shorten the JAR path in th plugin, the PR: Updating JAR output path to avoid using long path. logstash-filter-elastic_integration#114. We need to merge the PR and release new version of the plugin.

Why is it important/What is the impact to the user?

Non-user affected.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
[] ~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files (and/or docker env variables)~~
~~I have added tests that prove my fix is effective or that my feature works~~

Author's Checklist

Before merging:

Merge the plugin PR: Updating JAR output path to avoid using long path. logstash-filter-elastic_integration#114
Run DRA on UI (Package elastic_integration plugin. #15769/merge)
- DRA test

How to test this PR locally

Pull this change and run rake artifact:archives_oss (or any of [rpm_oss, deb_oss. docker_oss]) command.
Commands generate OSS distributions under build. If we unzip and check the dependencies, we don't include oss-excluded plugins.

NOTE: currently gradle tasks (./gradlew clean assembleOssTarDistribution, ./gradlew clean assembleOssZipDistribution) to build OSS distros are broken, and the reason looks like this issue

Related issues

Use cases

Screenshots

Logs

mashhurs · 2024-01-10T01:22:21Z

rakelib/plugin.rake

@@ -61,6 +61,16 @@ namespace "plugin" do
    task.reenable # Allow this task to be run again
  end

+  task "install-default-oss" => "bootstrap" do


review note: I am okay to add [:oss] as an argument and exclude OSS except plugins but I prefered to separate to make clearly visible. When we add an argument to the task, multiple (ex: prepare -> install-default) tasks should carry this argument.

rakelib/plugins-metadata.json

yaauie · 2024-01-11T20:02:46Z

I think in terms of our dependency graph, it may be better to start with the complete list, and then uninstall/deactivate the non-OSS-licensed plugins. This would ensure that the graphs are otherwise identical.

mashhurs · 2024-01-13T00:10:37Z

rakelib/artifacts.rake

@@ -378,6 +378,7 @@ namespace "artifact" do

  task "generate_build_metadata" do
    require 'time'
+    require 'tempfile'


review note: when firing this rake task alone, we get uninitialized tempfile error.

lib/pluginmanager/remove.rb

mashhurs · 2024-01-25T15:47:40Z

buildkite test this again

mashhurs · 2024-01-25T18:22:58Z

Retried CI, now 🟢

rakelib/plugin.rake

…SS exlucluded plugin list.

…s change removes the stayed files.

…ugin list.

…nstalled gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.

…nce we don't pack the cache.

elastic-sonarqube · 2024-01-30T07:56:10Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
0.0% 0.0% Coverage on New Code
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-01-30T08:05:54Z

💚 Build Succeeded

Buildkite Build
Commit: fce5a53

History

💚 Build #791 succeeded 770a1b0
💔 Build #790 failed 38ccd1f
💚 Build #777 succeeded 214fbcd
💚 Build #767 succeeded 71c6f5e
💛 Build #765 was flaky 0028e1a
💔 Build #755 failed 0028e1a

cc @mashhurs

mashhurs · 2024-01-30T18:06:32Z

@jsvd recap these changes:

When removing plugins (bin/logstash-plugin remove {plugin}) we have a Bundler::setup source, it resets all dependency graph. If we have multiple rake tasks running bundle in a sequence (ex: bootstrap, default plugins install) after plugin remove task, we end up getting error says Bundler::GemNotFound: Could not find rubocop-1.60.2... (example failed DRA). Worth noting here that, we hadn't a logic to remove plugin(s) during the distro building.
- So, the origin issue not with bundle but with Gem::Specification. In case of locally installed gems, unless we fully reload the dependency graph (Gemfile), Gem::Specification doesn't recognize the local gems. That's the reason Bundle::setup was placed. I have changed the logic, in a way that make sure that the plugin in our local gemset. Succeeded DRA
Order of the distro build matters: it is weird that if build non-OSS and then OSS (in the artifact:build tasks), maybe because of bundler cache default installed plugins stays same as non-OSS. And I confirmed (excluded some plugins in deb with default_plugin - a_plugin) that it is not due to above plugin remove change, our default plugins were constant so far, we didn't know I guess. I will add a comment if we don't have a better way, I also hate keep such unrelated contextual orders. Wonder if you have any ideas.

jsvd · 2024-01-31T12:32:18Z

rakelib/plugin.rake

+    puts("[plugin:remove-non-oss-plugins] Removing non-OSS plugins")
+
+    LogStash::RakeLib::OSS_EXCLUDED_PLUGINS.each do |plugin|
+      remove_plugin(plugin)


Just a note that for now this approach works but if in the future any non-OSS plugin has a dependency, we'll have one or two issues to be aware of:

non-OSS may introduce dependencies that will not be used by Logstash

if the non-OSS brings a non-OSS dependency we'll have to remove it as well.

jsvd

LGTM

mashhurs · 2024-02-14T15:11:00Z

@logstashmachine backport 8.13

* Exclude plugins feature in OSS distributions. * Set elastic_integration plugin default. * Remove non-OSS plugins after installing default plugins. * Testing local can't find gem bundler (= 2.3.26) issue. * Include extract non-OSS plugins logic indocker build operations. * Only default plugins can be excluded from OSS distros. * Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list. * Gem and specification files still stay after removing the plugin. This change removes the stayed files. * Rename oss-exclude to skip-oss to align namings with other params. * Make intersection method simpler. * [Test] Temporary excluding elastic integration plugin from default plugin list. * Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle. * Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache. (cherry picked from commit 3c9db65)

* Exclude plugins feature in OSS distributions. * Set elastic_integration plugin default. * Remove non-OSS plugins after installing default plugins. * Testing local can't find gem bundler (= 2.3.26) issue. * Include extract non-OSS plugins logic indocker build operations. * Only default plugins can be excluded from OSS distros. * Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list. * Gem and specification files still stay after removing the plugin. This change removes the stayed files. * Rename oss-exclude to skip-oss to align namings with other params. * Make intersection method simpler. * [Test] Temporary excluding elastic integration plugin from default plugin list. * Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle. * Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache. (cherry picked from commit 3c9db65) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>

mashhurs self-assigned this Jan 10, 2024

mashhurs commented Jan 10, 2024

View reviewed changes

rakelib/plugins-metadata.json Outdated Show resolved Hide resolved

roaksoax added the status:work-in-progress label Jan 10, 2024

mashhurs marked this pull request as ready for review January 10, 2024 20:45

mashhurs changed the title ~~Exclude plugins feature in OSS distributions.~~ Package elastic_integration plugin. Jan 10, 2024

roaksoax added status:needs-review and removed status:work-in-progress labels Jan 10, 2024

mashhurs commented Jan 13, 2024

View reviewed changes

mashhurs commented Jan 14, 2024

View reviewed changes

lib/pluginmanager/remove.rb Show resolved Hide resolved

mashhurs force-pushed the oss-excluded-plugin-feature branch from 96a9e3b to 1f28cbf Compare January 15, 2024 23:49

mashhurs requested review from yaauie and jsvd January 16, 2024 19:04

mashhurs mentioned this pull request Jan 22, 2024

Logstash Improvements. elastic/elastic-package#1637

Merged

mashhurs commented Jan 29, 2024

View reviewed changes

rakelib/plugin.rake Show resolved Hide resolved

mashhurs added 12 commits January 29, 2024 16:57

Exclude plugins feature in OSS distributions.

d2399bb

Set elastic_integration plugin default.

7aefb74

Remove non-OSS plugins after installing default plugins.

76db46b

Testing local can't find gem bundler (= 2.3.26) issue.

52d63d3

Include extract non-OSS plugins logic indocker build operations.

533dd0f

Only default plugins can be excluded from OSS distros.

4ea06d9

Simplification: instead conditional check, use intersection to make O…

22587ae

…SS exlucluded plugin list.

Gem and specification files still stay after removing the plugin. Thi…

16bb1f4

…s change removes the stayed files.

Rename oss-exclude to skip-oss to align namings with other params.

902e6e9

Make intersection method simpler.

21a6342

[Test] Temporary excluding elastic integration plugin from default pl…

8fe70f6

…ugin list.

Sets elastic_integration plugin default back. When removing locally i…

770a1b0

…nstalled gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.

mashhurs force-pushed the oss-excluded-plugin-feature branch from 38ccd1f to 770a1b0 Compare January 30, 2024 00:58

Test the build order, remove plugin from cache logic seems invalid si…

fce5a53

…nce we don't pack the cache.

jsvd reviewed Jan 31, 2024

View reviewed changes

kaisecheng mentioned this pull request Feb 13, 2024

bump lock file for 8.13 #15939

Merged

jsvd self-requested a review February 13, 2024 16:43

jsvd approved these changes Feb 14, 2024

View reviewed changes

roaksoax added status:approved and removed status:needs-review labels Feb 14, 2024

mashhurs merged commit 3c9db65 into elastic:main Feb 14, 2024
5 checks passed

mashhurs deleted the oss-excluded-plugin-feature branch February 14, 2024 15:10

github-actions bot mentioned this pull request Feb 14, 2024

Backport PR #15769 to 8.13: Package elastic_integration plugin. #15944

Merged

6 tasks

github-actions bot added the v8.13.0 label Feb 14, 2024

roaksoax removed the status:approved label Feb 14, 2024

mashhurs mentioned this pull request Mar 7, 2024

Docs for new logstash-filter-elastic_integration plugin are not being picked up for LSR elastic/docs-tools#105

Closed

kaisecheng mentioned this pull request Mar 25, 2024

Release notes for 8.13.0 #15978

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package `elastic_integration` plugin. #15769

Package `elastic_integration` plugin. #15769

mashhurs commented Jan 10, 2024 •

edited

mashhurs Jan 10, 2024

yaauie commented Jan 11, 2024

mashhurs Jan 13, 2024

mashhurs commented Jan 25, 2024

mashhurs commented Jan 25, 2024

elastic-sonarqube bot commented Jan 30, 2024

elasticmachine commented Jan 30, 2024

mashhurs commented Jan 30, 2024

jsvd Jan 31, 2024

jsvd left a comment

mashhurs commented Feb 14, 2024

Package elastic_integration plugin. #15769

Package elastic_integration plugin. #15769

Conversation

mashhurs commented Jan 10, 2024 • edited

Release notes

What does this PR do?

History

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mashhurs Jan 10, 2024

Choose a reason for hiding this comment

yaauie commented Jan 11, 2024

mashhurs Jan 13, 2024

Choose a reason for hiding this comment

mashhurs commented Jan 25, 2024

mashhurs commented Jan 25, 2024

elastic-sonarqube bot commented Jan 30, 2024

Quality Gate passed

elasticmachine commented Jan 30, 2024

💚 Build Succeeded

History

mashhurs commented Jan 30, 2024

jsvd Jan 31, 2024

Choose a reason for hiding this comment

jsvd left a comment

Choose a reason for hiding this comment

mashhurs commented Feb 14, 2024

Package `elastic_integration` plugin. #15769

Package `elastic_integration` plugin. #15769

mashhurs commented Jan 10, 2024 •

edited