[Help Wanted] OpenSearch / OpenSearch Dashboards - Puppet modules

[smortex]

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Provide Puppet modules to manage the deployment of OpenSearch / OpenSearch Dashboard.

Describe alternatives you've considered
There are modules for ElasticSearch components managed by elastic which are available on the Puppet Forge.

The modules are licensed under the Apache 2.0 license, but Elastic still wants the contributors to sign it's CLA before submitting code to them.

The module have some problems, and very low maintenance with more than 30 open PR, most having a broken status because of a broken test suite, and some of them being very trivial (e.g. typos) 😭

Additional context
Availability of packages is I think a prerequisite. Related issues:

• OpenSearch / OpenSearch Dashboards - DEB distribution (X64) opensearch-build#28
• [Meta] OpenSearch / OpenSearch Dashboards - Support for RPM distribution opensearch-build#27

Then I can either fork the Elastic repositories of the modules and adjust them to manage OpenSearch, fix CI and move on, or start from scratch (I guess both are valid because of the Apache-2.0 license of the module).

These modules could be published under an opensearch account on the Puppet Forge.

Progress

• Adopt the elasticsearch modules in the voxpupuli organization
  • puppet-elastic-stack
  • puppet-elasticsearch
  • puppet-kibana
  • puppet-logstash
• Adjust / modernize the modules to allow the voxpupuli testing framework to work with the modules
  • puppet-elastic-stack (Modulesync voxpupuli/puppet-elastic_stack#39)
  • puppet-elasticsearch (Modulesync 5.1.0 voxpupuli/puppet-elasticsearch#1125)
  • puppet-kibana (Modulesync 5.1.0 voxpupuli/puppet-kibana#52)
  • puppet-logstash (modulesync 5.5.0 voxpupuli/puppet-logstash#423)
• Choose one of:
  • Fork the modules
  • Create a new module from scratch
• … (to be refined later)

[stockholmux]

These modules could be published under an opensearch account on the Puppet Forge.

That's not a bad idea - just reserve the account for future use.

@peterzhuamazon / @bbarani what do you think?

[peterzhuamazon]

I move this issue to the devops repo and see if community can help with this. Thanks.

[smortex]

Hey everyone!

Elastic has just transferred their modules to the voxpupuli organization:

• https://github.com/voxpupuli/puppet-elastic-stack
• https://github.com/voxpupuli/puppet-elasticsearch
• https://github.com/voxpupuli/puppet-kibana
• https://github.com/voxpupuli/puppet-logstash

The next few weeks will be busy to integrate the code base to the organization (integration with our tools and CI, triage of issues, etc).

When done we will be able to fork the repos. For now if people wants to help, they can help us with the integration of the modules within the voxpupuli organization 😉

[stockholmux]

@smortex Is it possible to just have one repo that works for both ES and OpenSearch?

[smortex]

@stockholmux a bit early to be 100% sure. For now, we are not sure if it is possible nor if it is a good idea (we think we need to be mindful of elastic's position and discontent with AWS).

We are currently curating our tools to integrate the modules in our process, when done I think we will be in a better position to see what is the preferred plan and adjust it accordingly.

I will add info here about our progress as we move on. Stay tuned!

[smortex]

This is a follow-up in order to keep the community up-to-date with the current status of this task.

Integration of the elasticsearch modules to voxpupuli ecosystem is in progress. The goal is to have working CI (unit and acceptance tests) to make in easy to integrate contributions:

• https://github.com/voxpupuli/puppet-elastic-stack is done
• https://github.com/voxpupuli/puppet-elasticsearch is in progress (Modulesync 5.1.0 voxpupuli/puppet-elasticsearch#1125)
• https://github.com/voxpupuli/puppet-kibana is in progress (Modulesync 5.1.0 voxpupuli/puppet-kibana#52)
• https://github.com/voxpupuli/puppet-logstash has not been started (I an not a user of this one so low priority for me 😄 )

When these are merged, the modules will be forked to have corresponding OS ones, tags will be removed from the forks, ES specific code will be removed, and new releases should happen at some point 🎉.

If you are able to help, feel free to do so and submit PR with enhancements, review existing PRs, discuss on voxpupuli IRC channel, etc…

[stockholmux]

@smortex How is voxpupuli thinking to manage changes between the OpenSearch fork and the existing ES stuff? Will it be a hard fork or will one be upstream?

[smortex]

I an thinking about detaching completely both projects to follow how it's done on the OpenSearch side: as far as I can see, both projects will evolve in different directions.

Both modules being apache-2 licensed, we will still be able to "import" changes in one module to the other if some change make sense to both of them. Now that the ES modules are managed by voxpupuli, I think it make sense that vox also manage the OS ones?

[spapadop]

Detaching the two projects seems like a good idea for many reasons. I'd gladly help with refactoring and bringing the OpenSearch module up to speed asap. Having the above-mentioned PRs merged, is there anything preventing you from creating the fork?

[smortex]

@spapadop main changes are merged, some minor enhancements are in progress. I use an older version of the modules at $WORK that does not allow me to test these changes in real-case scenario, but I should have some feedback next Monday from another position. I'd prefer to be 100% sure the modules are working as expected before forking to avoid handling fixes twice (but the CI with acceptance test makes me feel quite confident).

If you currently use the ES 7 modules, can you please give the latest master a try and report success / failure if you run with this code in noop mode (it should not want to change anything I suppose 😁)?

If you are on IRC, reach out on #voxpupuli on Libera, there is also Vox Pupuli on Slack.

[psaiz]

I'm moving my applications from elasticsearch to opensearch. Up to now, my hosts were configured by puppet. I've looked into the module in vox pupuli, and I have issues with the breaking changes. I do have multiple instances of elasticsearch running on the same node.

If I were to look into the module and modify it to keep allowing multiple instances, could I submit those changes to be considered to be merged into the main branch?

[smortex]

@psaiz the multi-branch idea was probably a bad idea since the package did not supported this. Elastic got to remove this to continue maintaining the puppet module, with no upgrade path… Then they abandoned the modules, and they where adopted by Vox pupuli. There is no intention to bring back multi-instances support for ES nor OS (unless this is a core feature of the future OS packages).

[ComBin]

Hello everyone! Made it for my own needs: https://github.com/ComBin/puppet-opensearch. Yes, it's a creepy Frankenstein. But it works. So while people wait official module welcome to use this.

[svanschalkwyk]

Any word on an official repo ?

[smortex]

Following-up after this PR came on the table again today:

The awesome @crazymind1337 started a new module under the voxpupuli organization:
https://github.com/voxpupuli/puppet-opensearch/

This is early work and rather limited at the moment, but it is I think the place to go for the future™ and of course your contributions are welcome 😁

[smortex]

Hello!

Following-up a few months later:

• the puppet-opensearch module has been available on the Puppet forge for some times; and
• we just released the first version of the puppet-opensearch_dashboards module: 1.0.0 🎉 !

While we do not provide types and providers to manage the many aspects of the installation that the ElasticSearch modules provided, we believe to have the foundation to build a reliable module based on users feedback and contributions. These modules are now available in a 1.x.y version (following semver), which mean we consider them ready for production!

Using these modules, a single-node cluster can be bootstrapped with minimal effort:

class { 'opensearch':
}

class { 'opensearch_dashboards':
settings => {
  'opensearch.hosts'                => [
    'https://localhost:9200',
  ],
  'opensearch.ssl.verificationMode' => 'none',
  'opensearch.username'             => 'kibanaserver',
  'opensearch.password'             => 'kibanaserver',
  'logging.dest'                    => '/var/log/opensearch-dashboards/opensearch-dashboards.log',
}

A multi-node cluster can also be bootstrapped using the modules but have a few more requirements (because you need TLS certificates for the nodes).

You are invited to test these modules, join the projects, and take an active part to their development on GitHub:

• https://github.com/voxpupuli/puppet-opensearch/
• https://github.com/voxpupuli/puppet-opensearch_dashboards/

With the 1.0.0 release of the opensearch_dashboards module, I consider the Puppet community now has a place to go when switching from ElasticSearch to OpenSearch. It will not be a drop-in replacement, but should no longer be a blocker. For this reason, I will close this issue and invite people to continue working and commenting on the respective projects.

Thank you!