Operator security: for tls to work there is a dependance on securityConfigSecret.

[prudhvigodithi]

For operator to add tls setting as follows:

    tls:
      transport:
        generate: true
      http:
        generate: true

There is a dependance on

  security:
    config:
      securityConfigSecret:
##Pre create this secret with required roles and security configs
       name: <secret_name>

If only TLS is added
Error

ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:02,622][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-2] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2022-03-30T17:47:03,001][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)
[2022-03-30T17:47:03,004][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)
[2022-03-30T17:47:05,500][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)
[2022-03-30T17:47:05,503][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)
[2022-03-30T17:47:08,001][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)
[2022-03-30T17:47:08,004][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-2] Not yet initialized (you may need to run securityadmin)

Background:
OpenSearch once TLS is added for Node Transport and HTTP rest API, the embedded security plugin creates.opendistro_security index to enable security settings, for this the securityadmin.sh has to run to load new settings, else the demo install_demo_configuration.sh file will run by default if the TLS setting is not added (If you do not configure anything opensearch will use included demo TLS certificates that are not suited for real deployments.)

curl -k https://localhost:9200/_cat/indices -u admin:admin
green open security-auditlog-2022.03.29 SHZ_xtRBTGub4NFhbtugSw 1 1 7 0 116.4kb 96.8kb
green open .kibana_1                    UOntE6z9Soa73BSdk3JI5Q 1 1 0 0    416b   208b
green open .opendistro_security         RYmlNkB5RgWAKMZU3_S05Q 1 2 9 0 178.1kb 59.3kb

With the current setup from the PR https://github.com/Opster/opensearch-k8s-operator/pull/61/files#diff-190387233823a104ed9004f0cba248cf0aa504090c923cad3be1a901bd01e99f
the securityadmin.sh will be called by a kubernetes batch job.

securityadmin.sh need to run when we add tls or custom secrets and securityadmin.sh should also run when we add new config files.

With just adding TLS setting does not run the batch job, the following is seen in logs, as once TLS is added to operator opensearch.yml is already modified with Security settings, so the Demo Installer will quit

OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.2.3
Detected OpenSearch Security Version: 1.2.3.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
sed: cannot rename /usr/share/opensearch/config/seddRF6sR: Device or resource busy
Enabling OpenSearch Security Plugin

To move forward, we need to add securityConfigSecret for the security plugin to pick up TLS and passed in roles example as https://github.com/opensearch-project/security/tree/main/securityconfig
A Readme doc on configuring this setup would be helpful.
Once added

security:
   config:
     securityConfigSecret:
##Pre create this secret with required roles and security configs
      name: securityconfig-secret
   tls:
     transport:
       generate: true
     http:
       generate: true

To job runs to call securityadmin.sh

And now I can see all pods coming up

[dbason]

As discussed on Slack I think we should only mount the secrets if there is user provided config for the files, otherwise we should use the defaults from the Opensearch image.

If we do decide to maintain our own copy of the default files these should be embedded into the Operator, and we should have unit tests to confirm that the secrets are created correctly.

[shahar35]

Fixed