Using vault secret for elastic user

[r1cebank]

Hi,

I noticed in the vault secrets generation a elastic user was generated but the vault secrets is not used for elasticsearch. I created the externalsecret with that reads from the vault secret and modified some of the env reference to allow that to be used.

[ricsanfre]

Thanks @r1cebank for this PR.

The proposal you are making to manage ES admin user is different from the one included in the project documentation. In the original instructions, the way to trick ECK and provide elastic user password instead the auto-generated, was to create a Secret containing the elastic password admin before creating ElasticSearch cluster resource, so ECK operator does not overwrite the secret already created.

I like your proposal and I think it is better solution since you are using ES Auth Realms to configure ES' users and roles. No tricks are used to define elastic admin passwords and defining auth file realms and roles is directly supported by ECK operator.
Moreover, this solution allows us to automatically configure different users/roles to be used with ES. Instead of user the superuser for everything we can have different users with different roles (permissions),

I would like to change and extend the scope of this PR. I think we should implement the following:

1. Keep the default superuser elastic generated by ECK as it is. As backup superadmin user with random password auto-generated.
2. Change the username in Vault to admin user. So we can have our superuser with the desired password, included in the File Realm.
3. Create a specific user and role for fluentd user. Role with only the permission required by Fluentd (create Index, index templates, ILM, ingest data). In the meantime, fluentd could use the elastic default superuser.
4. Update documentation accordingly with the proposed changes

What do you think?

Can you please update this PR to address at least points 1) and 2)?. It would be great If you can also address 3) to define the fluentd specific role and user. If not we can leave point 3) activities for another PR, but please roll-back your changes in fluentd config to keep on using elastic user. I can take over the tasks to fully implement 3) and 4) in other PRs.

[r1cebank]

Thanks @ricsanfre, these are all very good suggestions. I have applied the suggestions you mentioned, I also created an user for prometheus exporter with the minimal permission needed. The docs has also been updated to reflect the change.

[r1cebank]

Thanks for merging the PR, just fyi I did discover one issue during my testing on my cluster, it appears logging and metrics disappear after a while.

What is happening is the fluentd containers are using a different password after a while, i suspect it could be due to the password was auto generated and during reapply it auto generated again?

@ricsanfre do you have some insights on this?

[ricsanfre]

Hi @r1cebank.

Yes, it is ArgoCD the one that is doing that. It is executing helm template commad during synch process. And since you put the password to be randomly generated when deploying the helm chart, every time a synch is executed a new Secret is generated containing a new random password.

We need to put those users ad credentials in vault. And define ExternalSecrets instead of Secrets. Random passwords can be auto-generated with Ansible (create_vault_credentials.yml)

Regards
Ricardo

[r1cebank]

Hi @ricsanfre, i didn't go through with adding the external secrets to the vault since it wasn't something we care, I'll put them as external secrets in another pr then.

[ricsanfre]

In vault.yml.j2 the structure should be something like this

  # elasticsearch and fluentd
  logging:
    es-admin:
      user: admin
      password: {{ elasticsearch_admin_password }}
    es-fluentd:
      user: fluentd
      password: {{ elasticsearch_fluentd_password }}
    es-prometheus:
      user: prometheus
      password: {{ elasticsearch_prometheus_password }}
    fluentd:
      shared_key: {{ fluentd_shared_key }}

And since it is changed secret name for elastic admin user. From elastic to es-admin, corresponding elasticsearch-admin-externalsecret.yml need to be updated

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: elasticsearch-admin-externalsecret
  namespace: {{ .Release.Namespace }}
spec:
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: es-admin-user-file-realm
    template:
      type: kubernetes.io/basic-auth
      data:
        username: |-
          {{ `{{ .username | toString }}` }}
        password: |-
          {{ `{{ .password | toString }}` }}
        roles: superuser
  data:
  - secretKey: username
    remoteRef:
      key: logging/es-admin
      property: user
      conversionStrategy: Default # ArgoCD sync issue
      decodingStrategy: None # ArgoCD sync issue
  - secretKey: password
    remoteRef:
      key: logging/es-admin
      property: password
      conversionStrategy: Default # ArgoCD sync issue
      decodingStrategy: None # ArgoCD sync issue

[ricsanfre]

Hi @r1cebank

Changes to fix ArgoCD sync issue already merged into master branch.

Regards
Ricardo