Proposed Xpack Design for Ansible Role

[gingerwizard]

The following proposes a design for XPack support in the role. This should allow us to handle plugins in the future which have similar complexity to shield. Feedback requested.

The XPack role will be limited to ES 2.3+ due to Shield changes. This implemented the native realm and renamed the esusers realm to file. Configuration for the xpack will be defined under the "files" directory of the role, with a subdirectory per plugin e.g.

/files/xpack/shield
/files/xpack/watcher
/files/xpack/marvel-agent
/files/xpack/license.txt - contains license details. Loaded if presence.

If a global param xpack_plugins will be provided where the user can list the plugins to be installed. If not empty, we will install the license plugin (and load the license from license.txt). The relevant directories above will in turn be processed.

Any relevant templates to adhere to a similar structure.

Shield
Initially proposing supporting the file and native based realms only. Other realms should be relatively easy to add.

For shield, I'm hesitant to user external modules such as this. These would introduce a dependency we then rely on and have to maintain.

The feature would where possible use core ansible modules, but will likely require some python with accompanying tests.

Under files/xpack/shield sub directory i propose a the following configuration files:

The feature would consume the following 3 files:

1. shield-users.yml - Likely a custom format allowing a list of users to be specified. For each user a password, list of roles and realm (either native/file) could be specified. The realm would denote how the user is added i.e. either via a REST call or using the command line tools to add the users to the file based realm.
2. roles.yml - A yml file containing roles to load into shield adhering the format of the normal roles.yml. Initially this will simply be copied to the instances config directory. Later we may parse this and invoke REST calls. The roles.yml file is monitored so this shouldn't require a restart. Roles are realm independent but copying the file seems simplest for now.
3. shield.yml - Any yml that is specific to shield for merging into Elasticsearch.yml on each node.

Other plugins

Initially the feature would just install the plugins requested. Custom actions per plugin TBD.

[gingerwizard]

@jakommo @tylerjl @dliappis @barryib @abs please review

[gingerwizard]

Modifying above for shield configs. Proposing that configs be realm specific.

• /xpack/shield/native/users.json - Same format as the REST interface here. Just a list of users to add with each a http call.
• /xpack/shield/native/roles.yml - Same format as the REST interface here. Just a list of roles to add with each a http call.
• /xpack/shield/file/roles.yml - normal roles.yml format for file realm. Copied to machine.
• /xpack/shield/file/users.yml - this will be just variable syntax and loaded through a variable lookup
• /xpack/shield.yml - Any yml that is specific to shield for merging into Elasticsearch.yml on each node.

This largely avoids any custom formats to parse. Discussion as to where these files should live. /templates seems appropriate as some need to be copied but they also need modifying.

[barryib]

@gingerwizard ,

I understand your hesitation about using external module, and I don't have actually a strong opinion on this. But what I like with modules, is the idempotence.

For XPACK support, I think it's a good idea. The directory structure is clear. The bad point (in my opionion) is that we will obliged to put configuration in different files. This is good for system administration, but I don't think if it'll ease the playbook usage.

Here is what I'm actually doing in our environment:
I create theses variables in my vars file (in one location).

ES users to create

es_users:
  - user: kibana4_server
    roles:
      - kibana4_server
  - user: my_second_user
    roles:
      - my_role_1
      - my_role_2

ES roles to create

es_roles:
  - name: my_kibana_user
    cluster: ['monitor']
    indices:
      - { 'names': ['*'], 'privileges': ['view_index_metadata', 'read'] }
      - { 'names': ['.kibana*'], 'privileges': ['all'] }
  - name: my_role_1
    cluster: ['all']
    indices:
      - { 'names': ['exemple_1_*'], 'privileges': ['all'] }
  - name: my_role_2
    cluster: ['all']
    indices:
      - { 'names': ['exemple_2_*'], 'privileges': ['all'] }

These vars are in the REST format of shield role and user you talked about.

Shield configuration

I wrote directly into es_config

[gingerwizard]

@barryib any implementation defn needs to be idempotent. I suspect we will use the ansible elasticsearch-plugin module for actual installation therefore. Any configuration will likely be done separately however - and again should be idempotent. I like the above format and the idea of centralising all users and roles in two single variables. Maybe we add for each role/user a realm variable to indicate how it should be processed? The alternative is the above format but es_users_native and es_users_file for example.

Other thoughts:

• File based users are simple as they don't require ES to be started. We can parse the above users and simply invoke the es_users tool. This however, wont be the primary means of user management moving forward
• Native users are trickier. After plugin installation ES will need to be restarted and then the calls made. At the moment we install the plugins and then restart. Installation of shield will mean install->restart->API calls. its not ideal IMO. Alternatively, we do the configuration on a call back handler to minimise restarts. If shield is already installed, and we are just updating users, then we can skip this restart. Thoughts appreciated.
• I want to focus on the broader Xpack installation as later versions of ES will change this drastically and like shield will require similar custom installation. Installation of shield isn't like to be standalone for example. A shield role is therefore probably NA. The elasticsearch-plugin module is however relevant.

[gingerwizard]

@barryib what about - similar structure for roles as well.

es_users:
  - native:
    - user: kibana4_server
      password: password123
      roles:
        - kibana4_server
    - user: my_second_user
      roles:
        - my_role_1
        - my_role_2
  - file:
    - user: simpleUser
      password: password123
      roles:
        - super_user
    - user: my_third_user
      roles:
        - my_role_1
        - my_role_2

[barryib]

@gingerwizard,

Humm, I like that es_users format 👍.

I think I didn't correctly understand your point with the directory structure:

/files/xpack/shield
/files/xpack/watcher
/files/xpack/marvel-agent

I just figured out that you were talking about task files. I thought that you were talking about vars structure. My bad.

LGTM

[gingerwizard]

Ok @barryib lets go with that. It makes it easy to manage other realms as we add them. Same for roles. Its worth noting that roles aren't actually associated with a realm - it will just effect the way they are added.

We can treat the role_mapping.yml file as a variable as well this way.

[barryib]

Yes @gingerwizard, absolutely. For role mapping we can use something like this:

# ES roles mapping
es_roles_mapping:
  admin:
    - "CN=GDL-APP-ES-ADMIN,ou=ELASTICSEARCH,ou=APPLI,ou=GROUPS"
  user:
    - "CN=GDL-APP-ES-USER,ou=ELASTICSEARCH,ou=APPLI,ou=GROUPS"
  kibana4:
    - "CN=GDL-APP-ES-KIBANA-USER,ou=ELASTICSEARCH,ou=APPLI,ou=GROUPS"

And I use {{ es_roles_mapping | to_nice_yml }} to generate the role_mapping.yml

[gingerwizard]

I will make a pull request hopefully today with shield and xpack support.

[barryib]

Awesome. +1000

Regards,
Thierno

Le 24 juil. 2016 16:16, "Dale McDiarmid" notifications@github.com a
écrit :

I will make a pull request hopefully today with shield and xpack support.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#124 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhnf1apcEL5iKXpqWZo_OtEqN-RAtMgks5qY3PMgaJpZM4JRBE6
.

[gingerwizard]

#129

[gingerwizard]

Merged in #129