Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beats With ILM Enabled Failed To check For Alias: 403 #694

Open
Alsheh opened this issue May 7, 2019 · 9 comments

Comments

Projects
None yet
4 participants
@Alsheh
Copy link

commented May 7, 2019

Auditbeat with ILM Enabled failed to check for Alias: 403

{
  "level": "error",
  "timestamp": "2019-05-07T01:20:53.284Z",
  "caller": "instance/ilm.go:80",
  "message": "Failed to check for alias: 403 Forbidden: : "
}

Auditbeat version: 6.7.1
ES version: 6.7.1
SG version: 25.0

@Alsheh Alsheh changed the title Auditbeat With ILM Enabled Failed To check For Alias: 403 Beats With ILM Enabled Failed To check For Alias: 403 May 8, 2019

@jochenkressin jochenkressin self-assigned this May 9, 2019

@jochenkressin

This comment has been minimized.

Copy link
Collaborator

commented May 9, 2019

Can you please add your beats configuration? Also, can you add the ES log file when the error is happening? That should reveal some more detailed information about why this is happening.

@floragunncom

This comment has been minimized.

Copy link
Owner

commented May 9, 2019

I guess ILM needs the "indices:admin/aliases*" permission for the indices which should be managed

@Alsheh

This comment has been minimized.

Copy link
Author

commented May 9, 2019

ES logs:

...
[2019-05-09T22:17:01,777][WARN ][c.f.s.a.BackendRegistry  ] [es-client-03] Authentication finally failed for beats_system from <masked-remote-ip-address>:43394
[2019-05-09T22:12:35,046][INFO ][c.f.s.p.PrivilegesEvaluator] [es-client-03] No index-level perm match for User [name=beats_system, roles=[beats], requestedTenant=null] Resolved [aliases=[*], indices=[*], allIndices=[*], types=[*], originalRequested=[], remoteIndices=[]] [Action [indices:admin/aliases/get]] [RolesChecked [sg_xp_monitoring, sg_own_index, sg_beats_writer]]
...

sg_beats_writer permissions:

sg_beats_writer:
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - indices:admin/template/get
    - indices:admin/template/put
  indices:
    '*beat*':
      '*':
        - WRITE
        - CREATE_INDEX
        - MANAGE_ALIASES

Auditbeat relevant config:

...
output.elasticsearch:
  hosts: '${ES_HOSTS}'
  username: '${ES_USER}'
  password: '${ES_PWD}'
  index: 'auditbeat-${ENVIRONMENT_NAME}-%{+yyyy.MM.dd}'
  ilm.enabled: true
  ilm.rollover_alias: "auditbeat-${ENVIRONMENT_NAME}"
  ilm.pattern: "{now/d}-000001"
...

ES Template:

{
    "index_patterns" : "auditbeat-<ENVIRONMENT_NAME>",
    "order" : 2,
    "settings": {
	"index.lifecycle.name": "auditbeat"
    }
}

The auditbeat ILM policy was created on Kibana.

@Alsheh

This comment has been minimized.

Copy link
Author

commented May 9, 2019

Related issue: elastic/beats#10421

@floragunncom

This comment has been minimized.

Copy link
Owner

commented May 12, 2019

@Alsheh so can we close this one?

@Alsheh

This comment has been minimized.

Copy link
Author

commented May 12, 2019

@floragunncom thanks for the follow up! In order to set up ILM policies , the following pre-defined Elastic privileges are required:

  1. manage_ilmmanage_index_templates, and monitor on cluster.
  2. manage on indices.

I wonder what would be the SG privilege equivalent for manage_ilm to make ILM work?

@floragunncom

This comment has been minimized.

Copy link
Owner

commented May 13, 2019

Can you try

sg_logstash:  
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - "indices:admin/template/*"
    - "indices:admin/ilm/*"
    - cluster:admin/ingest/pipeline/put
    - cluster:admin/ingest/pipeline/get
  indices:
    'logstash-*':
      '*':
        - CRUD
        - CREATE_INDEX
        - MANAGE
    '*beat*':
      '*':
        - CRUD
        - CREATE_INDEX
        - MANAGE
@Alsheh

This comment has been minimized.

Copy link
Author

commented May 23, 2019

@floragunncom the issue persists with the permissions you provided.

@floragunncom

This comment has been minimized.

Copy link
Owner

commented May 24, 2019

@pablolescotti can you have a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.