Elasticseach and Kibana related warning spam after upgrade to 7.x

[mikkolehtisalo]

If the ElasticSearch has also Kibana installed, after updating ElasticSearch to 7.x Graylog logs are full of spam like this:

2020-12-17T12:46:50.780+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:46:50.781+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:46:50.789+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-system-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:46:50.791+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-system-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.776+02:00 WARN  [RestClient] request [GET http://localhost:9200/graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.777+02:00 WARN  [RestClient] request [GET http://localhost:9200/graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.793+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.799+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.804+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-system-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]
2020-12-17T12:47:00.805+02:00 WARN  [RestClient] request [GET http://localhost:9200/gl-system-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]

As it is only a warning so far this is mostly an annoyance. The real issue is that ElasticSearch 8.x will probably break Graylog if also Kibana is used.

[dennisoelkers]

Hey @mikkolehtisalo,

thanks for reporting this!
I am trying to reproduce this for a while now and was unable to do it. Installing kibana next to Graylog does not trigger the warning for me. Also, a request getting aliases for e.g. graylog2_* should not access aliases for .kibana. Recreating the requests manually by executing a GET to http://localhost:9200/graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false did not return the error for me.

Can you do the following to support fixing this issue:

• Doing a curl -v "http://localhost:9200/graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false" and report back if you see the warning header?
• Doing a curl -v "http://localhost:9200/.kibana/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false" and report back the output?

Thanks a lot!

[mikkolehtisalo]

That's odd because I haven't done any manual operations that could cause this. However the Kibana version was older (not in synch with ES!) so that might have caused something.

In any case here are the outputs:

$ curl -v "http://localhost:9200/graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false"
* About to connect() to localhost port 9200 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 9200 (#0)
> GET /graylog2_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .security, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"
< content-type: application/json; charset=UTF-8
< content-length: 82
<
* Connection #0 to host localhost left intact
{"graylog2_95":{"aliases":{"graylog2_deflector":{}}},"graylog2_94":{"aliases":{}}}

$ curl -v "http://localhost:9200/.kibana/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false"
* About to connect() to localhost port 9200 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 9200 (#0)
> GET /.kibana/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-7.10.1-1c34507e66d7db1211f66f3513706fdf548736aa "this request accesses system indices: [.kibana_2], but in a future major version, direct access to system indices will be prevented by default"
< content-type: application/json; charset=UTF-8
< content-length: 40
<
* Connection #0 to host localhost left intact
{".kibana_2":{"aliases":{".kibana":{}}}}

[BenoitPoulet]

Hello,

I have the same behavior after an update from ES 6.8 to 7.10, i have like 1 line every second.

But this is for ".security", not ".kibana"

2021-01-26T09:30:09.021+01:00 WARN [RestClient] request [GET http://pirv-siem-es-05:9200/gl-system-events_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "this request accesses aliases with names reserved for system indices: [.security], but in a future major version, directaccess to system indices and their aliases will not be allowed"]

Note : i'm not using ES oos; but the regular release as i'm using xpack plugins (ILM)

graylog-server-4.0.1-1.noarch
elasticsearch-7.10.2-1.x86_64

Outputs :
`
$ curl $es_auth -v "http://localhost:9200/graylog_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false"

• Trying ::1...
• TCP_NODELAY set
• Connected to localhost (::1) port 9200 (#0)
• Server auth using Basic with user 'elastic'

GET /graylog_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false HTTP/1.1
Host: localhost:9200
Authorization: Basic xxxxxxxxxxxxxxxxxx
User-Agent: curl/7.61.1
Accept: /

< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "this request accesses aliases with names reserved for system indices: [.security], but in a future major version, directaccess to system indices and their aliases will not be allowed"
< content-type: application/json; charset=UTF-8
< content-length: 77
<

• Connection #0 to host localhost left intact
  {"graylog_0":{"aliases":{}},"graylog_1":{"aliases":{"graylog_deflector":{}}}}

$ curl $es_auth -v "http://localhost:9200/.security/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false"

• Trying ::1...
• TCP_NODELAY set
• Connected to localhost (::1) port 9200 (#0)
• Server auth using Basic with user 'elastic'

GET /.security/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false HTTP/1.1
Host: localhost:9200
Authorization: Basic xxxxxxxxxxxxxxxxxx
User-Agent: curl/7.61.1
Accept: /

< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "this request accesses system indices: [.security-6], but in a future major version, direct access to system indices will be prevented by default"
< content-type: application/json; charset=UTF-8
< content-length: 44
<

• Connection #0 to host localhost left intact
  {".security-6":{"aliases":{".security":{}}}}

`

[dennisoelkers]

Thanks @mikkolehtisalo & @BenoitPoulet,

what's puzzling me is that an alias request for graylog_* touches .kibana or .security. I do not understand this part yet and cannot reproduce it locally. I think there is a missing piece. If you have any input clarifying this or making it reproducible for me, it would help a lot.

[mikkolehtisalo]

In my case it's possible (although I am not sure) that I have at some point installed one of the xpack bundles, and not just Kibana. The installation is old. I would guess it's from around ElasticSearch 2.x. I doubt installing recent versions reproduces this. Installing ancient versions and doing version upgrades might, but I'm unable to tell which combination triggers the issue.

Could the following (source) provide a hint why the aliases behave oddly? Just a wild guess though.

There are some dot indices that do not necessarily fit the mold of a normal
system index; instead they store data that the system produces with the intent
that users can also query against this data. For these indices we will move to
adding a hidden index that will not be resolved by default for wildcards.
These indices can be specifically requested or an IndicesOption can be specified
so that these indices are not ignored during wildcard resolution. An index may
only be marked as hidden at creation time and will be done so through the use
of an index setting.

[BenoitPoulet]

I will try to explain my setup, which is not very complicated.

I started with a fresh install of Graylog 3.3.8 and ES 6.8 (i repeat, not the OSS version)

From this repository :

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And i activate this in the ES configuration (/etc/elasticsearch/elasticsearch.yml)

# needed for  non-oos (x-pack)
# action.auto_create_index: false
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*

# https://www.elastic.co/guide/en/elasticsearch/reference/6.8/configuring-security.html
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

I found one more report of this pb here : https://community.graylog.org/t/elastic-warnings-in-graylog-server-log/18046 (but with no solution but hiding the warnings)

[mikkolehtisalo]

My guess would be checkSystemAliasAccess might report especially .security because it is implicitly used because of xpack.security is enabled and the access rights were validated using that index. I'm a bit confused though, because shouldn't every ElasticSearch request trigger this in every security enabled instance, and why also other indices are mentioned.

[victorfeng19]

I have the same issue with .security. My setup is fresh, Graylog 4.0.5, commercial ElasticSearch 7.10.1(not ElasticSearch OSS). And xpack.security is enabled. Although I temporarily hide the warning message, I have the same concern that it would be broken in ES 8.

I put this in ES config as other people suggested
#get rid of [.security] warning
http.max_warning_header_count: 0

[ghost]

I've had a similar problem with this sort of spam in my depreciation log:
/var/log/elasticsearch/graylog_deprecation.log
[2021-03-29T14:08:08,874][DEPRECATION][o.e.d.c.m.IndexNameExpressionResolver] [p-sn-es-01] this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default

I tried the suggestion above:
http.max_warning_header_count: 0
But it just moved the problem to the primary logfile where is complained that it wasn't allow to write about the warning header
Dropping a warning header, as their total count reached the maximum allowed of [0] set in [http.max_warning_header_count]!

My current solution
Note: I'm running a SINGLE node and not a cluster of Elasticsearch. Graylog (Current release) and Elasticsearch (7.10.2)on seperate servers
In the end I managed to get rid of the message by lifting the Depreciation log level to "WARN" rather than "Depreciation" as discussed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html

I changed the line 95 in the following file /etc/elasticsearch/log4j2.properties
logger.deprecation.level = deprecation
to
logger.deprecation.level = warn

Although this will of course make an administrator oblivious to probable breaking changes in the future. But for Graylog I'm following their guidelines and staying on supported versions of Elasticsearch.

[dennisoelkers]

Starting with 4.1, you can mute warnings like these with the new elasticsearch_mute_deprecation_warnings flag. So if you set:

elasticsearch_mute_deprecation_warnings = true

in your Graylog config, you should not see those anymore.

This is obviously not solving the problem. It is not causing an impact for now, but indicates a potential future breakage. We are looking into that and checking if we can do some requests differently to avoid it.

[mikkolehtisalo]

In my case running

curl -X DELETE "localhost:9200/.kibana_2/_alias/.kibana?pretty"

stopped the message spam. Still doesn't explain why though.

[overhacked]

elasticsearch_mute_deprecation_warnings = true

This doesn't work because of a typo at

graylog2-server/graylog-storage-elasticsearch7/src/main/java/org/graylog/storage/elasticsearch7/ElasticsearchFilterDeprecationWarningsInterceptor.java

Lines 31 to 36 in 553fadb

	private String[] messagesToFilter = {
	"setting was deprecated in Elasticsearch",
	"but in a future major version, directaccess to system indices and their aliases will not be allowed",
	"in epoch time formats is deprecated and will not be supported in the next major version of Elasticsearch",
	org.graylog.shaded.elasticsearch7.org.elasticsearch.common.joda.JodaDeprecationPatterns.USE_NEW_FORMAT_SPECIFIERS
	};

.

A space was omitted in directaccess, which causes it not to match the Elasticsearch 7 deprecation warning which reads in full: 299 Elasticsearch-7.15.2-93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c "this request accesses aliases with names reserved for system indices: [.security], but in a future major version, direct access to system indices and their aliases will not be allowed.