X-Pack is being deprecated in 8.0.0

[picandocodigo]

Starting in 8.0.0, the "X-Pack" integration library codebase will be merged into elasticsearch-api. The elasticsearch-xpack gem will be deprecated and all the functionality will be available from elasticsearch-api.

The xpack namespace will be removed for accessing any APIs other than _xpack (client.xpack.info) and _xpack/usage (client.xpack.usage). But APIs which were previously available through the xpack namespace e.g.: client.xpack.machine_learning will only be available through client.machine_learning.

• master: the X-Pack namespace will go away, and all endpoints will be accessible through the client when using elasticsearch-api
  • X-Pack deprecation #1316
  • Cleanup: Removes X-Pack directory #1516
• 7.x: there'll be a transition where elasticsearch-xpack will depend on elasticsearch-api and we keep that functionality working without breaking the current API.
• Add deprecation notices in elasticsearch-xpack for 7.x:
  • [DOCS] Updates xpack and api READMEs #1524
• Update documentation - README in elasticsearch, elasticsearch-api, asciidocs.
• Add deprecation note in gemspec, print a warning on installation of elasticsearch-xpack:
  • [XPACK] Adds post installation deprecation warning #1525
  • [XPACK] Shows deprecation warning first time xpack is used #1365

[nchaturvedi-personal]

Using data_stream in the conf file is giving error as below:
Traceback (most recent call last):
29: from /usr/bin/fluentd:23:in <main>' 28: from /usr/bin/fluentd:23:in load'
27: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/bin/fluentd:15:in <top (required)>' 26: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
25: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 24: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/command/fluentd.rb:352:in <top (required)>'
23: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/supervisor.rb:665:in run_supervisor' 22: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/engine.rb:80:in run_configure'
21: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/engine.rb:105:in configure' 20: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/root_agent.rb:149:in configure'
19: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:64:in configure' 18: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:64:in each'
17: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:74:in block in configure' 16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:132:in add_match'
15: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:187:in configure' 14: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/out_copy.rb:39:in configure'
13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:99:in configure' 12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:99:in each'
11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:108:in block in configure' 10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:114:in new_output'
9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:169:in new_impl' 8: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:44:in lookup'
7: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:99:in search' 6: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:99:in each'
5: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:102:in block in search' 4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2: from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:7:in <top (required)>'
1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require': cannot load such file -- elasticsearch/xpack (LoadError)
31: from /usr/bin/fluentd:23:in <main>' 30: from /usr/bin/fluentd:23:in load'
29: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/bin/fluentd:15:in <top (required)>' 28: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
27: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 26: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/command/fluentd.rb:352:in <top (required)>'
25: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/supervisor.rb:665:in run_supervisor' 24: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/engine.rb:80:in run_configure'
23: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/engine.rb:105:in configure' 22: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/root_agent.rb:149:in configure'
21: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:64:in configure' 20: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:64:in each'
19: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:74:in block in configure' 18: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/agent.rb:132:in add_match'
17: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:187:in configure' 16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/out_copy.rb:39:in configure'
15: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:99:in configure' 14: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:99:in each'
13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin/multi_output.rb:108:in block in configure' 12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:114:in new_output'
11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/plugin.rb:169:in new_impl' 10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:44:in lookup'
9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:99:in search' 8: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:99:in each'
7: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.4/lib/fluent/registry.rb:102:in block in search' 6: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
5: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 4: from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:7:in <top (required)>'
3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in require' 2: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in rescue in require'
1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in require' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-xpack-7.17.0/lib/elasticsearch/xpack.rb:43:in <top (required)>': uninitialized constant Elasticsearch::API::COMMON_PARAMS (NameError)

Not sure if the syntax has changed since upgrade to 8.0.0.

Below is the list of gems installed:
async (1.30.1)
async-http (0.54.0)
async-io (1.32.2)
async-pool (0.3.9)
benchmark (default: 0.1.0)
bigdecimal (1.4.4)
bundler (2.3.7)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.9)
console (1.14.0)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
elastic-transport (8.0.0)
elasticsearch (8.0.0)
elasticsearch-api (8.0.0, 7.17.0)
elasticsearch-xpack (7.17.0)
etc (default: 1.1.0)
excon (0.91.0)
ext_monitor (0.1.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-elasticsearch (5.1.4)
fluentd (1.14.4)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
json (2.4.1)
logger (default: 1.4.2)
matrix (default: 0.2.0)
msgpack (1.4.5)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
prime (default: 0.1.1)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby2_keywords (0.0.5)
serverengine (2.2.5)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2021.5)
uri (default: 0.10.0)
webrick (default: 1.6.1)
yajl-ruby (1.4.1)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

Also, below is the conf file syntax for reference:

@type forward port 24224 bind 0.0.0.0

<match app1** >
@type copy

@type elasticsearch
@type elasticsearch_data_stream
data_stream_name logs-prod1-app-app1
host 10.120.115.33^M
port 9200^M
..........

[picandocodigo]

@nchaturvedi-personal It appears you're trying to use elasticsearch-xpack 7.17 with version 8.0 of the client (and elasticsearch-api 8.0). Like the error says, Elasticsearch::API::COMMON_PARAMS is no longer defined in version 8.0 of elasticsearch-api.

If you want to use 8.0, you need to uninstall elasticsearch-xpack and version 7.17 of elasticsearch-api, and call the functions that were using elasticsearch-xpack from elasticsearch-api like this issue explains. Otherwise you need to use version 7.17 of the client.

[nchaturvedi-personal]

@picandocodigo - thanks for the response.
I guess my question was unclear. I was able to make my use-case work by using version 7.17.
But I was not able to understand from he documentation, what changes to make to the fluent.conf template(when using v 8.0) so that correct functions are called.
Currently, if I use the @type elasticsearch_data_stream, as mentioned in the sample above, the driver, internally, seems to want to call the elasticsearch-xpack methods.
My expectation was that since the thread does not mention any changes to the attributes used in the conf. template, just installing the gems with version 8.0, would suffice, and invoke the correct functions.

Note: Just for a bit more context, I am running a fluentd docker container(with a specific conf template), and using it to stream logs from other containers(using fluentd as the log driver), running on the same VM, to elastic serach.

[picandocodigo]

I don't know about the fluent configuration, but the error you pasted seems to come from what I described:
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in require' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-xpack-7.17.0/lib/elasticsearch/xpack.rb:43:in <top (required)>': uninitialized constant Elasticsearch::API::COMMON_PARAMS (NameError)
This is failing because the code is using elasticsearch-xpack v7.17 and elasticsearch-api v8.0, where that constant is not defined anymore.

Are you using fluent-plugin-elasticsearch? If so there's a workaround in this issue and some work has been already been committed to support version 8.0 of the client.

[nchaturvedi-personal]

@picandocodigo - Yes, I am using fluent-plugin-elasticsearch.
I did manage to get it working using version 7.17, which is one of the suggestions in the link(s) that you have provided.
Will try out the other suggestion also, using version 8.0.
Once again, thanks for the assistance.

[TMDavisB]

Hello. After installing elasticsearch-xpack-7.17.1, I get an error when try to start td-agent.

Ive using the following command to install elasticsearch-xpack-7.17.1 'td-agent-gem install elasticsearch-xpack' if that helps.

Sep 09 10:53:34 server systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sep 09 10:53:35 server fluentd[428227]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-xpack-7.17.1/lib/elasticsearch/xpack.rb:46:in <module:Elasticsearch>': Client is not a class (TypeError) Sep 09 10:53:35 server fluentd[428227]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport.rb:40: previous definition of Client was here Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-xpack-7.17.1/lib/elasticsearch/xpack.rb:45:in <top (required)>'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in require' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:158:in rescue in require'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in require' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:7:in <top (required)>'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in block in search' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in each'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in search' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in lookup'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in new_impl' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in new_output'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:72:in block in configure' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in each'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in configure' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in configure'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in configure' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in add_match'
Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in block in configure' Sep 09 10:53:35 server fluentd[428227]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in each'