Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mimic: rgw: multiple es related fixes and improvements #26517

merged 19 commits into from Mar 19, 2019


None yet
6 participants
Copy link

commented Feb 19, 2019

Backport of #26106


  • add support for storage_class field (relevant after tiering work)
  • add support for != operator (improvement)
  • don't fail init if index already exists
  • send content-type header to elasticsearch when querying (needed for v6+)
  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

theanalyst and others added some commits May 4, 2018

rgw: es: introduce ES types
This commit introduces an enum mapping various ES types. Also encode_json now
has a new member for string type so that this can be changed easily when
upgrading ES versions for eg. wherein new ES engines do not support the string
type anymore.

Signed-off-by: Abhishek Lekshmanan <>
rgw: rgw_cr_rest: allow headers to be specified in Put CRs
Adding another constructor overload for cases when we need to override the
default headers

Signed-off-by: Abhishek Lekshmanan <>
rgw: es: introduce new classes for storing ES version info
Getting the basic information from elasticsearch, while we're only currently
interested in elasic search version, other data such as the ES cluster name etc
may be useful in the future as well.

Signed-off-by: Abhishek Lekshmanan <>
rgw: es: support for ES 5.0+
from Elasticsearch 5.0+ the type string is deprecated, while it is still allowed
in 5.0, it returns a 400 error on 6.0 clusters as this type is fully removed. We
now determine the es version while initializing the cluster from elasticsearch's
default endpoint and use that to determine what string type to use. This way we
support both 2.x and 5.x,6.x es versions as we default to string type for
clusters < 5.0

Signed-off-by: Abhishek Lekshmanan <>
rgw_cr_rest: support extra headers for get resource crs
Adding support to add additional headers as a part of GET and READ requests.

Signed-off-by: Abhishek Lekshmanan <>
rgw: es: support username and password for ES
For ES endpoints terminated with a username and password, either via xpack or
fronted by another webserver with http basic auth, we now support "username" and
"password" configurable which should be capable of doing HTTP basic

Signed-off-by: Abhishek Lekshmanan <>
rgw: ES sync module: drop ESVersion struct
using std::pair as the version struct. Another decoder class is currently
introduced as we have a nested json type for version right now.

Signed-off-by: Abhishek Lekshmanan <>
rgw: http_client: propagate send_data_hint differently

send_data_hint is passed so that the lower http client layer
to know in in cases where it's an http request that usually doesn't
send data (e.g., GET) that the request will actually send data,
and initialize the curl client correctly. In a case that was
used by the elasticsearch mechanism it wasn't passed correctly.
Now made it a data member in RGWHTTPClient, which also cleans
the interface a bit.

Signed-off-by: Yehuda Sadeh <>
rgw: ES sync: be more restrictive on object system attrs
This commit drops system attributes like url-keys, nfs-ganesha unix keys,
encryption keys and other system attributes when uploading metadata to ES.
This also brings in a tiny change where the attribute substring is no longer
compared and we compare the whole attribute name against the constants in

Signed-off-by: Abhishek Lekshmanan <>
rgw: need to give a type in list constructor
Clang complains about the missing type:
/home/jenkins/workspace/ceph-master/src/rgw/ error: no viable constructor or deduction guide for deduction of template arguments of 'initializer_list'
  static constexpr std::initializer_list rgw_sys_attrs = {RGW_ATTR_PG_VER,
/usr/include/c++/v1/initializer_list:59:28: note: candidate function template not viable: requires 1 argument, but 7 were provided

Tried several type definitions, but this was the only one that make
Clang happy.

Signed-off-by: Willem Jan Withagen <>
rgw: es: fix attrs trimming

Since we don't necessarily keep attr headers with null terminating char,
need to be careful when trying to trim them.

Signed-off-by: Yehuda Sadeh <>
rgw: es: fix content_type queries

ES attribute is stored as content_type, while we were searching
for contenttype.

Allow searching by contenttype, content_type, and also lastmodified,
and last_modified.

Signed-off-by: Yehuda Sadeh <>
rgw: es: add support for searchable storage_class field
Note that this will not add field to existing index field mapping.

Signed-off-by: Yehuda Sadeh <>
rgw: es: add support for != operator
Signed-off-by: Yehuda Sadeh <>
rgw: es: fix v5+ index map settings
and rework the way it's being done

Signed-off-by: Yehuda Sadeh <>
rgw: rest cr: can return json decoded errors
Signed-off-by: Yehuda Sadeh <>
rgw: es: don't error out on index initialization
if index already exists. Assuming index was created externally.

Signed-off-by: Yehuda Sadeh <>
rgw: es: query sent to elasticsearch needs content-type
Signed-off-by: Yehuda Sadeh <>
rgw: es: use default request headers for query requests
Needed for sending auth headers.

Signed-off-by: Yehuda Sadeh <>

@yehudasa yehudasa changed the title Wip rgw es fixes mimic mimic: rgw: multiple es related fixes and improvements Feb 19, 2019

@smithfarm smithfarm removed the backport label Mar 12, 2019

@smithfarm smithfarm modified the milestones: jewel, mimic Mar 12, 2019

@smithfarm smithfarm requested review from theanalyst and cbodley Mar 12, 2019

@smithfarm smithfarm added the needs-qa label Mar 12, 2019


This comment has been minimized.

Copy link

commented Mar 14, 2019

@yuriw yuriw merged commit 37a1139 into ceph:mimic Mar 19, 2019

4 checks passed

Docs: build check OK - docs built
Signed-off-by all commits in this PR are signed
Unmodified Submodules submodules for project are unmodified
make check make check succeeded

This comment has been minimized.

Copy link

commented May 6, 2019

@yehudasa We forgot git cherry-pick -x here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.