Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rgw: metadata search part 2 #14351

Merged
merged 71 commits into from
Jun 2, 2017
Merged

rgw: metadata search part 2 #14351

merged 71 commits into from
Jun 2, 2017

Commits on May 30, 2017

  1. rgw: initial implementation of mdsearch query compiler 

    convert infix queries that look as follows:

[(]<name> <operator> <value> [)] [<and|or> ...]

into a prefix structure that is understood by elasticsearch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8fd55cc View commit details
    Browse the repository at this point in the history

  2. rgw: add init callback to sync modules 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    6d4733c View commit details
    Browse the repository at this point in the history

  3. rgw: define es index mapping 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    49b27f4 View commit details
    Browse the repository at this point in the history

  4. rgw: implement init_sync() callback in es module 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    67bfdb4 View commit details
    Browse the repository at this point in the history

  5. rgw: handle nested fields in es queries 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    6de586c View commit details
    Browse the repository at this point in the history

  6. rgw: move code into class 

    just cleaning up

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    86ea424 View commit details
    Browse the repository at this point in the history

  7. rgw: simplify es compile interface 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    0fdd24f View commit details
    Browse the repository at this point in the history

  8. rgw: rename rgw_rest_es.cc to rgw_es_query.cc 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    42130df View commit details
    Browse the repository at this point in the history

  9. rgw: work on REST handler for es module 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    bb46f19 View commit details
    Browse the repository at this point in the history

  10. rgw: meta search rest handler can access es module 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    e515bf3 View commit details
    Browse the repository at this point in the history

  11. rgw: rename a few methods 

    just rename calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    c9d6882 View commit details
    Browse the repository at this point in the history

  12. rgw: make key param in RGWRESTStreamRWRequest::send_request() optional 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    9f494ff View commit details
    Browse the repository at this point in the history

  13. rgw: can send data in RGWRESTStreamRWRequest::send_request() 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    952e72d View commit details
    Browse the repository at this point in the history

  14. rgw: move code around 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    b6fe05d View commit details
    Browse the repository at this point in the history

  15. rgw: extend interfaces to allow sending data 

    when retrieving REST resource

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    698763f View commit details
    Browse the repository at this point in the history

  16. rgw: send request to elasticsearch and parse response 

    and send back response to user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    86fe122 View commit details
    Browse the repository at this point in the history

  17. rgw: also dump custom meta entries 

    when doing metadata search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    0180650 View commit details
    Browse the repository at this point in the history

  18. rgw: add permissions and bucket to es query 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    99b49ac View commit details
    Browse the repository at this point in the history

  19. rgw: metadata search pagination 

    Leveraging elasticsearch "size" and "from" for doing pagination. This
is not optimal, but sadly it's currently the way to go as elasticsearch
does not have an efficient way to retrieve ordered entries (other than
using stateful api). This should be revisited in the future. A preferable
way would be to hold a unique identifier for each object doc, and have
the query sort by that unique identifier (that could be used as marker).
At the moment sorting by anything would require elasticseatch to load
all data into memory.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    1e88647 View commit details
    Browse the repository at this point in the history

  20. rgw: initial support for multiple types in es requests 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    d026a23 View commit details
    Browse the repository at this point in the history

  21. rgw: support for generic entities types in es 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    42746ac View commit details
    Browse the repository at this point in the history

  22. rgw: support more meaningful compilation error string 

    in es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    21ee7c1 View commit details
    Browse the repository at this point in the history

  23. rgw: support field aliases in es compiler 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    484fd2b View commit details
    Browse the repository at this point in the history

  24. rgw: add dout defs 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    de5bd34 View commit details
    Browse the repository at this point in the history

  25. rgw: guard restricted fields in es query 

    don't allow users to use the 'permissions' field.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    3f62f17 View commit details
    Browse the repository at this point in the history

  26. rgw: new api to configure bucket's custom keys for mdsearch 

    POST /bucket?mdsearch
x-rgw-meta-search: <x-amz-meta-key>[;<str|int|date>][,...]

note that x-amz-meta-search header can also be used.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8249f76 View commit details
    Browse the repository at this point in the history

  27. rgw: es: limit indexing to only configured user meta 

    only metadata keys that are configured on user's bucket are allowed
to be indexed. Also, use the type of meta that was configured.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    e4e6e1e View commit details
    Browse the repository at this point in the history

  28. rgw: es: also return custom int and date results 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    7b61c99 View commit details
    Browse the repository at this point in the history

  29. rgw: es: api to retrieve bucket mdsearch config 

    GET /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    acba916 View commit details
    Browse the repository at this point in the history

  30. rgw: es: api to delete bucket mdsearch config 

    DELETE /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    603863a View commit details
    Browse the repository at this point in the history

  31. rgw: es: handle get config api in es zone 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    874223b View commit details
    Browse the repository at this point in the history

  32. common/ceph_json: map type can handle 2 or 3 template arguments 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    f1e681b View commit details
    Browse the repository at this point in the history

  33. rgw: expose conf_get(), conf_get_int(), conf_get_bool() 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    6d5555a View commit details
    Browse the repository at this point in the history

  34. rgw: add "explicit_custom_meta" configurable to es sync module 

    Modify all the map<string, string> that is used to pass in module config
to map<string, string, ltstr> so that it can be used with the conf get
vals util. Also, switch to using shared_ptr to hold the ElasticConfig,
so that later when we hold maps and such it doesn't need to be copied
on every call.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    85310f0 View commit details
    Browse the repository at this point in the history

  35. rgw: create helper for configurable item selection 

    with optional prefix and suffix.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    b02d616 View commit details
    Browse the repository at this point in the history

  36. rgw: es: configurable list of buckets and owners to index 

    and a tier configuration that allows to select both list of buckets
and list of bucket owners (if both configured then will intersect
the selection).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    151a6eb View commit details
    Browse the repository at this point in the history

  37. rgw: es: elasticsearch index path unique per sync instance 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    f5b158f View commit details
    Browse the repository at this point in the history

  38. rgw: es: add a configurable to override index path 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    fbae80b View commit details
    Browse the repository at this point in the history

  39. rgw: es: configurable number of replicas and shards 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8bfe137 View commit details
    Browse the repository at this point in the history

  40. rgw: add missing custom meta map initialization 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    3f40938 View commit details
    Browse the repository at this point in the history

  41. rgw: fix time format that is sent to elasticsearch 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    4a91a54 View commit details
    Browse the repository at this point in the history

  42. rgw: es: generic keys are not case sensitive 

    make map case insensitive, and explicitly define all generic keys
so that they can be forced to be lower case.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    2c9a7bc View commit details
    Browse the repository at this point in the history

  43. rgw: es: use "null" as instance in case key.instance is empty 

    when indexing objects. We can get either.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    a680aa1 View commit details
    Browse the repository at this point in the history

  44. rgw-admin: zone create handles --tier-config 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    de9d2cb View commit details
    Browse the repository at this point in the history

  45. rgw: don't pass sync module to rest filter in creation 

    sync module instance might change due to reconfiguration. Pass it into the
handler when executing op.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    1ca77ab View commit details
    Browse the repository at this point in the history

  46. rgw: move data sync instance_id initialization to caller 

    so that caller can easily know the instance id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    6d3cc4b View commit details
    Browse the repository at this point in the history

  47. rgw: es: index and return versioned epoch 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    024d98c View commit details
    Browse the repository at this point in the history

  48. rgw/test_multi: add support for elasticsearch testing 

    Add support for different zone types, and create an elasticsearch
zone type that deals with es testing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    ba0626f View commit details
    Browse the repository at this point in the history

  49. test/rgw/test_multi: differentiate between zone and zone connection 

    Instead of having a Zone type used for the connection, create a new
ZoneConn type that represents the connection. This frees us from the
need to pass in credentials all around.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    366ffd5 View commit details
    Browse the repository at this point in the history

  50. test/rgw/test_multi: initial es functional tests 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    bbc5d48 View commit details
    Browse the repository at this point in the history

  51. rgw/test_multi/es: extend test 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    7d3be06 View commit details
    Browse the repository at this point in the history

  52. rgw/test_multi: zone_conn can hold more than one bucket per zone 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    f503762 View commit details
    Browse the repository at this point in the history

  53. rgw/test_multi: add tests for different key types 

    add int and date tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    131f6a3 View commit details
    Browse the repository at this point in the history

  54. rgw: don't send raw date header to elasticsearch 

    parse the header, and encode it in the json doc using
a format that ES can understand. Skip header if fails to
parse.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    c13ec1c View commit details
    Browse the repository at this point in the history

  55. rgw/test-multi: test more complicated queries 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    d230915 View commit details
    Browse the repository at this point in the history

  56. rgw/test_multi: add test_es_bucket_conf test 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    cd3fb79 View commit details
    Browse the repository at this point in the history

  57. test_multi: realm checkpoint after init 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    c9e2aff View commit details
    Browse the repository at this point in the history

  58. rgw: use RGW_AMZ_META_PREFIX 

    instead of defining X_AMZ_META_PREFIX

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    7e4e743 View commit details
    Browse the repository at this point in the history

  59. rgw: adjust log levels 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    3a81d09 View commit details
    Browse the repository at this point in the history

  60. rgw: minor cleanup 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    a42aad0 View commit details
    Browse the repository at this point in the history

  61. test/rgw/test_multi: cleanup 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    0b20353 View commit details
    Browse the repository at this point in the history

  62. test_multi: don't pass array as default param to constructor 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    49f92f7 View commit details
    Browse the repository at this point in the history

  63. rgw: system users override elasticsearch permission filter 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    16e07f0 View commit details
    Browse the repository at this point in the history

  64. rgw: package radosgw-es 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    ea902d1 View commit details
    Browse the repository at this point in the history

  65. rgw: es: fix system user check 

    check was inverted

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    dd83616 View commit details
    Browse the repository at this point in the history

  66. rgw: minor fixes following review 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    c3422bd View commit details
    Browse the repository at this point in the history

  67. rgw: check init_sync return code 

    fix following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    0a836ba View commit details
    Browse the repository at this point in the history

  68. rgw: get_rest_filter() delete original rest manager 

    When overriding rest manager, delete original.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8f020cc View commit details
    Browse the repository at this point in the history

  69. test/rgw: drop use of urllib 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8519dc4 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2017

  1. rgw: fix import 

    Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed May 31, 2017
    Configuration menu
    Copy the full SHA
    4aea15f View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2017

  1. qa/tasks/rgw_multisite.py: adjust zone init 

    zone is now a ZoneConn object. Also, change import to make it relative
so that qa task can locate it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
    @yehudasa
    yehudasa committed Jun 1, 2017
    Configuration menu
    Copy the full SHA
    6594d97 View commit details
    Browse the repository at this point in the history