Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: frontend subsystem rework #10767

Merged
merged 76 commits into from Nov 1, 2016

Commits on Oct 21, 2016

  1. rgw: add submodule for Beast header-only library

    Beast is a c++ http protocol library build on top of boost::asio
    
    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    273e540 View commit details
    Browse the repository at this point in the history
  2. rgw: add skeleton for boost::asio frontend

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c5543a4 View commit details
    Browse the repository at this point in the history
  3. vstart: add --rgw_frontend option

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    2c48dc6 View commit details
    Browse the repository at this point in the history
  4. rgw: implement Frontend interface for asio

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c50bdf3 View commit details
    Browse the repository at this point in the history
  5. rgw: RGWEnv::set takes boost::string_ref

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    52d511e View commit details
    Browse the repository at this point in the history
  6. rgw: asio frontend calls process_request

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    0dff374 View commit details
    Browse the repository at this point in the history
  7. rgw: implement ClientIO interface for asio + beast

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    b2cc1d9 View commit details
    Browse the repository at this point in the history
  8. rgw: asio frontend uses async_read

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8340439 View commit details
    Browse the repository at this point in the history
  9. rgw: s3 GetObj sends status first

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    a084b82 View commit details
    Browse the repository at this point in the history
  10. rgw: s3 PutObj sends status first

    Signed-off-by: Casey Bodley <cbodley@redhat.com>
    cbodley authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    04057a0 View commit details
    Browse the repository at this point in the history
  11. rgw: mark asio frontend as experimental

    Users will need to set "rgw-asio-frontend" as an experimental
    feature.
    
    Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
    mattbenjamin authored and rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8143913 View commit details
    Browse the repository at this point in the history
  12. rgw: introduce RGWStreamIOBase to separate all non-IO related things.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    e4ee1d0 View commit details
    Browse the repository at this point in the history
  13. rgw: move transfer accounting management from RGWClientIO to RGWStrea…

    …mIO.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c599078 View commit details
    Browse the repository at this point in the history
  14. rgw: improve const-correctness in the front-end infrastructure.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    cbb26fe View commit details
    Browse the repository at this point in the history
  15. rgw: strip out AWSv4 signature calculation from RGWStreamIOBase.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    63dd818 View commit details
    Browse the repository at this point in the history
  16. rgw: further cleaning in RGWStreamIOBase.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    980c3c7 View commit details
    Browse the repository at this point in the history
  17. rgw: rename RGWStreamIOBase to RGWStreamIOEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    18104f1 View commit details
    Browse the repository at this point in the history
  18. rgw: add basic infrastructure for decorators over RGWStreamIOBase.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    3e52525 View commit details
    Browse the repository at this point in the history
  19. rgw: dissect accounting interface from RGWClientIO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8c7a271 View commit details
    Browse the repository at this point in the history
  20. rgw: dissect non-core ops from RGWStreamIOEngine to RGWStreamIOFacade.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    21af773 View commit details
    Browse the repository at this point in the history
  21. rgw: implement RGWStreamIOLegacyWrapper for using new front-ends with…

    … RGWStreamIO.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    16f7324 View commit details
    Browse the repository at this point in the history
  22. rgw: RGWMongoose uses RGWStreamIOEngine instead of RGWStreamIO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    082e414 View commit details
    Browse the repository at this point in the history
  23. rgw: implement RGWStreamIOAccountingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    ba00508 View commit details
    Browse the repository at this point in the history
  24. rgw: RGWClientIO::get_env() is pure virtual now.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    754bf7c View commit details
    Browse the repository at this point in the history
  25. rgw: implement RGWStreamIOBufferingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    1c492cb View commit details
    Browse the repository at this point in the history
  26. rgw: eradicate dynamic memory allocations from RGWEnv.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    61e35e4 View commit details
    Browse the repository at this point in the history
  27. rgw: ONLY rename RGWMongoose to RGWCivetWeb and marks overrides.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    4d79ba9 View commit details
    Browse the repository at this point in the history
  28. rgw: ONLY rename RGWMongooseFrontend to RGWCivetWebFrontend.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    98907b1 View commit details
    Browse the repository at this point in the history
  29. rgw: implement RGWStreamIOConLenControllingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c178026 View commit details
    Browse the repository at this point in the history
  30. rgw: implement RGWStreamIOReorderingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    d54eb7e View commit details
    Browse the repository at this point in the history
  31. rgw: implement RGWStreamIOChunkingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    e2bfd3e View commit details
    Browse the repository at this point in the history
  32. rgw: switch out RGWCivetWeb to the new frontend infrastructure.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    e03cdd5 View commit details
    Browse the repository at this point in the history
  33. rgw: cleaning in the glue code for CivetWeb.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    1a8a0d7 View commit details
    Browse the repository at this point in the history
  34. rgw: port the ASIO frontend to the new RGWStreamIOEngine infrastructure.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    2825cc7 View commit details
    Browse the repository at this point in the history
  35. rgw: port the FCGI frontend to the new RGWStreamIOEngine infrastructure.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    ddf61a8 View commit details
    Browse the repository at this point in the history
  36. rgw: ONLY improve formatting in the FCGI frontend.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    2995102 View commit details
    Browse the repository at this point in the history
  37. rgw: port the LoadGen frontend to the new RGWStreamIOEngine infrastru…

    …cture.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8e73888 View commit details
    Browse the repository at this point in the history
  38. rgw: cleanup in the LoadGen frontend.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    0ae4e13 View commit details
    Browse the repository at this point in the history
  39. rgw: handle errors via exceptions in RGWStreamIOEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    bbdcc62 View commit details
    Browse the repository at this point in the history
  40. rgw: switch to std::size_t in the front-end infrastructure.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    011a10f View commit details
    Browse the repository at this point in the history
  41. rgw: add support for exhibiting URL subtree over a specifc frontend i…

    …nstance.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    cc77769 View commit details
    Browse the repository at this point in the history
  42. rgw: ONLY reformat RGWFrontend and RGWCivetWebFrontend.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c82abe0 View commit details
    Browse the repository at this point in the history
  43. rgw: implement RGWStreamIO[Engine]::send_header().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    d64c604 View commit details
    Browse the repository at this point in the history
  44. rgw: switch to RGWStreamIO::send_header().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    4787b53 View commit details
    Browse the repository at this point in the history
  45. rgw: drop RGWStreamIOFacade::print().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    f3d4327 View commit details
    Browse the repository at this point in the history
  46. rgw: RGWStreamIOReorderingEngine understands send_header().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    f04f4dc View commit details
    Browse the repository at this point in the history
  47. rgw: implement RGWStreamIOEngine::{send,recv}_body().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    1b28da3 View commit details
    Browse the repository at this point in the history
  48. rgw: switch to RGWStreamIOEngine::{send,recv}_body().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    ed2bf82 View commit details
    Browse the repository at this point in the history
  49. rgw: rework quoting value of HTTP headers.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    c71c652 View commit details
    Browse the repository at this point in the history
  50. rgw: add control logic for chunked transfer encoding.

    Needed for tempest.api.object_storage.test_object_slo.ObjectSloTest.test_delete_large_object.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    3c2afc6 View commit details
    Browse the repository at this point in the history
  51. rgw: replace dump_pair() with dump_header_if_nonempty().

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    e7f7f80 View commit details
    Browse the repository at this point in the history
  52. rgw: ONLY move RGWDecoratedStreamIO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    4553c5b View commit details
    Browse the repository at this point in the history
  53. rgw: remove the legacy, non-throwing variant of RGWStreamIO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    16c2586 View commit details
    Browse the repository at this point in the history
  54. rgw: change naming based on RGWStreamIO in favor of RGWRestfulIO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8bb27bc View commit details
    Browse the repository at this point in the history
  55. rgw: delegate transfer accounting to RGWRestfulIOAccountingEngine.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    6db8699 View commit details
    Browse the repository at this point in the history
  56. rgw: rename and split STREAM_IO into RESTFUL_IO and ACCOUNTING_IO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    e524c58 View commit details
    Browse the repository at this point in the history
  57. rgw: make RGWRestfulIOEngine::Exception alias to std::system_error.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    dad4d16 View commit details
    Browse the repository at this point in the history
  58. rgw: ONLY replace std::size_t with size_t.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    f028557 View commit details
    Browse the repository at this point in the history
  59. rgw: switch to namespaces and change naming in the frontend subsystem.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    79923f4 View commit details
    Browse the repository at this point in the history
  60. rgw: fix off-by-one in RGWAsioClientIO::init_env.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    9f1713c View commit details
    Browse the repository at this point in the history
  61. rgw: fix setting HTTP/0.0 in AsioConnection::write_bad_request.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8c32686 View commit details
    Browse the repository at this point in the history
  62. rgw: improve debugs in RGWAsioClientIO and AsioConnection.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    27338a0 View commit details
    Browse the repository at this point in the history
  63. rgw: employ necessary filter over RGWFCGX frontend.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    d69a1f6 View commit details
    Browse the repository at this point in the history
  64. rgw: use filters over AsioConnection.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    758b251 View commit details
    Browse the repository at this point in the history
  65. rgw: introduce AWS_AUTHv4_IO.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    908c073 View commit details
    Browse the repository at this point in the history
  66. rgw: switch to size_t and exception in rgw::io::BasicClient::complete…

    …_request.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    3e1a90d View commit details
    Browse the repository at this point in the history
  67. rgw: fix perfect forwarding of template arguments in IO filters.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    8b03f87 View commit details
    Browse the repository at this point in the history
  68. rgw: add support for dispatching requests with respect to frontend pr…

    …efix.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    abccedc View commit details
    Browse the repository at this point in the history
  69. rgw: protect the {ACCOUNTING,AWS_AUTHv4,RESTFUL}_IO casts with assert.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    a33462f View commit details
    Browse the repository at this point in the history
  70. rgw: ONLY move rgw_client_io_decoimpl.h to rgw_client_io_filters.h.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    5da12f6 View commit details
    Browse the repository at this point in the history
  71. rgw: don't use boost::string_ref::to_string() method.

    The method has been introduced in Boost 1.54. However,
    some systems we are targeting (like CentOS 7) offer
    older version of the library.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    d1c8276 View commit details
    Browse the repository at this point in the history
  72. rgw: document interfaces of the rgw::io namespace.

    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    b3bbe20 View commit details
    Browse the repository at this point in the history
  73. rgw: fix interface compliance of RGWCivetWeb::write_data().

    This is a potential fix for partial writes and handling of error
    codes that might come from mg_write() of the CivetWeb front-end.
    None of the potential issues has been observed in testing.
    
    The commit also documents the same aspects regarding the ASIO
    and FastCGI front-ends.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    3a9f50c View commit details
    Browse the repository at this point in the history
  74. rgw: make the ASIO front-end compilation optional.

    This is because of versioning issues between Boost and Beast
    on CentOS 7. It is intended that this patch will be reverted
    after merging the in-tree Boost facility.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 21, 2016
    Copy the full SHA
    46a4464 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2016

  1. rgw: sanitize length of metadata attributes.

    std::string and thus boost::string_ref ARE OBLIGED to carry multiple
    0x00 and count them to the length of a string. We need to take that
    into consideration and sanitize the size of a ceph::buffer::list used
    to store metadata values (x-amz-meta-*, X-Container-Meta-*, etags).
    Otherwise we might send 0x00 to clients.
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 25, 2016
    Copy the full SHA
    ac4928a View commit details
    Browse the repository at this point in the history
  2. rgw: implement buffered HTTP header composition in front-ends.

    Without the patch front-ends issueed a lot of small IO operations
    leading to increased overhead on syscalls and to the fragmentation
    of an HTTP message across multiple TCP segments. The later was
    occuring when the Nagle's algorithm hadn't been able to form
    a single TCP segment (usually when running on extremely fast
    network interfaces like loopback).
    
    Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
    rzarzynski committed Oct 25, 2016
    Copy the full SHA
    573c564 View commit details
    Browse the repository at this point in the history