Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: continuation of the auth rework -- AWSv4 #14885

Merged
merged 69 commits into from Jun 13, 2017
Merged
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ac1a3f4
rgw: aws4: add AWS4 auth support for S3 Post Object API
jmunhoz Mar 10, 2017
98e153f
rgw: rework interfaces of AWSv4 helper primitives.
rzarzynski Apr 13, 2017
c9afd24
rgw: get_v4_canonical_request_hash doesn't depend on req_state anymore.
rzarzynski Apr 13, 2017
26952b9
rgw: dissect AWSv4's Canonical URI crafting into a separated function.
rzarzynski Apr 13, 2017
b000b35
rgw: dissect AWSv4's Canonical QS crafting into a separated function.
rzarzynski Apr 13, 2017
985c2bc
rgw: dissect AWSv4's Canonical Headers crafting into a separated func…
rzarzynski Apr 13, 2017
b18996c
rgw: eradicate req_state::http_auth.
rzarzynski Apr 13, 2017
67db70d
rgw: dissect basic AWSv4's credentials parsing into separated function.
rzarzynski Apr 14, 2017
0b6adc6
rgw: rgw::auth::s3::get_v4_signature doesn't depend on req_state anym…
rzarzynski Apr 14, 2017
9f594c5
rgw: drop req_state::aws4_auth::payload_hash as it doesn't need to be…
rzarzynski Apr 14, 2017
4b78946
rgw: minimise the number of parameters of rgw::auth::s3::get_v4_signa…
rzarzynski Apr 14, 2017
4e55c1d
rgw: add std::array-aware variants of calc_hmac_sha256() and buf_to_h…
rzarzynski Apr 14, 2017
195bf5c
rgw: split generation AWSv4's SigningKey into a separate func.
rzarzynski Apr 15, 2017
33f013b
rgw: refactor buf_to_hex and improve its const-correctness.
rzarzynski Apr 15, 2017
405d31d
rgw: clean up rgw::auth::s3::get_v4_signing_key.
rzarzynski Apr 15, 2017
71417ca
rgw: eradicate req_state::aws4_auth::signed_hdrs.
rzarzynski Apr 15, 2017
1514408
rgw: eradicate req_state::aws4_auth::credential.
rzarzynski Apr 15, 2017
619c8ed
rgw: decouple AWSv4 signature verification and payload fingerprint ch…
rzarzynski Apr 15, 2017
92409e1
rgw: further minimise the req_state::aws4_auth shared state.
rzarzynski Apr 15, 2017
79b826e
rgw: clean up the AWSv4 completion calls across RGWOps.
rzarzynski Apr 15, 2017
06944f2
rgw: AWSv4 completion verifies the payload's fingerprint only.
rzarzynski Apr 15, 2017
fe4ef87
rgw: decompose rgw::auth::s3::parse_credentials regarding the query s…
rzarzynski Apr 15, 2017
d0b4231
rgw: extend Version2ndEngine::Extractor to handle AWSv4 as well.
rzarzynski Apr 17, 2017
8b4e142
rgw: ONLY change names due to AWSv4 adoption in the auth infra.
rzarzynski Apr 17, 2017
e9a17de
rgw: all S3 auth engines are able to use AWSv4 now.
rzarzynski Apr 17, 2017
4200e12
rgw: rgw::auth::s3::parse_credentials() extracts access_key_id now.
rzarzynski Apr 18, 2017
a33eb15
rgw: make the get_v4_canonical_request_hash identified shorter.
rzarzynski Apr 18, 2017
6e1daa8
rgw: dissect AWSv4's expected payload extraction into a dedicated fun…
rzarzynski Apr 18, 2017
420f1c0
rgw: AWSv4's completer_factory takes the secret_key optionally.
rzarzynski Apr 18, 2017
e126cad
rgw: integrate AWSv4 auth schema with the new auth infra.
rzarzynski Apr 18, 2017
a30d97a
rgw: make AWS_AUTHv4_IO const-friendly and const-correct.
rzarzynski Apr 18, 2017
61d4f73
rgw: switch to the Completer interface for the AWSv4 payload check.
rzarzynski Apr 18, 2017
2d4e000
rgw: extend rgw::auth::Completer to handle commiting modifications to…
rzarzynski Apr 19, 2017
affa201
rgw: implement the rgw::auth::Completer for AWSv4.
rzarzynski Apr 21, 2017
36e1c34
rgw: drop the old AWSv4 code.
rzarzynski Apr 21, 2017
47d014c
rgw: implement rgw::auth::s3::is_v4_payload_empty.
rzarzynski Apr 21, 2017
29095d2
rgw: ONLY move AWSv4Completer from rgw_rest_s3.cc to rgw_auth_s3.cc.
rzarzynski Apr 23, 2017
139de6a
rgw: extend RGWRestfulIO to cover dynamic filter injection.
rzarzynski Apr 23, 2017
2778a8a
rgw: turn AWSv4Completer into a filter over rgw::io::RestfulClient.
rzarzynski Apr 23, 2017
48093dc
rgw: drop aws4_auth_needs_complete from req_state.
rzarzynski Apr 24, 2017
def8f64
rgw: AWSv4Completer dechunks data in the streaming mode.
rzarzynski Apr 27, 2017
25b7106
rgw: implement calc_hash_sha256_restart_stream().
rzarzynski Apr 29, 2017
eeb8599
rgw: the AWSv4 completer verifies chunks' signatures now.
rzarzynski Apr 28, 2017
0845d9c
rgw: remove the old AWS v4 streaming mode's implementation.
rzarzynski Apr 28, 2017
68bc0d0
rgw: split the AWSv4Completer and clean-up the code.
rzarzynski Apr 29, 2017
dc270a6
rgw: add support for HTTP_X_AMZ_DECODED_CONTENT_LENGTH.
rzarzynski Apr 29, 2017
043abfa
rgw: handle AWSv4 in Browser Upload using the new auth infra.
rzarzynski Apr 30, 2017
0b1d8b0
rgw: clean-up rgw::auth::s3::get_v4_signature().
rzarzynski May 2, 2017
503d687
rgw: use std::make_shared for AWSv4 completers creation.
rzarzynski May 4, 2017
1f628eb
rgw: clean-up AWSv4's Canonical QS crafting.
rzarzynski May 10, 2017
5184842
rgw: optimize and clean-up the AWSv4 signature processing.
rzarzynski May 10, 2017
a5a8b27
rgw: introduce rgw::auth::s3::AWS4_HMAC_SHA256_STR to kill magics.
rzarzynski May 11, 2017
0e4e0e4
rgw: switch from boost::string_ref to string_view in AWSv4-related code.
rzarzynski May 11, 2017
8dab93d
rgw: switch from boost::string_ref to string_view in AWSv4-related co…
rzarzynski May 15, 2017
1cb269e
rgw: remove the duplicative trim_whitespace from rgw_common.cc.
rzarzynski May 15, 2017
7de4557
rgw: drop 'using ceph::crypto::SHA256' from rgw_common.h.
rzarzynski May 15, 2017
c557fe2
rgw: rework and optimise crafting of AWSv4's canonical query string.
rzarzynski May 16, 2017
64cfc43
rgw; rework interface and implementation of url_decode.
rzarzynski May 17, 2017
51383c3
rgw: rework the implementation of rgw::auth::s3::get_v4_canonical_hea…
rzarzynski May 17, 2017
e8dd37a
rgw: use preallocated std::strings when concatenating in AWSv4.
rzarzynski May 19, 2017
7e8d1d7
rgw: replace magic strings in the AWSv4 code.
rzarzynski May 19, 2017
5363643
common/sstring: switch to boost::string_view as string_ref is depreca…
rzarzynski May 23, 2017
a8a9a84
common/backport14: add the constexpr-capable variant of std::max().
rzarzynski May 24, 2017
c892228
rgw: switch to Ceph's sstring in AWS signature generation process.
rzarzynski May 23, 2017
2060308
rgw: optimize AWSv4 parsing with Boost's small_vector.
rzarzynski May 25, 2017
73e78ae
rgw: only rename AWSv2AuthStrategy -> AWSAuthStrategy.
rzarzynski Jun 2, 2017
f3317f6
rgw: introduce string_to_sign_t abstraction to the AWS auth.
rzarzynski Jun 6, 2017
bd81c21
rgw: introduce rgw::auth::Strategy::apply() to deduplicate code.
rzarzynski Jun 6, 2017
2417b64
rgw: handle the Boto2 compatibility of AWSv4 in an abstract way.
rzarzynski Jun 7, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 10 additions & 3 deletions src/rgw/rgw_auth.h
Expand Up @@ -103,8 +103,8 @@ class IdentityApplier : public Identity {
virtual void load_acct_info(RGWUserInfo& user_info) const = 0; /* out */

/* Apply any changes to request state. This method will be most useful for
* TempURL of Swift API or AWSv4. */
virtual void modify_request_state(req_state * s) const {} /* in/out */
* TempURL of Swift API. */
virtual void modify_request_state(req_state* s) const {} /* in/out */
};


Expand All @@ -126,14 +126,21 @@ class IdentityApplier : public Identity {
* E. execute-commit - commit the modifications from point C. */
class Completer {
public:
typedef std::unique_ptr<Completer> cmplptr_t;
/* It's expected that Completers would tend to implement many interfaces
* and be used not only in req_state::auth::completer. Ref counting their
* instances woild be helpful. */
typedef std::shared_ptr<Completer> cmplptr_t;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: the comment about shared vs. unique, where is this pointer shared outside of req_state::completer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's passed to the front-end subsystem as the AWSv4 completers are also filters over IO.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, thanks. i was hoping that we could stick with unique_ptr for req_state::completer and rely on the lifetime of req_state to share raw pointers elsewhere, but i see that RGWRestfulIO will outlive req_state. so using shared_ptr is probably easier than allowing filters to be removed from RGWRestfulIO before the req_state goes away


virtual ~Completer() = default;

/* Complete the authentication process. Return boolean indicating whether
* the completion succeeded. On error throws rgw::auth::Exception storing
* the reason. */
virtual bool complete() = 0;

/* Apply any changes to request state. The initial use case was injecting
* the AWSv4 filter over rgw::io::RestfulClient in req_state. */
virtual void modify_request_state(req_state* s) = 0; /* in/out */
};


Expand Down
23 changes: 13 additions & 10 deletions src/rgw/rgw_auth_keystone.cc
Expand Up @@ -289,9 +289,9 @@ TokenEngine::authenticate(const std::string& token,
* Try to validate S3 auth against keystone s3token interface
*/
std::pair<boost::optional<rgw::keystone::TokenEnvelope>, int>
EC2Engine::get_from_keystone(const std::string& access_key_id,
EC2Engine::get_from_keystone(const boost::string_view& access_key_id,
const std::string& string_to_sign,
const std::string& signature) const
const boost::string_view& signature) const
{
/* prepare keystone url */
std::string keystone_url = config.get_endpoint_url();
Expand Down Expand Up @@ -335,9 +335,9 @@ EC2Engine::get_from_keystone(const std::string& access_key_id,
JSONFormatter credentials(false);
credentials.open_object_section("");
credentials.open_object_section("credentials");
credentials.dump_string("access", access_key_id);
credentials.dump_string("access", sview2cstr(access_key_id).data());
credentials.dump_string("token", rgw::to_base64(string_to_sign));
credentials.dump_string("signature", signature);
credentials.dump_string("signature", sview2cstr(signature).data());
credentials.close_section();
credentials.close_section();

Expand Down Expand Up @@ -413,11 +413,14 @@ EC2Engine::get_creds_info(const EC2Engine::token_envelope_t& token,
};
}

rgw::auth::Engine::result_t EC2Engine::authenticate(const std::string& access_key_id,
const std::string& signature,
const std::string& string_to_sign,
/* Passthorugh only! */
const req_state* s) const
rgw::auth::Engine::result_t EC2Engine::authenticate(
const boost::string_view& access_key_id,
const boost::string_view& signature,
const std::string& string_to_sign,
const signature_factory_t& signature_factory,
const completer_factory_t& completer_factory,
/* Passthorugh only! */
const req_state* s) const
{
/* This will be initialized on the first call to this method. In C++11 it's
* also thread-safe. */
Expand Down Expand Up @@ -472,7 +475,7 @@ rgw::auth::Engine::result_t EC2Engine::authenticate(const std::string& access_ke

auto apl = apl_factory->create_apl_remote(cct, s, get_acl_strategy(*t),
get_creds_info(*t, accepted_roles.admin));
return result_t::grant(std::move(apl));
return result_t::grant(std::move(apl), completer_factory(boost::none));
}
}

Expand Down
19 changes: 11 additions & 8 deletions src/rgw/rgw_auth_keystone.h
Expand Up @@ -7,6 +7,7 @@

#include <utility>
#include <boost/optional.hpp>
#include <boost/utility/string_view.hpp>

#include "rgw_auth.h"
#include "rgw_rest_s3.h"
Expand Down Expand Up @@ -71,7 +72,7 @@ class TokenEngine : public rgw::auth::Engine {
}; /* class TokenEngine */


class EC2Engine : public rgw::auth::s3::Version2ndEngine {
class EC2Engine : public rgw::auth::s3::AWSEngine {
using acl_strategy_t = rgw::auth::RemoteApplier::acl_strategy_t;
using auth_info_t = rgw::auth::RemoteApplier::AuthInfo;
using result_t = rgw::auth::Engine::result_t;
Expand All @@ -87,29 +88,31 @@ class EC2Engine : public rgw::auth::s3::Version2ndEngine {
const std::vector<std::string>& admin_roles
) const noexcept;
std::pair<boost::optional<token_envelope_t>, int>
get_from_keystone(const std::string& access_key_id,
get_from_keystone(const boost::string_view& access_key_id,
const std::string& string_to_sign,
const std::string& signature) const;
result_t authenticate(const std::string& access_key_id,
const std::string& signature,
const boost::string_view& signature) const;
result_t authenticate(const boost::string_view& access_key_id,
const boost::string_view& signature,
const std::string& string_to_sign,
const signature_factory_t& signature_factory,
const completer_factory_t& completer_factory,
const req_state* s) const override;
public:
EC2Engine(CephContext* const cct,
const rgw::auth::s3::Version2ndEngine::Extractor* const extractor,
const rgw::auth::s3::AWSEngine::VersionAbstractor* const ver_abstractor,
const rgw::auth::RemoteApplier::Factory* const apl_factory,
rgw::keystone::Config& config,
/* The token cache is used ONLY for the retrieving admin token.
* Due to the architecture of AWS Auth S3 credentials cannot be
* cached at all. */
rgw::keystone::TokenCache& token_cache)
: Version2ndEngine(cct, *extractor),
: AWSEngine(cct, *ver_abstractor),
apl_factory(apl_factory),
config(config),
token_cache(token_cache) {
}

using Version2ndEngine::authenticate;
using AWSEngine::authenticate;

const char* get_name() const noexcept override {
return "rgw::auth::keystone::EC2Engine";
Expand Down
4 changes: 2 additions & 2 deletions src/rgw/rgw_auth_registry.h
Expand Up @@ -25,9 +25,9 @@ class StrategyRegistry {
using s3_strategy_t = rgw::auth::s3::AWSv2AuthStrategy<ExtractorT>;

using s3_main_strategy_t = \
s3_strategy_t<rgw::auth::s3::RGWS3V2Extractor>;
s3_strategy_t<rgw::auth::s3::AWSGeneralAbstractor>;
using s3_post_strategy_t = \
s3_strategy_t<rgw::auth::s3::RGWGetPolicyV2Extractor>;
s3_strategy_t<rgw::auth::s3::AWSBrowserUploadAbstractor>;

s3_main_strategy_t s3_main_strategy;
s3_post_strategy_t s3_post_strategy;
Expand Down