Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: continuation of the auth rework -- AWSv4 #14885

Merged
merged 69 commits into from Jun 13, 2017
Merged
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ac1a3f4
rgw: aws4: add AWS4 auth support for S3 Post Object API
jmunhoz Mar 10, 2017
98e153f
rgw: rework interfaces of AWSv4 helper primitives.
rzarzynski Apr 13, 2017
c9afd24
rgw: get_v4_canonical_request_hash doesn't depend on req_state anymore.
rzarzynski Apr 13, 2017
26952b9
rgw: dissect AWSv4's Canonical URI crafting into a separated function.
rzarzynski Apr 13, 2017
b000b35
rgw: dissect AWSv4's Canonical QS crafting into a separated function.
rzarzynski Apr 13, 2017
985c2bc
rgw: dissect AWSv4's Canonical Headers crafting into a separated func…
rzarzynski Apr 13, 2017
b18996c
rgw: eradicate req_state::http_auth.
rzarzynski Apr 13, 2017
67db70d
rgw: dissect basic AWSv4's credentials parsing into separated function.
rzarzynski Apr 14, 2017
0b6adc6
rgw: rgw::auth::s3::get_v4_signature doesn't depend on req_state anym…
rzarzynski Apr 14, 2017
9f594c5
rgw: drop req_state::aws4_auth::payload_hash as it doesn't need to be…
rzarzynski Apr 14, 2017
4b78946
rgw: minimise the number of parameters of rgw::auth::s3::get_v4_signa…
rzarzynski Apr 14, 2017
4e55c1d
rgw: add std::array-aware variants of calc_hmac_sha256() and buf_to_h…
rzarzynski Apr 14, 2017
195bf5c
rgw: split generation AWSv4's SigningKey into a separate func.
rzarzynski Apr 15, 2017
33f013b
rgw: refactor buf_to_hex and improve its const-correctness.
rzarzynski Apr 15, 2017
405d31d
rgw: clean up rgw::auth::s3::get_v4_signing_key.
rzarzynski Apr 15, 2017
71417ca
rgw: eradicate req_state::aws4_auth::signed_hdrs.
rzarzynski Apr 15, 2017
1514408
rgw: eradicate req_state::aws4_auth::credential.
rzarzynski Apr 15, 2017
619c8ed
rgw: decouple AWSv4 signature verification and payload fingerprint ch…
rzarzynski Apr 15, 2017
92409e1
rgw: further minimise the req_state::aws4_auth shared state.
rzarzynski Apr 15, 2017
79b826e
rgw: clean up the AWSv4 completion calls across RGWOps.
rzarzynski Apr 15, 2017
06944f2
rgw: AWSv4 completion verifies the payload's fingerprint only.
rzarzynski Apr 15, 2017
fe4ef87
rgw: decompose rgw::auth::s3::parse_credentials regarding the query s…
rzarzynski Apr 15, 2017
d0b4231
rgw: extend Version2ndEngine::Extractor to handle AWSv4 as well.
rzarzynski Apr 17, 2017
8b4e142
rgw: ONLY change names due to AWSv4 adoption in the auth infra.
rzarzynski Apr 17, 2017
e9a17de
rgw: all S3 auth engines are able to use AWSv4 now.
rzarzynski Apr 17, 2017
4200e12
rgw: rgw::auth::s3::parse_credentials() extracts access_key_id now.
rzarzynski Apr 18, 2017
a33eb15
rgw: make the get_v4_canonical_request_hash identified shorter.
rzarzynski Apr 18, 2017
6e1daa8
rgw: dissect AWSv4's expected payload extraction into a dedicated fun…
rzarzynski Apr 18, 2017
420f1c0
rgw: AWSv4's completer_factory takes the secret_key optionally.
rzarzynski Apr 18, 2017
e126cad
rgw: integrate AWSv4 auth schema with the new auth infra.
rzarzynski Apr 18, 2017
a30d97a
rgw: make AWS_AUTHv4_IO const-friendly and const-correct.
rzarzynski Apr 18, 2017
61d4f73
rgw: switch to the Completer interface for the AWSv4 payload check.
rzarzynski Apr 18, 2017
2d4e000
rgw: extend rgw::auth::Completer to handle commiting modifications to…
rzarzynski Apr 19, 2017
affa201
rgw: implement the rgw::auth::Completer for AWSv4.
rzarzynski Apr 21, 2017
36e1c34
rgw: drop the old AWSv4 code.
rzarzynski Apr 21, 2017
47d014c
rgw: implement rgw::auth::s3::is_v4_payload_empty.
rzarzynski Apr 21, 2017
29095d2
rgw: ONLY move AWSv4Completer from rgw_rest_s3.cc to rgw_auth_s3.cc.
rzarzynski Apr 23, 2017
139de6a
rgw: extend RGWRestfulIO to cover dynamic filter injection.
rzarzynski Apr 23, 2017
2778a8a
rgw: turn AWSv4Completer into a filter over rgw::io::RestfulClient.
rzarzynski Apr 23, 2017
48093dc
rgw: drop aws4_auth_needs_complete from req_state.
rzarzynski Apr 24, 2017
def8f64
rgw: AWSv4Completer dechunks data in the streaming mode.
rzarzynski Apr 27, 2017
25b7106
rgw: implement calc_hash_sha256_restart_stream().
rzarzynski Apr 29, 2017
eeb8599
rgw: the AWSv4 completer verifies chunks' signatures now.
rzarzynski Apr 28, 2017
0845d9c
rgw: remove the old AWS v4 streaming mode's implementation.
rzarzynski Apr 28, 2017
68bc0d0
rgw: split the AWSv4Completer and clean-up the code.
rzarzynski Apr 29, 2017
dc270a6
rgw: add support for HTTP_X_AMZ_DECODED_CONTENT_LENGTH.
rzarzynski Apr 29, 2017
043abfa
rgw: handle AWSv4 in Browser Upload using the new auth infra.
rzarzynski Apr 30, 2017
0b1d8b0
rgw: clean-up rgw::auth::s3::get_v4_signature().
rzarzynski May 2, 2017
503d687
rgw: use std::make_shared for AWSv4 completers creation.
rzarzynski May 4, 2017
1f628eb
rgw: clean-up AWSv4's Canonical QS crafting.
rzarzynski May 10, 2017
5184842
rgw: optimize and clean-up the AWSv4 signature processing.
rzarzynski May 10, 2017
a5a8b27
rgw: introduce rgw::auth::s3::AWS4_HMAC_SHA256_STR to kill magics.
rzarzynski May 11, 2017
0e4e0e4
rgw: switch from boost::string_ref to string_view in AWSv4-related code.
rzarzynski May 11, 2017
8dab93d
rgw: switch from boost::string_ref to string_view in AWSv4-related co…
rzarzynski May 15, 2017
1cb269e
rgw: remove the duplicative trim_whitespace from rgw_common.cc.
rzarzynski May 15, 2017
7de4557
rgw: drop 'using ceph::crypto::SHA256' from rgw_common.h.
rzarzynski May 15, 2017
c557fe2
rgw: rework and optimise crafting of AWSv4's canonical query string.
rzarzynski May 16, 2017
64cfc43
rgw; rework interface and implementation of url_decode.
rzarzynski May 17, 2017
51383c3
rgw: rework the implementation of rgw::auth::s3::get_v4_canonical_hea…
rzarzynski May 17, 2017
e8dd37a
rgw: use preallocated std::strings when concatenating in AWSv4.
rzarzynski May 19, 2017
7e8d1d7
rgw: replace magic strings in the AWSv4 code.
rzarzynski May 19, 2017
5363643
common/sstring: switch to boost::string_view as string_ref is depreca…
rzarzynski May 23, 2017
a8a9a84
common/backport14: add the constexpr-capable variant of std::max().
rzarzynski May 24, 2017
c892228
rgw: switch to Ceph's sstring in AWS signature generation process.
rzarzynski May 23, 2017
2060308
rgw: optimize AWSv4 parsing with Boost's small_vector.
rzarzynski May 25, 2017
73e78ae
rgw: only rename AWSv2AuthStrategy -> AWSAuthStrategy.
rzarzynski Jun 2, 2017
f3317f6
rgw: introduce string_to_sign_t abstraction to the AWS auth.
rzarzynski Jun 6, 2017
bd81c21
rgw: introduce rgw::auth::Strategy::apply() to deduplicate code.
rzarzynski Jun 6, 2017
2417b64
rgw: handle the Boto2 compatibility of AWSv4 in an abstract way.
rzarzynski Jun 7, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
108 changes: 59 additions & 49 deletions src/rgw/rgw_auth_s3.cc
Expand Up @@ -513,63 +513,73 @@ static inline std::string aws4_uri_encode(const std::string& src)
return result;
}

static inline std::string aws4_uri_recode(const boost::string_view& src)
{
/* TODO(rzarzynski): we might want to have a string_view-aware variant of
* url_decode. */
const auto src_str = src.to_string();

std::string decoded;
url_decode(src_str, decoded);
if (decoded.length() != src.length()) {
return src_str;
} else {
return aws4_uri_encode(decoded);
}
}

std::string get_v4_canonical_qs(const req_info& info, const bool using_qs)
{
std::string canonical_qs = info.request_params;

if (!canonical_qs.empty()) {

/* Handle case when query string exists. Step 3 in
* http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html */
map<string, string> canonical_qs_map;
istringstream cqs(canonical_qs);
string keyval;

while (getline(cqs, keyval, '&')) {
string key, val;
istringstream kv(keyval);
getline(kv, key, '=');
getline(kv, val, '=');
if (!using_qs || key != "X-Amz-Signature") {
string encoded_key;
string encoded_val;
if (key != "X-Amz-Credential") {
string key_decoded;
url_decode(key, key_decoded);
if (key.length() != key_decoded.length()) {
encoded_key = key;
} else {
encoded_key = aws4_uri_encode(key);
}
string val_decoded;
url_decode(val, val_decoded);
if (val.length() != val_decoded.length()) {
encoded_val = val;
} else {
encoded_val = aws4_uri_encode(val);
}
} else {
encoded_key = key;
encoded_val = val;
}
canonical_qs_map[encoded_key] = encoded_val;
}
}
if (info.request_params.empty()) {
/* Optimize the typical flow. */
return std::string();
}

canonical_qs = "";
/* Handle case when query string exists. Step 3 described in: http://docs.
* aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html */
std::map<std::string, std::string> canonical_qs_map;
for (const auto& s : get_str_vec(info.request_params, "&")) {
boost::string_view key, val;
const auto parsed_pair = parse_key_value(s);
if (parsed_pair) {
std::tie(key, val) = *parsed_pair;
} else {
/* Handling a parameter without any value (even the empty one). That's
* it, we've encountered something like "this_param&other_param=val"
* which is used by S3 for subresources. */
key = s;
}

map<string, string>::iterator last = canonical_qs_map.end();
--last;
if (using_qs && key == "X-Amz-Signature") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't Amazon describe lowering these? initial caps looks strange in the code

/* Preserving the original behaviour of get_v4_canonical_qs() here. */
continue;
}

for (map<string, string>::iterator it = canonical_qs_map.begin();
it != canonical_qs_map.end(); ++it) {
canonical_qs.append(it->first + "=" + it->second);
if (it != last) {
canonical_qs.append("&");
}
if (key == "X-Amz-Credential") {
/* FIXME(rzarzynski): I can't find any comment in the previously linked
* Amazon's docs saying that X-Amz-Credential should be handled in this
* way. */
canonical_qs_map[key.to_string()] = val.to_string();
} else {
canonical_qs_map[aws4_uri_recode(key)] = aws4_uri_recode(val);
}
}

/* Thanks to the early exist we have the guarantee that canonical_qs_map has
* at least one element. */
auto iter = std::begin(canonical_qs_map);
std::string canonical_qs;
canonical_qs.append(iter->first)
.append("=", ::strlen("="))
.append(iter->second);

for (iter++; iter != std::end(canonical_qs_map); iter++) {
canonical_qs.append("&", ::strlen("&"))
.append(iter->first)
.append("=", ::strlen("="))
.append(iter->second);
}

return canonical_qs;
}

Expand Down