rgw: Added code for REST APIs for AWS Roles.

[pritha-srivastava]

No description provided.

[cbodley]

most of verify_permission() is duplicated by all of the Role ops. could we factor this into a base class so each op only has to say whether it's a read or write? or maybe have a RGWRoleRead and RGWRoleWrite that the others inherit from?

[cbodley]

since these ops are all specific to s3, maybe we don't need the usual Op, Op_ObjStore, ObjStore_S3 hierarchy. can we just put them all in a separate rgw_rest_role.cc?

[pritha-srivastava]

@cbodley : I have refactored the code and added them as separate commits. Once they are approved by you, i'll rebase the commits to the previous ones.

[mattbenjamin]

@pritha-srivastava can you rebase when you get a chance? Tx!

[mattbenjamin]

@pritha-srivastava context: @aemerson suggests merging these first, then adapting to following policy change; Do you anticipate and problems with going in that order?

[pritha-srivastava]

@mattbenjamin : I dont see any problem merging this patch first and then adding policy stuff. Just wanted to point out that the version number will have to be bumped while encode/ decode operations, when we switch to the 'actual' policy.
Casey has a couple of comments on the REST API part. I'll address them and then rebase the PR, and then it will be good to merge.

[pritha-srivastava]

@cbodley: Most of send_response() is also the same for all ops (as you have mentioned in your comment above), i'll try to refactor this as well.

[adamemerson]

Do we have to bump the version number if we get the new version in before release? (That is, as long as the 'old' format isn't used in released code?)

[pritha-srivastava]

@adamemerson : People in the team have suggested to bump the number always when we make changes to a structure, even though it may be before a release, reason being that someone may start using master branch and then the changed structure will cause decode to fail. Just to be on the safe side, so that we don't break someone's existing systems, bumping version number has been recommended. @cbodley @mattbenjamin can comment better on this.

[cbodley]

you should be able to leave out the default ctor and init() function - they'll still be accessible from the base class

[pritha-srivastava]

@cbodley: sorry i didnt understand this comment.

[cbodley]

just saying that these overrides don't do anything other than forward the call to RGWOp. if you remove them from RGWRestRole, you'll get the same behavior

[cbodley]

get_params() is an override also

[pritha-srivastava]

@cbodley : get_params() is declared and defined only in CreateRole, DeleteRole etc classes. It is not there in any of the other classes up till RGWOp.

[cbodley]

it might be worth putting these options in their own Role options: section, so they're not confused with other commands

[cbodley]

can oid be const string& here?

[mattbenjamin]

@pritha-srivastava @cbodley these suggestions seem worthwhile, other than that, I think we just need to prove nothing has broken to merge this shortly?

[cbodley]

@mattbenjamin agreed, these certainly aren't blockers

[adamemerson]

On second thought, so we don't have to old an old encode version forever, just ping me when you're ready and I'll grab this branch, stick my stuff on it, and hook in the profile work?

[pritha-srivastava]

@adamemerson : I'm struggling with some build issues at the moment. I have to incorporate some of Casey's comments. I'll do that and let you know.

[pritha-srivastava]

@cbodley : I have incorporated your comments

[cbodley]

@adamemerson i'd like to see this merge asap so we don't have to keep rebasing

regarding the pains of backward-compatible decoding, we should be able to avoid that by removing the policy-as-string stuff from @pritha-srivastava's v1 format of RGWRole encode/decode

then when your work is ready, you can just add it on top in the v2 format

thoughts?

[adamemerson]

If we can just pull the string out entirely and save having to own an unused encode/decode version forever I approve entirely.

Laotian Generals Train Mercenaries.

[aemerson]

FYI I'm unsubscribing as I was mentioned in this thread accidentally, I have nothing to do with this project.

[cbodley]

@pritha-srivastava i assume that trust_policy and perm_policy_map are the strings that @adamemerson will replace later - do you see any issue with removing them from v1 encode/decode, so he can add them to v2 later without worrying about backward compatibility?

[adamemerson]

I'd like to not have a v1/v2 at all from this. If the choice is between it going in now and having to take another version before it's ever practically used and me rebasing it a few times as part of my work to keep it up to master, I'd prefer the second.

[cbodley]

there is really no cost to bumping the version number, as long as you're only adding to an existing format.. when your work is ready, the only diff would be:

   void encode(bufferlist& bl) const {
-    ENCODE_START(1, 1, bl);
+    ENCODE_START(2, 1, bl);
     ::encode(id, bl);
     ::encode(name, bl);
     ::encode(path, bl);
     ::encode(creation_date, bl);
+    ::encode(trust_policy, bl);
+    ::encode(perm_policy_map, bl);
     ENCODE_FINISH(bl);
   }
 
   void decode(bufferlist::iterator& bl) {
-    DECODE_START(1, bl);
+    DECODE_START(2, bl);
     ::decode(id, bl);
     ::decode(name, bl);
     ::decode(path, bl);
     ::decode(creation_date, bl);
+    if (struct_v >= 2) {
+      ::decode(trust_policy, bl);
+      ::decode(perm_policy_map, bl);
+    }
     DECODE_FINISH(bl);
   }

[adamemerson]

All right. It seems untidy and pointless for something that hasn't been released but not that bad.

[pritha-srivastava]

@cbodley @adamemerson : There are places where the input permission document is being read and is being passed as a string to the constructor. These places will also need changes.
Rebased the PR.

[pritha-srivastava]

@adamemerson : A Role also has an ARN which I have coded up to be something like the above. Does this look fine? Or do I need to change this?

[adamemerson]

I think it's mostly fine. One thing Marcus and I had talked about off and on was whether we wanted to use the same namespaces/identifiers as Amazon. (e.g. whether we wanted to use arn:aws: as our namespace prefix.)

I had been leaning toward keeping all our identifiers the same since it improves compatibility and makes copying policies from one place to the other much easier.

The main argument against is that we aren't actually Amazon. I know Marcus at one point suggested making things like that configurable. I'm not sure if there's much benefit to that. (If we have a single IDP backing both AWS STS and Ceph RGW's STS having them both use different ARNs seems a huge negative.)

Anyway, this is a ramble. I think what you have is perfectly fine, but we might want to discuss it tomorrow on the RGW standup just to air the idea out and get feedback in case there's something I'm not thinking of.

[pritha-srivastava]

@adamemerson : I took a look at the IAM Policy Elements link that you shared with me. It looks like ARN plays an important role while deducing permissions for an entity and it will be worthwhile to discuss the format of the ARN with the team for the reasons mentioned above, and come to a conclusion.
Are you planning to attend the standup today(Friday) to discuss this. Because I usually dont attend stand-ups on Fridays (its too late for me here), but if you plan to discuss this today, I will attend. Please let me know. Thanks.

[adamemerson]

Sorry, I caught your answer too late to answer. I'll be at the RGW Core Standup on Monday and we can talk about it then. I think I might just post to ceph-devel and ASK if anyone would find the ability to change it useful.

[adamemerson]

@pritha-srivastava Can you please rebase and repush this?

[pritha-srivastava]

@adamemerson : I have rebased the PR, but I see build failures which aren't related to the code in the PR.

[adamemerson]

Give it another shot when it's convenient? I get the impressin that sometimes gitbuilder is just out to lunch and needs to be prodded until it's working.

Also, did you run an RGW test suite against this at some point?

[pritha-srivastava]

@adamemerson : I rebased again, and this time the build finished successfully. No, I haven't run teuthology against this PR, do you want me to that? I have scheduled a run.

[adamemerson]

Yes please. Run teuthology against this and if it doesn't break anything, I'll merge it in.

[pritha-srivastava]

@adamemerson : The teuthology results are here: http://qa-proxy.ceph.com/teuthology/prsrivas-2017-01-27_12:47:17-rgw-wip_sts_role_rest---basic-smithi/755464/teuthology.log

There is one valgrind failure which is in one of the osds. I discussed this with @cbodley , and he told this can be ignored. All other tests have passed.

[adamemerson]

Thank you. Sorry to ask for another change this late, but can you add a string for another policy? We aren't going to use it YET, but eventually we're going to want both the access-policy and the assume-role policy and having another string thrown in there that gets encoded/decoded will keep us from having to version things later.

[pritha-srivastava]

@adamemerson : I have already added a string to hold the assume-role-policy (trust_policy here) and a map of permission policies (access policies). A role can have more than one access policies attached and also the operations for access policies are such that we need an association between the policy name and the actual policy.

Is there anything else that you want me to add.

[adamemerson]

Sorry about that, I didn't realize we could have multiple access policies. Everything looks good to me, then.