Odd Openstack Access Behavior

[badllama]

Hey folks,

I'm working on getting Security Monkey working with Openstack and I'm trying to tune the permissions with the team that supports our Openstack environment. They would like to use an existing readonly role, rather than create a new one, and it looks like the permissions are equivalent to those specified in the iam_openstack.md doc. This existing readonly role enables visibility to all of the assets via CLI and Horizon. However, Security Monkey is only getting a subset of the security groups. The main question that I have is why there might be a discrepancy between what's visible in CLI/Horizon versus Security Monkey?

Thanks for the help!

[mstair]

Hi @badllama. What version of OpenStack are you running? Are you just seeing subset of security groups from the same tenant? Do you have multiple regions in your setup?

[badllama]

We're on Pike and it's just one region for the purposes of your question. In a single tenant, if I see 10 security groups in Horizon, SM only sees 4. It's the same account for both access methods, so by definition, it'd be the same role.

[mstair]

I'll look around at what might be problem or where to check for debug. Can you try, in your config.py to set SECURITYGROUP_INSTANCE_DETAIL = NONE, to isolate some issue capturing instance details, then restart the SM services?

[badllama]

Looks like that made it worse, now it can't 'see' any security groups. Lots of 'Problem connecting...' messages.

[mstair]

Odd, that just limits the code that gets run, nothing additional. I should have mentioned that SECURITYGROUP_INSTANCE_DETAIL = FULL is the default, so make sure you are updating that entry vs adding another (look for SECURITYGROUP_INSTANCE_DETAIL).

[badllama]

Ok, nope, I did something silly. I flipped that to 'NONE' and now it can pull all of the Security Groups. What is that actually limiting?

[mstair]

It will not try to associate security group assignment to instances, https://github.com/Netflix-Skunkworks/cloudaux/blob/master/cloudaux/orchestration/openstack/security_group.py#L16.

Can you try to set SECURITYGROUP_INSTANCE_DETAIL = SUMMARY?

[badllama]

SUMMARY results in the same behavior as FULL, which is to say that I have problems connecting again. Is there a specific permission that enables that access? It seems like it may be associated with the nova IAM policy as opposed to the neutron policy, given that it's trying to access information about compute nodes.

[mstair]

Oh good call, yes, its calling the compute service for instance details. See bottom of https://github.com/Netflix/security_monkey/blob/develop/docs/iam_openstack.md for os_compute_api:servers:show policy note.

[badllama]

Ok, was sort of hoping that there was more than that. We already have that permission applied to the nova policy for our role. Would there be more robust failure messages somewhere in logs that I could go check out?

In the course of troubleshooting, I've successfully run the find_changes process (with instance_detail set to FULL) with a different role assigned to the account we're using. We don't want to use the other role because it's TOO permissive. So the problem appears to be IAM-related, but I can't for the life of me figure out where the permissions discrepancy is. I'm sort of hoping that a more robust error message might point to what permission is missing.

[mstair]

I am assuming you already checked the /var/log/security_monkey logs, there is an exception table in the db, I need to go back and remember how you query it.

I use a more permissive account as we do some inline mitigation for issues, so have not used a more restrictive user outside of writing up the instructions. Perhaps with "os_compute_api:servers:show:host_status"?

[badllama]

Yeah, the security_monkey log just shows "Skipping (SG RULE INFO) due to a region-level exception "Problem Connecting to openstack_securitygroup/PROJECTNAME/None:\n'NoneType' object is not iterable" Perhaps that's more helpful to you than it is to me?

I can try the host_status perm.

[mstair]

It is helpful, let me run this down. Go ahead and set back to NONE to get you running, with understanding you won't have instance mappings.

[badllama]

Thanks for all the help @mstair, it's much appreciated.

[mstair]

You mind adding os_compute_api:os-security-groups to your nova policy for your read-only account and trying FULL again? I don't readily have access to a setup where I can modify OS project policies.

https://github.com/openstack/nova/blob/stable/pike/nova/policies/security_groups.py

[badllama]

That did it, looks like everything is working just fine now. Thanks again!

[mstair]

Great, I'll update docs!