New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lambda policies need a virtualenv installation. #193

Closed
frc9 opened this Issue Jun 11, 2016 · 35 comments

Comments

Projects
5 participants
@frc9

frc9 commented Jun 11, 2016

Getting the following error in the cloudwatch log stream associated with the lambda function provisioned after running a policy:

Unable to import module 'custodian_policy': No module named pkg_resources

using the policy below and invoking it with custodian run -c policy.yml -s out --log-group=/cloud-custodian/dev-account/us-west-2 --region us-west-2 -m

Is there something I'm missing that is causing the mu to not create the lambda function zip file correctly?

policies:
- name: ec2-require-non-public-and-encrypted-volumes
  resource: ec2
  description: |
   Provision a lambda and cloud watch event target
   that looks at all new instances not in an autoscale group
   and terminates those with unencrypted volumes.
  mode:
   type: cloudtrail
   role: arn:aws:iam::XXXXXXXXXXXX:role/custodianRole
   events:
  - RunInstances
    filters:
  - "tag:aws:autoscaling:groupName": absent
  - type: ebs
    key: Encrypted
    value: false
    actions:
  - terminate
- name: tag-compliance
  resource: ec2
  description:
   Schedule a resource that does not meet tag compliance policies
   to be stopped in four days.
  filters:
  - State.Name: running
  - "tag:Environment": absent
  - "tag:AppId": absent
  - or:
    - "tag:OwnerContact": absent
    - "tag:DeptID": absent
      actions:
  - type: mark-for-op
    op: stop
    days: 4
@kapilt

This comment has been minimized.

Member

kapilt commented Jun 11, 2016

I haven't seen that one before. It sounds like an issue with the installed environment that's uploading. Could you provide a few more details around the machine your using to run custodian, specifically

  • what python version is this?
  • which os ?
  • How did you install custodian?
  • Which custodian version are you using? ($ custodian version)
@frc9

This comment has been minimized.

frc9 commented Jun 12, 2016

Hey - false alarm. I did a fresh install and reran my policies and this issue went away.

@frc9 frc9 closed this Jun 12, 2016

@kapilt kapilt changed the title from Invocation errors within auto provisioned lamba functions to check both dist-packages/site-packages within virtualenv Jun 13, 2016

@kapilt

This comment has been minimized.

Member

kapilt commented Jun 13, 2016

i got a report in gitter 1x1 that

seems that "mu.py" was searching for "pkg_resources" in the python2.7/site-packages path, rather than python2.7/dist-packages
updating that path directive and removing the mu.pyc has made a build that works 

we should check both, linux distros have some variations here wrt to location on install that we should respect.

@kapilt kapilt reopened this Jun 13, 2016

@kapilt kapilt closed this Sep 6, 2016

@barundel

This comment has been minimized.

barundel commented Jan 12, 2017

Hello

Hope I can re-open this as am getting the same thing and unable to get past it based on what has already been discussed.

Error when running lambda:
Unable to import module 'custodian_policy': No module named pkg_resources.extern

I updated mu.py line 205 to be dist-packages but that produced the same error with .extern before it was just:

Unable to import module 'custodian_policy': No module named pkg_resources

I have confirmed that pkg_resources exist, tried a fresh AMI, pip install --upgrade setuptools,
pip install --upgrade distribute but no luck

what python version is this? - 2.7.12
which os ? - Amazon Linux AMI
How did you install custodian? - sudo pip install c7n
Which custodian version are you using? ($ custodian version) - 0.8.22.0

@kapilt

This comment has been minimized.

Member

kapilt commented Jan 12, 2017

at this time.. we need a virtualenv installation for lambda policies

@kapilt kapilt reopened this Jan 12, 2017

@barundel

This comment has been minimized.

barundel commented Jan 13, 2017

Thanks

virtualenv was installed in usr/lib/python2.7/dist-packages but not installed to usr/local/lib/python2.7/site-packages where pkg_resources is currently

I removed and installed it to usr/local/lib/python2.7/site-packages and rebuilt with no success

I'm going to try and uninstall and reinstall the packages to the usr/local python, see what that does.

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

Could you please suggest how to do this virtualenv installation as well ?

I am getting the same error

@barundel

This comment has been minimized.

barundel commented Jan 27, 2017

$ [sudo] pip install virtualenv

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

okay thanks let me try

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

it doesn't work. I did below

root@ip-10-14-15-77 ec2-user]# pip install virtualenv
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/local/lib/python2.7/site-packages

i see installation files here
[root@ip-10-14-15-77 ec2-user]# ls -lrt /usr/local/lib/python2.7/site-packages

But still get below error

Unable to import module 'custodian_policy': No module named pkg_resources

Also there is no files under
[root@ip-10-14-15-77 ec2-user]# ls -lrt /usr/lib/python2.7/site-packages/
total 0

Please advise what i am doing wrong?

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

I am unable to proceed further on this.

can anyone help me how to resolve this

I ran below commands
pip install c7n
pip install virtualenv

After that i executed my policy. When i run my lambda function i get the below error

Unable to import module 'custodian_policy': No module named pkg_resources

Please advise how to resolve this issue

@barundel

This comment has been minimized.

barundel commented Jan 27, 2017

What OS are you running this on?
Update:
ok I see your using ec2-user so its amazon Linux?

I resolved the issue by switching OS to ubuntu

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

Its amazon linux AMI free tier
4.4.5-15.26.amzn1.x86_64

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 27, 2017

okay thx let me try that

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 30, 2017

I spinned-up an Ubuntu server in Amazon and did the following

pip install virtualenv
pip install c7n

The lambda functions that is getting created still throws the below error

TART RequestId: 6d15b00a-e701-11e6-879f-891763142e64 Version: $LATEST
Unable to import module 'custodian_policy': No module named pkg_resources

END RequestId: 6d15b00a-e701-11e6-879f-891763142e64

any suggesting please ?

This is blocking our evaluation of cloud custodian

@kapilt

This comment has been minimized.

Member

kapilt commented Jan 30, 2017

you have to create and activate the virtualenv, merely installing is not sufficient. ie

$ virtualenv custodian
$ source custodian/bin/activate
(custodian)$ pip install c7n
@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 30, 2017

thx kapil, let me try this

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 30, 2017

it didn't work, but i am going to re-create my ubuntu instance and do a clean installation. will let you know.

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 31, 2017

Now i get a different error

Unable to import module 'custodian_policy': No module named packaging.version

please advise what i am doing wrong here

@vijayrcse

This comment has been minimized.

vijayrcse commented Jan 31, 2017

I just reverted back to Amazon linux AMI and it worked now ( under virtualenv)

Thanks for the help Kapil, barundel...

@kapilt

This comment has been minimized.

Member

kapilt commented Feb 11, 2017

that packaging.version issue is resolved in 0.8.23.0 release

@vijayrcse

This comment has been minimized.

vijayrcse commented Feb 12, 2017

thanks kapil will take latest release

@kapilt kapilt added this to Priority BackLog in blue team Feb 23, 2017

@kapilt

This comment has been minimized.

Member

kapilt commented Feb 23, 2017

@whit537 could you have a look at this one.

@kapilt kapilt changed the title from check both dist-packages/site-packages within virtualenv to lambda policies need a virtualenv installation. Feb 23, 2017

@kapilt

This comment has been minimized.

Member

kapilt commented Feb 23, 2017

nutshell we need check both dist-packages/site-packages within virtualenv also some variation on linux distros.

@kapilt

This comment has been minimized.

Member

kapilt commented Feb 23, 2017

in the meantime we should update the install docs, till this is resolved.

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Feb 23, 2017

@whit537 could you have a look at this one.

Sure thing!

@chadwhitacre chadwhitacre moved this from Priority BackLog to In Progress in blue team Feb 24, 2017

@chadwhitacre chadwhitacre self-assigned this Feb 24, 2017

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Feb 24, 2017

in the meantime we should update the install docs, till this is resolved.

#983

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Feb 24, 2017

Here's the test case I've whittled down to:

Steps

  1. Spin up and login to an EC2 instance (Amazon Linux AMI, free tier).
  2. sudo pip install c7n to install globally.
  3. Create file policy.yml with below content.
  4. Populate ~/.aws/credentials.
  5. custodian run policy.yml -s . --region us-east-1 to install a Lambda.
  6. Hit the Lambda using the "test" feature in the Console.
policies:
- name: foo
  resource: ec2
  mode:
    type: cloudtrail
    role: arn:aws:iam::644160558196:role/custodian-mu
    events:
     - RunInstances

Expected

'detail': KeyError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 78, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 528, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 332, in run
resources = self.resolve_resources(event)
File "/var/task/c7n/policy.py", line 306, in resolve_resources
resource_ids = CloudWatchEvents.get_ids(event, mode)
File "/var/task/c7n/cwe.py", line 136, in get_ids
resource_ids = cls.get_trail_ids(event, mode)
File "/var/task/c7n/cwe.py", line 107, in get_trail_ids
event_name = event['detail']['eventName']
KeyError: 'detail'

Actual

Unable to import module 'custodian_policy': No module named ipaddress
@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Feb 25, 2017

nutshell we need check both dist-packages/site-packages within virtualenv also some variation on linux distros.

In the Amazon Linux case I'm working with, the libraries we need are hiding out in /usr/lib/python2.7/site-packages, whereas we're looking under /usr/local.

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Feb 25, 2017

Okay! PR in #986.

@barundel

This comment has been minimized.

barundel commented Feb 28, 2017

I'm now getting "Unable to import module 'custodian_policy': No module named ipaddress" after upgrade so waiting eagerly for fix :)

@kapilt

This comment has been minimized.

Member

kapilt commented Feb 28, 2017

if you use a virtualenv for your custodian install this issue won't effect you. this fix is about making it work with system python installations.

@chadwhitacre chadwhitacre removed their assignment Mar 13, 2017

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Mar 17, 2017

This should be closeable now that #997 is in.

@chadwhitacre chadwhitacre moved this from In Progress to Review in blue team Mar 17, 2017

@chadwhitacre

This comment has been minimized.

Contributor

chadwhitacre commented Mar 17, 2017

I've manually tested the mailer and sentry tools on master. Both still work. 👍

@kapilt

This comment has been minimized.

Member

kapilt commented Mar 17, 2017

awesome

@kapilt kapilt closed this Mar 17, 2017

@chadwhitacre chadwhitacre moved this from Review to Done in blue team Mar 17, 2017

kapilt added a commit that referenced this issue May 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment