Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Amazon AWS Config event catcher #1049

Merged
merged 1 commit into from Dec 2, 2014

Conversation

Projects
None yet
3 participants
@blomquisg
Copy link
Member

blomquisg commented Nov 12, 2014

Amazon has launched the new AWS Config service that emits configuration diffs on
a scheduled basis. The new AWS Config event catcher listens to an SQS Queue and
discovers configuration changes. These configurations changes are parsed and
treated as an event stream from AWS.

Each appliance that attempts to receive events for a single AWS account will
create a unique SQS queue, allowing multiple appliances to listen for the same
events.

Captured events are parsed to determine the type of object changed and the
manner in which it was changed. For instance, when a stopped instance is
started, the configuration diff will indicate that the item changed was an
instance, and that the state changed from stopped to starting. When this
configuration diff is recevied, the Amazon Event Monitor enqueues a event called
"AWS_EC2_Instance_STARTED".

There are still other events that need to be detemined from the event stream,
but an initial set has been added to the event handling file to illustrate the
types of events ManageIQ will process.

The only requirement for now imposed on the AWS account configuration is that
the SNS Topic configured for AWS Config needs to be called "AWSConfig_topic".
With that in place, ManageIQ's Amazon Event Monitor can create the appropriate
SQS queue and begin receiving events.

https://trello.com/c/qdeChZPC

@blomquisg blomquisg force-pushed the blomquisg:amazon-events branch Nov 13, 2014

@Fryguy Fryguy added the enhancement label Nov 13, 2014

@blomquisg blomquisg force-pushed the blomquisg:amazon-events branch Nov 13, 2014

@Fryguy

View changes

lib/Amazon/events/amazon_event_monitor.rb Outdated
#
def initialize(aws_access_key_id,
aws_secret_access_key,
aws_region, queue_id,

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

If you're going to multline it (which I personally don't like), then you should split up every line so it's clear there are 5 paraetmers...this particular line could get overlooked.

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

I have four choices:

  1. take the arguments in as an options hash
  2. shrink the arg names down to illegible short names
  3. break the rubocop rule (leave the line long)
  4. multiline

I guess there's a fifth option: 5. stop taking in so many freaking arguments...

I don't mind breaking the rubocop rule and leaving it a long line.

This comment has been minimized.

Copy link
@Fryguy

Fryguy Dec 1, 2014

Member

I like single line personally and just ignore rubocop.

queue_name = "manageiq-awsconfig-queue-#{@queue_id}"

begin
$aws_log.debug("#{log_header} Looking for Amazon SQS Queue #{queue_name} ...")

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

Can we possibly create a logger / logger= method, and let the caller call AmazonEventMonitor.new.logger = $aws_log. That way, we're not dependent on this global var to be made, and for debugging it can defaulted to Logger.new(STDOUT)

In fact, for pluggable providers stuff, I think a logging mechanism like this will be needed everywhere eventually.

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

I like this. But, I would rather wait until Pluggable Providers to consolidate it all at once.

Thoughts?

@Fryguy

View changes

lib/Amazon/events/amazon_event_monitor.rb Outdated
state = "STOPPED"
when "running"
state = "STARTED"
end

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

Is this necessary? Can we just emit separate events for each individual state?

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

We could ... it will mean some overlap in the event handling. But, it would mean less maintenance here in this logic.

So, the maintenance moves to the event_handling.yml, which is easier to maintain.

Yeah, sure ... I'll do that.

@Fryguy

View changes

lib/Amazon/events/amazon_event_monitor.rb Outdated

def sqs
@sqs ||= connect(@aws_access_key_id, @aws_secret_access_key, "SQS", @aws_region)
end

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

I feel like this is duplicated from the AmazonConnection class itself, or perhaps belongs there.

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

Belongs there...

Moving it.

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

Actually, sort of.

I'm gonna add class (stateless) methods for getting directly to SNS and SQS services. However, they'll still require all the other connection information (access_key_id, secret_access_key, etc...). So, I'm also going to leave these instance (stateful) methods here for getting and keeping SNS and SQS connections in the event monitor.

Thoughts?

@Fryguy

View changes

lib/Amazon/events/amazon_event_monitor.rb Outdated

def connect(access_key_id, secret_access_key, service, region)
AmazonConnection.raw_connect(access_key_id, secret_access_key, service, region)
end

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

Can we add a with_connection method and use that instead of calling connect directly?

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

Add with_connection to AmazonConnection? Or, here? Or, you don't care where...

The only thing with with_connection is that I don't really see it buys us anything. Unless I'm understanding it wrong...

The way I see it, we still have to pass in all the same arguments. And, since these are restful-style calls, we don't close a connection when we're done with it. So, the with_connection isn't doing resource cleanup.

Now, I could see changing the AmazonConnection class to be stateful with an initializer that takes in the access_key_id and the secret_access_key (and maybe region and proxy_uri). Then, with_connection could be called with just the service name. And, each time the internal workings of AmazonConnection would be handling building the individual connections.

Maybe that makes more sense. I'm not sure it should be tied to this PR, since that could probably effect other unrelated places in the code.

This comment has been minimized.

Copy link
@Fryguy

Fryguy Dec 1, 2014

Member

It's more of a consistency thing...all providers provider the same API. For restful stuff, "close" is a no-op and that's fine

sleep_poll_normal
end
ensure
reset_event_monitor_handle

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

Should this be stop?

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

All the other event catchers reset here. If it should be stop, let's address that in pluggable providers.

end

def monitor_events
event_monitor_handle.start

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 14, 2014

Member

If start stop pair are needed can we create a with_event_monitor_handle method that does the start/stop logic?

This comment has been minimized.

Copy link
@blomquisg

blomquisg Nov 20, 2014

Author Member

Probably ... this sounds like a good task for the Pluggable Providers, Eventing Edition.

Adding Amazon AWS Config event catcher
Amazon has launched the new AWS Config service that emits configuration diffs on
a scheduled basis.  The new AWS Config event catcher listens to an SQS Queue and
discovers configuration changes.  These configurations changes are parsed and
treated as an event stream from AWS.

Each appliance that attempts to receive events for a single AWS account will
create a unique SQS queue, allowing multiple appliances to listen for the same
events.

Captured events are parsed to determine the type of object changed and the
manner in which it was changed.  For instance, when a stopped instance is
started, the configuration diff will indicate that the item changed was an
instance, and that the state changed from stopped to starting.  When this
configuration diff is recevied, the Amazon Event Monitor enqueues a event called
"AWS_EC2_Instance_STARTED".

There are still other events that need to be detemined from the event stream,
but an initial set has been added to the event handling file to illustrate the
types of events ManageIQ will process.

The only requirement for now imposed on the AWS account configuration is that
the SNS Topic configured for AWS Config needs to be called "AWSConfig_topic".
With that in place, ManageIQ's Amazon Event Monitor can create the appropriate
SQS queue and begin receiving events.

Trello:
Providers: AWS Support for new Config Service
https://trello.com/c/qdeChZPC

@blomquisg blomquisg force-pushed the blomquisg:amazon-events branch to dff42ca Dec 1, 2014

@miq-bot

This comment has been minimized.

Copy link
Member

miq-bot commented Dec 1, 2014

Checked commit blomquisg@dff42ca with rubocop 0.27.1
7 files checked, 15 offenses detected

lib/Amazon/events/amazon_event_monitor.rb

vmdb/app/models/ems_event.rb

vmdb/app/models/ems_event/parsers/amazon.rb

vmdb/lib/workers/event_catcher_amazon.rb

Fryguy added a commit that referenced this pull request Dec 2, 2014

Merge pull request #1049 from blomquisg/amazon-events
Adding Amazon AWS Config event catcher

@Fryguy Fryguy merged commit b26a7a2 into ManageIQ:master Dec 2, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details

@Fryguy Fryguy deleted the blomquisg:amazon-events branch Dec 2, 2014

@Fryguy Fryguy added this to the Sprint 16 Ending Dec 2, 2014 milestone Dec 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.