Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS Kinesis Enhancements #279

Merged
merged 70 commits into from Dec 17, 2019
Merged

AWS Kinesis Enhancements #279

merged 70 commits into from Dec 17, 2019

Conversation

@kyleknighted
Copy link
Contributor

kyleknighted commented Sep 26, 2019

Overview

This PR makes the following changes for the graylog-plugin-integrations AWS Kinesis/CloudWatch input:

  1. Improve authorization support: Previously, only an explicit AWS Key/Secret could be used to authorize the input. Now, the user can pick between using a key/secret and automatic AWS authorization, which uses the DefaultAWSCredentialsProviderChain, which allows many more flexible authentication options.

image

See #264 for more info.

As part of this change, the Assume Role ARN option was also moved to the first page (previously was tucked-away on a subsequent page, which was too late, since many AWS SDK calls are initiate starting from the first page):

image

  1. Add ability to specify VPC endpoints for all AWS API services used (CloudWatch, DynamoDB, IAM, and Kinesis). Fields for this are now present on the first page of the setup, since the endpoints must be used throughout the setup wizard. See #271 for more info.

image

Lots of code was changed to implement these, because many API calls support this functionality, and almost all of them had to be updated :)

Closes #264, #271

@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Sep 30, 2019

I am planning to integrate the backend changes into this PR too. I'll start work on that tomorrow or Wednesday.

@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Oct 16, 2019

@kyleknighted I have begun working on the backend for this. Sorry for the delay!

@danotorrey danotorrey closed this Oct 16, 2019
@danotorrey danotorrey reopened this Oct 16, 2019
@danotorrey danotorrey self-assigned this Oct 16, 2019
@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Oct 16, 2019

@kyleknighted I noticed this error in the browser when running this branch:

client.js:196 /Users/danieltorrey/workspace/graylog/master/graylog-project-repos/graylog-plugin-integrations/src/web/aws/common/awsAuth.jsModule not found: Error: Can't resolve './awsKeySecret' in '/Users/danieltorrey/workspace/graylog/master/graylog-project-repos/graylog-plugin-integrations/src/web/aws/common'

@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Oct 28, 2019

@kyleknighted I have begun working on the backend for this again. I'l keep you updated on my progress.

danotorrey and others added 7 commits Oct 28, 2019
The initial policy needs to be fully resolved before attempting to assume a role. This is because the initial role must have the sts:AssumeRole permission to assume a role. This also allows any authentication scenario to support assuming a role. This matches the auth logic previously supported in the original AWS plugin.
Allow nullable assume role ARN as it will not always be specified.
@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Oct 30, 2019

@kyleknighted Initial testing looks good so far. Noticed one possible issue:

Looks like the Assume Role ARN is not being passed in the Streams request (specified in input field, but is not included in the post request JSON payload):

image

@danotorrey danotorrey requested review from ceruleancee and alex-konn Dec 11, 2019
@lingpri lingpri closed this Dec 11, 2019
@lingpri lingpri reopened this Dec 11, 2019
@lingpri

This comment has been minimized.

Copy link
Contributor

lingpri commented Dec 11, 2019

@danotorrey @kyleknighted @ceruleancee Great work ! If you need any help with any specific task Please keep me posted. I had a chance to look at creating the kinesis input with Claudia's help and we are able to see the log messages. Also, we weren't sure how to test the VPC endpoints(Dynamo DB API Endpoint override ). Is there a Swagger documentation for these end points, that we can access. Thanks.

@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Dec 16, 2019

@lingpri @ceruleancee I investigated and found that there are valid use cases to allow an override endpoint for the IamClient AWS API URL. I feel that we should provide endpoint overrides for all AWS SDK services that we communicate to (just as the PR is currently coded to do).

Please also note that for the security concern of allowing IAM overrides, we are also adding URL white list capability, which allows the user to control which URLs are allowed to be used within Graylog. I will add support for this in a later PR.

@danotorrey

This comment has been minimized.

Copy link
Contributor

danotorrey commented Dec 16, 2019

@ceruleancee @lingpri This PR is ready to be merged unless you have any other concerns.

@danotorrey danotorrey requested a review from lingpri Dec 16, 2019
Copy link
Contributor

lingpri left a comment

Great Work . Thank you very much for your patience. I enjoyed reviewing the PR .

import com.google.auto.value.AutoValue;
import org.graylog.autovalue.WithBeanGetter;

import javax.annotation.Nullable;

This comment has been minimized.

Copy link
@ceruleancee

ceruleancee Dec 17, 2019

Contributor

delete unneeded imports
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.JsonProperty;
import javax.annotation.Nullable;

@ceruleancee ceruleancee merged commit c3301d8 into master Dec 17, 2019
3 checks passed
3 checks passed
ci-web-linter Jenkins build graylog-plugin-pr-linter-check 633 has succeeded
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 7425 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details
@ceruleancee ceruleancee deleted the aws_multi_auth branch Dec 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.