New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[apps] Aliyun ActionTrail app #792

Merged
merged 8 commits into from Jul 27, 2018

Conversation

Projects
None yet
4 participants
@GarretReece
Contributor

GarretReece commented Jul 25, 2018

to: @airbnb/streamalert-maintainers
cc:
size: medium

Background

SteamAlert doesn't have an app that collects ActionTrail events from Aliyun

Changes

  • Adds an app that collects ActionTrail events from Aliyun
  • Schema definition for the new log type is included
  • Unit tests for the app as well as schema validation tests are included
  • Documentation for authorizing the app is included in the README in the apps directory as well as in a section of the app-configuration.rst in the docs directory

Testing

Barebones StreamAlert session created, the Aliyun app configured and deployed, and verifying that events are listed as captured in the log

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 25, 2018

Coverage Status

Coverage increased (+0.03%) to 97.418% when pulling 32a1757 on trailofbits:garret/actiontrail_app into fae89fb on airbnb:master.

coveralls commented Jul 25, 2018

Coverage Status

Coverage increased (+0.03%) to 97.418% when pulling 32a1757 on trailofbits:garret/actiontrail_app into fae89fb on airbnb:master.

@ryandeivert

this is nice work!! added a few quick first round comments :)

Show outdated Hide outdated app_integrations/apps/aliyun.py
from app_integrations import LOGGER
from app_integrations.apps.app_base import StreamAlertApp, AppIntegration

This comment has been minimized.

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

add an extra new line here please (2 lines between imports and class declarations)

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

add an extra new line here please (2 lines between imports and class declarations)

Show outdated Hide outdated app_integrations/apps/aliyun.py
Show outdated Hide outdated app_integrations/apps/aliyun.py
Show outdated Hide outdated app_integrations/apps/aliyun.py
'access_key_id': {
'description': ('The access key id generated for a RAM user. This '
'should be a string of alphanumeric characters.'),
'format': re.compile(r'.*')

This comment has been minimized.

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

are there no restrictions similar to AWS for their access and secret keys? ie - AWS access keys are 20 alphanumeric all-caps characters and must start with A, etc

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

are there no restrictions similar to AWS for their access and secret keys? ie - AWS access keys are 20 alphanumeric all-caps characters and must start with A, etc

This comment has been minimized.

@GarretReece

GarretReece Jul 26, 2018

Contributor

I can't find any documentation anywhere regarding the format. The values I have are a mix of digits and upper and lower case letters, with the id 16 characters in length and the secret 30 characters in length. I'm perfectly willing to put those in as restrictions, but I don't have any reference for that being the supported format.

@GarretReece

GarretReece Jul 26, 2018

Contributor

I can't find any documentation anywhere regarding the format. The values I have are a mix of digits and upper and lower case letters, with the id 16 characters in length and the secret 30 characters in length. I'm perfectly willing to put those in as restrictions, but I don't have any reference for that being the supported format.

'region_id': {
'description': ('The region for the Aliyun API. This should be '
'a string like \'ap-northeast-1\'.'),
'format': region_validator

This comment has been minimized.

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

++ this is great

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

++ this is great

Returns:
int: Number of seconds the polling function should sleep for
"""
return 0

This comment has been minimized.

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

I think if this is 0 it can be left off completely. I do like the context you've added in the docstring, though, so I'm not sure if I feel strongly about removing the method

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

I think if this is 0 it can be left off completely. I do like the context you've added in the docstring, though, so I'm not sure if I feel strongly about removing the method

@@ -203,6 +203,8 @@ class AppIntegrationPackage(LambdaPackage):
package_name = 'stream_alert_app'
precompiled_libs = {'boxsdk[jwt]==2.0.0a11'}
third_party_libs = {
'aliyun-python-sdk-core==2.8.5',

This comment has been minimized.

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

this is great - thank you for pinning the version!

@ryandeivert

ryandeivert Jul 25, 2018

Contributor

this is great - thank you for pinning the version!

Show outdated Hide outdated app_integrations/apps/aliyun.py
@chunyong-lin

Hey @GarretReece thanks for working on this App. It looks great too me. I added some comments.

https://www.alibabacloud.com/help/doc-detail/28849.htm
"""
_MAX_RESULTS = 50

This comment has been minimized.

@chunyong-lin

chunyong-lin Jul 25, 2018

Contributor

Any reason we don't pull 100 results each time?

@chunyong-lin

chunyong-lin Jul 25, 2018

Contributor

Any reason we don't pull 100 results each time?

This comment has been minimized.

@GarretReece

GarretReece Jul 26, 2018

Contributor

The documentation lists the range for the MaxResult parameter as 0-50; I'll admit I haven't tested with higher values to see if it worked anyhow.

@GarretReece

GarretReece Jul 26, 2018

Contributor

The documentation lists the range for the MaxResult parameter as 0-50; I'll admit I haven't tested with higher values to see if it worked anyhow.

This comment has been minimized.

@chunyong-lin

chunyong-lin Jul 26, 2018

Contributor

Can I have one more testing request? Can you add a test case when NextToken is available in your testing environment? (I am not talking about unit test)

A while ago, I have a simple script to retrieve ActionTrail events, and set max results per request to 2. I encountered the issue that NextToken doesn't take effect, and the 2nd request returned as same events as 1st request recieved.
Can you verify this in the testing environment? Thanks!

@chunyong-lin

chunyong-lin Jul 26, 2018

Contributor

Can I have one more testing request? Can you add a test case when NextToken is available in your testing environment? (I am not talking about unit test)

A while ago, I have a simple script to retrieve ActionTrail events, and set max results per request to 2. I encountered the issue that NextToken doesn't take effect, and the 2nd request returned as same events as 1st request recieved.
Can you verify this in the testing environment? Thanks!

Show outdated Hide outdated app_integrations/apps/aliyun.py
@@ -1,3 +1,5 @@
aliyun-python-sdk-core==2.8.5
aliyun-python-sdk-actiontrail==2.0.0

This comment has been minimized.

@chunyong-lin

chunyong-lin Jul 25, 2018

Contributor

Can you add these two packages to requirements-top-level.txt as well?

@chunyong-lin

chunyong-lin Jul 25, 2018

Contributor

Can you add these two packages to requirements-top-level.txt as well?

This comment has been minimized.

@GarretReece

GarretReece Jul 26, 2018

Contributor

added them in!

@GarretReece

GarretReece Jul 26, 2018

Contributor

added them in!

Show outdated Hide outdated app_integrations/apps/aliyun.py
Show outdated Hide outdated app_integrations/apps/aliyun.py
Show outdated Hide outdated conf/logs.json
@chunyong-lin

LGTM! 🎉

@chunyong-lin chunyong-lin merged commit 18735de into airbnb:master Jul 27, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.03%) to 97.418%
Details

@dguido dguido deleted the trailofbits:garret/actiontrail_app branch Jul 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment