Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VCP Flowlog seem to broken now #6

Open
kyokoshome opened this issue Feb 26, 2019 · 1 comment
Open

VCP Flowlog seem to broken now #6

kyokoshome opened this issue Feb 26, 2019 · 1 comment

Comments

@kyokoshome
Copy link

I am following the guide at Readme.md on MAC, but failed at import log step:

$ python importLogs.py --logtype vpc --logdir /Users/wumark/workspace/temp/vpclog/

Beginning import process
Creating mapping in ES for index: vpc_flowlogs
Creating Ingest Pipeline for index: vpc_flowlogs
Creating new index-pattern in .kibana index
Setting formatted fields on index-pattern
Setting index-pattern as default index
Deleting useless index-patterns in .kibana index
Deleting index-pattern: .ml-anomalies-*
Deleting index-pattern: .ml-notifications
importing saved objects into Kibana
Begin importing log files
File: .DS_Store is not the correct format. File need to end with *.gz
Importing log file: /Users/wumark/workspace/temp/vpclog//667164967571_vpcflowlogs_ap-northeast-1_fl-09efe29fb030b37b0_20190225T0645Z_9ba3c655.log.gz
Traceback (most recent call last):
File "importLogs.py", line 356, in
loadFiles()
File "importLogs.py", line 211, in loadFiles
processFiles(f)
File "importLogs.py", line 168, in processFiles
for i in status:
File "/usr/local/aws/lib/python2.7/site-packages/elasticsearch/helpers/init.py", line 306, in parallel_bulk
_chunk_actions(actions, chunk_size, max_chunk_bytes, client.transport.serializer)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
elasticsearch.helpers.BulkIndexError: (u'500 document(s) failed to index.', [{u'index': {u'status': 500, u'_type': u'vpc_flowlogs', u'_index': u'vpc_flowlogs', u'error': {u'caused_by': {u'caused_by': {u'reason': u'Provided Grok expressions do not match field value: [version account-id interface-id srcaddr dstaddr sr (Skip)

@kyokoshome
Copy link
Author

Sample VPC log format now:

2 667162227571 eni-1231231231231231 10.3.14.217 1.1.1.1 29800 443 6 9 814 1551076978 1551077038 ACCEPT OK
2 667164967571 eni-1231231231231231 10.3.14.217 2.2.2.2 12191 443 6 21 5692 1551076978 1551077038 ACCEPT OK
2 667164967571 eni-1231231231231231 3.3.3.3 10.3.14.217 443 5949 6 4032 161316 1551076978 1551077038 ACCEPT OK

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant