Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding fields 'aws_log_group' and 'aws_log_stream' to flow log and raw log codecs #55

Merged
merged 2 commits into from
Nov 9, 2017
Merged

Adding fields 'aws_log_group' and 'aws_log_stream' to flow log and raw log codecs #55

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
979cb1d
Adding fields 'aws_log_group' and 'aws_log_stream' to flow log and ra…
Nov 2, 2017
5ac1277
Extracted new 'aws_log_group' and 'aws_log_stream' fields to constants.
Nov 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/main/java/org/graylog/aws/cloudwatch/CloudWatchLogData.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@
* </pre>
*/
public class CloudWatchLogData {

@JsonProperty("logEvents")
public List<CloudWatchLogEvent> logEvents;

@JsonProperty("logGroup")
public String logGroup;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this ever by null (i. e. not exist)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the documentation and example, there is nothing to suggest that this field is optional / might not be present in any situation. Additionally, in all of my testing to date it has never not been present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅


@JsonProperty("logStream")
public String logStream;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this ever by null (i. e. not exist)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the documentation and example, there is nothing to suggest that this field is optional / might not be present in any situation. Additionally, in all of my testing to date it has never not been present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅

}
4 changes: 3 additions & 1 deletion src/main/java/org/graylog/aws/inputs/codecs/CloudWatchFlowLogCodec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public CloudWatchFlowLogCodec(@Assisted Configuration configuration, @AWSObjectM

@Nullable
@Override
public Message decodeLogData(@Nonnull final CloudWatchLogEvent logEvent) {
public Message decodeLogData(@Nonnull final CloudWatchLogEvent logEvent, @Nonnull final String logGroup, @Nonnull final String logStream) {
try {
final FlowLogMessage flowLogMessage = FlowLogMessage.fromLogEvent(logEvent);

Expand All @@ -51,6 +51,8 @@ public Message decodeLogData(@Nonnull final CloudWatchLogEvent logEvent) {
flowLogMessage.getTimestamp()
);
result.addFields(buildFields(flowLogMessage));
result.addField("aws_log_group", logGroup);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please extract the key names as constants into the existing AWS class and use them in CloudWatchFlowLogCodec and in CloudWatchRawLogCodec.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've extracted these fields to constants.

result.addField("aws_log_stream", logStream);
result.addField(AWS.SOURCE_GROUP_IDENTIFIER, true);

return result;
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/graylog/aws/inputs/codecs/CloudWatchLogDataCodec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ public Collection<Message> decodeMessages(@Nonnull RawMessage rawMessage) {

for (final CloudWatchLogEvent logEvent : data.logEvents) {
try {
final Message message = decodeLogData(logEvent);
final Message message = decodeLogData(logEvent, data.logGroup, data.logStream);
if (message != null) {
messages.add(message);
}
Expand All @@ -60,7 +60,7 @@ public Collection<Message> decodeMessages(@Nonnull RawMessage rawMessage) {
}

@Nullable
protected abstract Message decodeLogData(@Nonnull final CloudWatchLogEvent event);
protected abstract Message decodeLogData(@Nonnull final CloudWatchLogEvent event, @Nonnull final String logGroup, @Nonnull final String logStream);

@Nonnull
@Override
Expand Down
12 changes: 10 additions & 2 deletions src/main/java/org/graylog/aws/inputs/codecs/CloudWatchRawLogCodec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,18 @@ public CloudWatchRawLogCodec(@Assisted Configuration configuration, @AWSObjectMa

@Nullable
@Override
public Message decodeLogData(@Nonnull final CloudWatchLogEvent logEvent) {
public Message decodeLogData(@Nonnull final CloudWatchLogEvent logEvent, @Nonnull final String logGroup, @Nonnull final String logStream) {
try {
final String source = configuration.getString(CloudTrailCodec.Config.CK_OVERRIDE_SOURCE, "aws-raw-logs");
return new Message(logEvent.message, source, new DateTime(logEvent.timestamp));
Message result = new Message(
logEvent.message,
source,
new DateTime(logEvent.timestamp)
);
result.addField("aws_log_group", logGroup);
result.addField("aws_log_stream", logStream);

return result;
} catch (Exception e) {
throw new RuntimeException("Could not deserialize AWS FlowLog record.", e);
}
Expand Down