Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docker-compose] Add Filebeat container to ship logs from Mordred to ES #293

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

canasdiaz
Copy link
Contributor

Filebeat container was added to the compose file to ship the logs written by Mordred to the Elasticsearch cluster. A pretty simple configuration file was added to the "default-grimoirelab-settings" folder. As a result Filebeat creates daily indexes in the cluster with the content of the Mordred log file.

The README file was also modified to help the user know how to add the index pattern needed to visualize the logs.

Fixes #276

Signed-off-by: Luis Cañas-Díaz lcanas@bitergia.com

@valeriocos
Copy link
Member

Thank you @sanacl for the PR, I'll have a look at it today

@valeriocos valeriocos self-requested a review February 8, 2020 11:18
Copy link
Member

@valeriocos valeriocos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanacl the idea of having the logs in the instance is really nice! The risk I see is that sensible information (credentials, local paths, db names, etc.) might be exposed in the logs.

Logstash seems to provide a mechanism to filter part of the text in the log messages (https://discuss.elastic.co/t/remove-part-of-message-string/66809). I wasn't able to find the same feature for filebeat. Is there a specific reason to use filebeat? Would like to investigate how to hide parts of the text in the log messages?

@canasdiaz
Copy link
Contributor Author

Logstash seems to provide a mechanism to filter part of the text in the log messages (https://discuss.elastic.co/t/remove-part-of-message-string/66809). I wasn't able to find the same feature for filebeat. Is there a specific reason to use filebeat? Would like to investigate how to hide parts of the text in the log messages?

@valeriocos

Logstash was split years ago into different components. Filebeat is just a component that reads a file a puts its content somewhere else (ES cluster or Logstash). I could have a look to see whether Logstash is still able of reading a file by itself, so we won't need a second container for this dropping. The Logstash filter is able to detect a field a drop the item, but as far as I know it does not replace the content.

In any case, do u think hiding these parameters is important for our community members?

@valeriocos
Copy link
Member

In any case, do u think hiding these parameters is important for our community members?

I would say so. Community members may deploy their own instance, thus providing the means to hide these params (or understand if this is possible) can be useful for the community.

Filebeat container was added to the compose file to ship the logs written by
Mordred to the Elasticsearch cluster. A pretty simple configuration file was added
to the "default-grimoirelab-settings" folder. As a result Filebeat creates
daily indexes in the cluster with the content of the Mordred log file.

The README file was also modified to help the user know how to add the index
pattern needed to visualize the logs.

Signed-off-by: Luis Cañas-Díaz <lcanas@bitergia.com>
@canasdiaz
Copy link
Contributor Author

I've been having a look at this and in order to modify all the places where strings like "https://user:pass" appear, we would need to deploy filebeat + logstash, which from my point of view is overkill for our average community members.

In some cases this is already done by Mordred, but it is not done for all of them.
image

I see two options:

  1. add a new container with logstash and add there a "mutation" to overwrite strings like "https://user:pass"
  2. include a warning in the documention before adding the "mordred" index pattern, making clear some credentials could be exposed (as it happens with the file logs BTW)

What do u think @valeriocos ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GrimoireLab logs should be sent to Elasticsearch
2 participants