whitelisting firehose messages by org

[bonzofenix]

We are currently facing the following situation:

• Stress tests on some apps can take down our ELK cluster.
• We want logsearch and l4cf only consuming operations apps logs that are in some specific cf orgs.

For this reason we would like to have l4cf only parsing logs from whitelisted Orgs.
@mrdavidlaing, @malston I would like your input on this.
Where do you think we should implement this? Among the options that I thought are:

• Firehose-to-syslog
• Cloudfoundry-ingrestor
• log_parser
• Firehose

[mrdavidlaing]

Stress tests on some apps can take down our ELK cluster.

Could you provide some specifics? How much load are you throwing at the apps, and a how many logs / second are they producing through the firehose?

[mrdavidlaing]

take down our ELK cluster.

Also, which bits of the ELK cluster fail?

[bonzofenix]

We are currently persisting around 36gb worth of data per day (on average). Our ES cluster has 4 shards distributed on 8 ES persistent nodes. We seen 20k worth x 5sec. We think the last crash was due to too many writes and a corrupted shard on the ES cluster. We are working on setting disks on different Data stores to avoid disk IO limitations if any. We do not have stats on this so we are not sure if this will help at all. We seen the system crashed on different situations and we scale accordingly.

• Redis not being able to start because the AOF file being too big to load on memory and being restart all the time. -> Fixed by adding parsers to keep up with the load and deleting AOF
• ES 90% heaps when shard was corrupted and cluster was partial offline. -> Fixed by deleting shard.

Those are the once I can recall.

[bonzofenix]

Long term we are thinking on having 2 deployments of Logsearch+l4cf per environment. We want one with predictable growth and longer logs retention for operations that we can monitor and relay on. We will have another deployment that will be available for our cf users which we do not necessary mind if it goes down.

[shinji62]

What about adding a broker like kafka to offload the load to it ...
In firehose-to-syslog we already filter by message type, so maybe filtering here can make sens ...

[simonjohansson]

@shinji62 Do you mead to be able to filter in firehose-to-syslog on arbitrary fields in events?

[bonzofenix]

@shinji62 can you point out to the filtering that you mentioned? @simonjohansson looking into the code of firehose-to-syslog it should be fairly easy to implement. I do not know if the PR will be accepted or not.

[simonjohansson]

@bonzofenix sure, but I doubt firehose-to-syslog is the right place for this as it should just be a forwarder for general messages you want to a aggregation tool.

[mrdavidlaing]

Given the failure scenarios, I think its best to try filter the logs before they hit the queue.

Currently, the logs flow like this:

firehose --> ingestor_cloudfoundry-firehose_ctl / firehose-to-syslog --> ingestor_cloudfoundry-firehose_ctl / logstash --> queue

We could make the logstash part of the ingestor_cloudfoundry-firehose job configurable with extra logstash config.

I think this would be a good place to use logstash's drop filter

I propose we update the ingestor_cloudfoundry-firehose job to support the filters property like the upstream ingestor_syslog / logstash_ingestor.filters job, which would then allow you to configure deployment specific drop rules via your deploy manifest:

properties:
  ingestor_cloudfoundry-firehose:
      filters: |
                if [loglevel] == "debug" {
                   drop { }
                } 

[shinji62]

@simonjohansson Yeah you are right firehose-to-syslog should be simple as possible.
So maybe the @mrdavidlaing idea make more sens.

[MaheshRudrachar]

@mrdavidlaing in the same context, I have another usecase.
whitelist only those logs for a given appname and orgname

At present, I am in process of implementing and customizing the ELK stack, where in user should have flexibility to bind the app to elk and only those app logs needs to be flowed/pushed from doppler to fierhose-to-sylog plugin.. What would be your option and approach to address this usecase. Your valuable inputs are very much appreciated..

Thanks

[hannayurkevich]

Hi guys,

Is it still an issue?

[hannayurkevich]

Closing this issue. Created enhancement #173 because filtering feature can be quite useful.