Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
iamigo committed Jul 17, 2019
1 parent 2ce6f4a commit 38ceca5
Showing 1 changed file with 55 additions and 22 deletions.
77 changes: 55 additions & 22 deletions README.md
@@ -1,37 +1,70 @@
# refocus-logging

Common logging service for Refocus and its associated services.
Common logging service for Refocus and its associated services using
[Apache Kafka](https://kafka.apache.org/).

This project uses [Apache Kafka](https://kafka.apache.org/) to get receive messages and log them. You can
publish your log messages from your app to [Topics](https://kafka.apache.org/intro#intro_topics) in the Kafka Cluster and this app would receive the messages from those topics and log them out. To set up this centralized logging system you need to follow these steps both this logging application and your application from where you want to send the logs (Producer logic).
"Producer" applications (e.g. Refocus, Refocus Whitelist, Refocus Real-Time,
etc.) publish their log lines as messages to designated
[Topics](https://kafka.apache.org/intro#intro_topics) in the Kafka Cluster.

The Refocus Logging app receives those messages from those topics and logs them
out.

## Installation

Deploy to Heroku.

Provision your Kafka cluster.

Define your topics.

Optionally, provision a logging add-on.

## Configuration

#### Logging Application
Before launching, this service needs the following environment variables to establish a successful connection:
You must add environment variables in both the Refocus Logging application and
the "producer" applications.

__process.env.TOPICS__: Topics from the Kafka Cluster that you need to subscribe to.
__process.env.KAFKA_CLIENT_CERT__: File path to the file that contains the client certificate (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_CLIENT_CERT_KEY__: File path to the file that contains the client certificate key (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_URL__: The URL to where you setup your Kafka Cluster (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_CONSUMER_MAX_WAIT_TIME_MS__: Maximum amount of time in milliseconds to block waiting if insufficient data is available at the time the fetch request is issued, defaults to 100ms
__process.env.KAFKA_CONSUMER_MAX_BYTES__: Maximum size of messages in a fetch response, defaults to 1MB
__process.env.KAFKA_CONSUMER_IDLE_TIMEOUT__: Timeout between fetch calls, defaults to 1000ms
### Configure the Refocus Logging Application

#### Producer Application

Instead of using the logging library that your app currently uses. Copy paste [this file](https://raw.githubusercontent.com/salesforce/refocus-whitelist/kafka-logging/src/logger.js) and [this file](https://raw.githubusercontent.com/salesforce/refocus-whitelist/kafka-logging/src/kafkaConfig.js) into your project and replace the `require([logging library])` line with `require([path to logger.js])`.
Configure the following environment variables in the Refocus Logging
application:

Similar to the consumer application you need to provide the following env variables to the application.
__process.env.KAFKA_CLIENT_CERT__: File path to the file that contains the client certificate (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_CLIENT_CERT_KEY__: File path to the file that contains the client certificate key (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_URL__: The URL to where you setup your Kafka Cluster (if you are using Heroku go to the reveal ENV variables options and this should already have a value)
__process.env.KAFKA_LOGGING__: If this env variable has any non null value, it will establish the Kafka connection
__process.env.LOCAL_LOGGING__: Set this value to false if you want to turn of local logging. Otherwise the default settings sets the local logging to true.
- `TOPICS`: Comma-delimited list of topic names.
- `KAFKA_CLIENT_CERT`: File path to the file that contains the client
certificate.
(If you are using Heroku, this is set for you automatically when you provision
your Kafka cluster.)
- `KAFKA_CLIENT_CERT_KEY`: File path to the file that contains the client
certificate key.
(If you are using Heroku, this is set for you automatically when you provision
your Kafka cluster.)
- `KAFKA_URL`: Your Kafka Cluster URL.
(If you are using Heroku, this is set for you automatically when you provision
your Kafka cluster.)
- `KAFKA_CONSUMER_MAX_WAIT_TIME_MS`: Maximum amount of time in milliseconds to
block waiting if insufficient data is available at the time the fetch request
is issued, defaults to 100ms.
- `KAFKA_CONSUMER_MAX_BYTES`: Maximum size of messages in a fetch response,
defaults to 1MB.
- `KAFKA_CONSUMER_IDLE_TIMEOUT`: Timeout between fetch calls, defaults to
1000ms.

## Installation
### Configure the Producer Applications

Configure the following environment variables in each of the producer
applications:

Simply deploy to Heroku after adding Sumo Logic and a Kafka Cluster!
- `KAFKA_CLIENT_CERT`: File path to the file that contains the client
certificate.
- `KAFKA_CLIENT_CERT_KEY`: File path to the file that contains the client
certificate key.
- `KAFKA_URL`: Your Kafka Cluster URL.
- `KAFKA_LOGGING`: Set to true to send log lines to the Refocus Logging
application, defaults to false.
- `LOCAL_LOGGING`: Set to false if you want to turn off local logging, defaults
to true.

## Version History

Expand Down

0 comments on commit 38ceca5

Please sign in to comment.