Stackdriver logs are not only for Google Cloud Platform. You can drop your logs into Stackdriver from anywhere and this docker-compose will help you achieve that.
Main idea of Centralized Logging is collecting all logs' data across a system into one storage. By parsing, filtering and searching logs you will be able to monitor health of the system from one place. By having all the logs in one place, you gain comprehensive source of truth about your system. Based on it you can search for patterns, dig for information and prepare system reports which can be later utilized in process of decision making. If you are not convinced, please check this reddit thread for some inspiration.
Centralized Logging can be provisioned On Premises, mostly by utilizing ELK stack or emerging Kubernates Logging. Maintaining own logging stack come with performance and costs issues. The most popular mitigation of those issues is done by reducing logs' data retention.
On the other hand Centralized Logging as a Service like Sentry or Papertrail can also be costly. How much? It depends how much data you will push in and what data retention should be granted.
Knowing above, Stackdriver Logging comes into the game. It's pricing plan guarantee first 50 GiB free allotment per month per project and 30 days retention period (last checked at August 12th, 2019).
50 GiB per month? Is it much? Again it depends, nowadays we are working in a project which generates at least 5 GiB logs per day. But let's face it, your project really need to explode to reach this amounts.
Pricing in Stackdriver starts when you would like to automate monitoring based on collected logs. Just logs almost comes for free.
Disclaimer: This package is for educational purpose, it is not production ready solution. To run it locally you will need to install:
- Docker 18.06.0+
- Clone this repo
- Register account in GCP and create a new project
- Generate Service Account Key and place it into
./config/service-key.json - In docker-compose.yml file set environment variables
STACKDRIVER_VM_IDandSTACKDRIVER_ZONE_IDaccordingly to your needs - Start Docker Compose
$ docker-compose up - Enter http://0.0.0.0:8080 to open default Nginx page, logs from access_log should be sent to Stackdriver in a background
- Open your project in GCP and navigate to Logging tab
- From
resourcesdropdown chooseGCE VM Instance - Your Nginx's logs should appear
It is always cool to read about parts which were used to build a solution, this is a reference:
- Docker
- Fluentd
- Stackdriver
- Nginx (just for an example purpose)
- Nginx Docker is configured to push logs to STDOUT/STDERR by default
- Docker Compose is configured to use Fluentd Logging Driver
- Fluentd Docker grab all the logs which are exposed by Logging Driver
- And forward them via Fluentd's Google Cloud plugin
- In the plugin, Stackdriver Logging API is used to push logs to Stackdriver
Please be informed that official Stackdriver Logging Agent support only GCP or AWS instances. Also Fluentd's Google Cloud plugin doesn't inform that pushing logs from outside is possible.
But it has configuration which makes it possible and also Stackdriver Logging API is open and there is a lot of movement in Knative which states that Kubernetes Clusters can push logs to Stackdriver no matter on what Cloud Provider it is provisioned.