generated from google/docsy-example
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
bourbonkk
committed
Apr 29, 2022
1 parent
933df6a
commit e71e2db
Showing
11 changed files
with
374 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
|
|
||
| --- | ||
| title: "2.1.0 (2022-04-30)" | ||
| linkTitle: "v2.1.0" | ||
| date: 2022-04-30 | ||
| --- | ||
| 1. gateway, influxdb, opentsdb, tdengine success/failure metric integration completed issue: [#40](https://github.com/Clymene-project/Clymene/issues/40) | ||
| 2. Added promtail-ingester component that can use kafka to promtail's log transfer backend [#47](https://github.com/Clymene-project/Clymene/issues/47) | ||
| 3. promtail - log data transfer to gateway(gRPC, HTTP) [#46](https://github.com/Clymene-project/Clymene/issues/46) | ||
| 4. Development of promtail-gateway component for log data pipeline [#53](https://github.com/Clymene-project/Clymene/issues/53) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| --- | ||
| categories: ["Examples"] | ||
| tags: ["test", "sample", "docs"] | ||
| title: "Promtail Gateway Getting Start" | ||
| linkTitle: "Promtail Gateway" | ||
| date: 2017-01-18 | ||
| weight: 6 | ||
| description: > | ||
| The Promtail Gateway is an optional service that can receive logs data from the agent through gRPC, HTTP communication. | ||
| --- | ||
| The Promtail Gateway is an optional service that can receive logs data from the agent through gRPC, HTTP communication. | ||
|
|
||
| 1. gRPC, HTTP Service | ||
| 2. Logs data insert to Database(ElasticSearch, Loki, ETC) (Optional) | ||
|
|
||
| ## How to setting gateway | ||
| ``` | ||
| --admin.http.host-ports string The host:ports (e.g. 127.0.0.1:15690 or :15690) for the admin server, including health check, /metrics, etc. (default ":15690") | ||
| --gateway.grpc-server.host-port string The host:port (e.g. 127.0.0.1:15610 or :15610) of the gateway's GRPC server (default ":15610") | ||
| --gateway.grpc.tls.cert string Path to a TLS Certificate file, used to identify this server to clients | ||
| --gateway.grpc.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted) | ||
| --gateway.grpc.tls.enabled Enable TLS on the server | ||
| --gateway.grpc.tls.key string Path to a TLS Private Key file, used to identify this server to clients | ||
| --gateway.http-server.host-port string The host:port (e.g. 127.0.0.1:15610 or :15611) of the gateway's HTTP server (default ":15611") | ||
| --gateway.http.tls.cert string Path to a TLS Certificate file, used to identify this server to clients | ||
| --gateway.http.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted) | ||
| --gateway.http.tls.enabled Enable TLS on the server | ||
| --gateway.http.tls.key string Path to a TLS Private Key file, used to identify this server to clients | ||
| --log-level string Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info") | ||
| --metrics-backend string Defines which metrics backend to use for metrics reporting: expvar, prometheus, none (default "prometheus") | ||
| --metrics-http-route string Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics") | ||
| ``` | ||
|
|
||
| ## How to set up the Storage Type | ||
| #### 1. Setting environmental variables | ||
|
|
||
|
|
||
| ElasticSearch | ||
| ``` | ||
| STORAGE_TYPE=elasticsearch | ||
| ``` | ||
| Loki | ||
| ``` | ||
| STORAGE_TYPE=loki | ||
| ``` | ||
|
|
||
| Promtail-gateway | ||
| ``` | ||
| STORAGE_TYPE=gateway | ||
| ``` | ||
|
|
||
| Kafka | ||
| ``` | ||
| STORAGE_TYPE=kafka | ||
| ``` | ||
|
|
||
| #### 2. Option description by storage type | ||
|
|
||
| - [ElasticSearch option](http://clymene-project.github.io/docs/database-options/elasticsearch) | ||
| - [Loki option](http://clymene-project.github.io/docs/database-options/loki) | ||
| - [Kafka option](http://clymene-project.github.io/docs/database-options/kafka) | ||
| - [Promtail-gateway](http://clymene-project.github.io/docs/database-options/gateway) | ||
|
|
||
|
|
||
| ### Docker-compose Example | ||
| ```yaml | ||
| version: '2' | ||
| services: | ||
| clymene-ingester: | ||
| image: bourbonkk/promtail-gateway:latest | ||
| ports: | ||
| - "15610:15610" | ||
| environment: | ||
| - STORAGE_TYPE=elasticsearch | ||
| command: | ||
| - --log-level=debug | ||
| - --es.server-urls=http://[ELASTICSEARCH-IP]:9200 | ||
| ``` | ||
|
|
||
| ### k8s Example | ||
| ```yaml | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: promtail-gateway | ||
| namespace: clymene | ||
| labels: | ||
| app: promtail-gateway | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: promtail-gateway | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: promtail-gateway | ||
| spec: | ||
| containers: | ||
| - name: promtail-gateway | ||
| image: bourbonkk/promtail-gateway:latest | ||
| imagePullPolicy: Always | ||
| ports: | ||
| - containerPort: 15610 | ||
| args: | ||
| - --es.server-urls=http://[ELASTICSEARCH-IP]:9200 | ||
| - --log-level=info | ||
| env: | ||
| - name: STORAGE_TYPE | ||
| value: elasticsearch | ||
| securityContext: | ||
| runAsUser: 1000 | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| --- | ||
| categories: ["Examples"] | ||
| tags: ["test", "sample", "docs"] | ||
| title: "Promtail ingester Getting Start" | ||
| linkTitle: "Promtail ingester" | ||
| date: 2017-01-18 | ||
| weight: 5 | ||
| description: > | ||
| Promtail ingester is an optional service responsible for insert logs data loaded on kafka into the database | ||
| --- | ||
| Promtail ingester is an optional service responsible for insert logs data loaded on kafka into the database | ||
|
|
||
| 1. Kafka message consume | ||
| 2. logs data insert to Database(ElasticSearch, Loki, ETC) (Optional) | ||
|
|
||
| ## How to setting kafka consumer | ||
|
|
||
| ``` | ||
| --admin.http.host-ports string The host:ports (e.g. 127.0.0.1:15694 or :15694) for the admin server, including health check, /metrics, etc. (default ":15694") | ||
| --kafka.consumer.authentication string Authentication type used to authenticate with kafka cluster. e.g. none, kerberos, tls, plaintext (default "none") | ||
| --kafka.consumer.brokers string The comma-separated list of kafka brokers. i.e. '127.0.0.1:9092,0.0.0:1234' (default "127.0.0.1:9092") | ||
| --kafka.consumer.client-id string The Consumer Client ID that clymene-ingester will use (default "clymene") | ||
| --kafka.consumer.encoding string The encoding of metrics ("json", "protobuf") consumed from kafka (default "protobuf") | ||
| --kafka.consumer.group-id string The Consumer Group that clymene-ingester will be consuming on behalf of (default "clymene") | ||
| --kafka.consumer.kerberos.config-file string Path to Kerberos configuration. i.e /etc/krb5.conf (default "/etc/krb5.conf") | ||
| --kafka.consumer.kerberos.keytab-file string Path to keytab file. i.e /etc/security/kafka.keytab (default "/etc/security/kafka.keytab") | ||
| --kafka.consumer.kerberos.password string The Kerberos password used for authenticate with KDC | ||
| --kafka.consumer.kerberos.realm string Kerberos realm | ||
| --kafka.consumer.kerberos.service-name string Kerberos service name (default "kafka") | ||
| --kafka.consumer.kerberos.use-keytab Use of keytab instead of password, if this is true, keytab file will be used instead of password | ||
| --kafka.consumer.kerberos.username string The Kerberos username used for authenticate with KDC | ||
| --kafka.consumer.plaintext.mechanism string The plaintext Mechanism for SASL/PLAIN authentication, e.g. 'SCRAM-SHA-256' or 'SCRAM-SHA-512' or 'PLAIN' (default "PLAIN") | ||
| --kafka.consumer.plaintext.password string The plaintext Password for SASL/PLAIN authentication | ||
| --kafka.consumer.plaintext.username string The plaintext Username for SASL/PLAIN authentication | ||
| --kafka.consumer.promtail.topic string The name of the promtail kafka topic to consume from (default "clymene-logs") | ||
| --kafka.consumer.protocol-version string Kafka protocol version - must be supported by kafka server | ||
| --kafka.consumer.tls.ca string Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore) | ||
| --kafka.consumer.tls.cert string Path to a TLS Certificate file, used to identify this process to the remote server(s) | ||
| --kafka.consumer.tls.enabled Enable TLS when talking to the remote server(s) | ||
| --kafka.consumer.tls.key string Path to a TLS Private Key file, used to identify this process to the remote server(s) | ||
| --kafka.consumer.tls.server-name string Override the TLS server name we expect in the certificate of the remote server(s) | ||
| --kafka.consumer.tls.skip-host-verify (insecure) Skip server's certificate chain and host name verification | ||
| --log-level string Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info") | ||
| --metrics-backend string Defines which metrics backend to use for metrics reporting: expvar, prometheus, none (default "prometheus") | ||
| --metrics-http-route string Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics") | ||
| --promtail-ingester.deadlockInterval duration Interval to check for deadlocks. If no messages gets processed in given time, clymene-ingester app will exit. Value of 0 disables deadlock check. (default 0s) | ||
| --promtail-ingester.parallelism string The number of messages to process in parallel (default "1000") | ||
| ``` | ||
|
|
||
| ## How to set up the Storage Type | ||
|
|
||
| #### 1. Setting environmental variables | ||
|
|
||
| ElasticSearch | ||
| ``` | ||
| STORAGE_TYPE=elasticsearch | ||
| ``` | ||
| Loki | ||
| ``` | ||
| STORAGE_TYPE=loki | ||
| ``` | ||
|
|
||
| Promtail-gateway | ||
| ``` | ||
| STORAGE_TYPE=gateway | ||
| ``` | ||
|
|
||
| Kafka | ||
| ``` | ||
| STORAGE_TYPE=kafka | ||
| ``` | ||
|
|
||
| #### 2. Option description by storage type | ||
|
|
||
| - [ElasticSearch option](../clymene-promtail/elasticsearch/es-option.md) | ||
| - [Loki option](../clymene-promtail/loki/loki-option.md) | ||
| - [Kafka option](../clymene-promtail/kafka/kafka-option.md) | ||
| - [Promtail-gateway](../clymene-promtail/gateway/gateway-option.md) | ||
|
|
||
|
|
||
| ### Docker-compose Example | ||
|
|
||
| ```yaml | ||
| version: '2' | ||
| services: | ||
| promtail-ingester: | ||
| image: bourbonkk/promtail-ingester:latest | ||
| ports: | ||
| - "15694:15694" | ||
| environment: | ||
| # - STORAGE_TYPE=elasticsearch,prometheus # use composite writer | ||
| - STORAGE_TYPE=elasticsearch | ||
| command: | ||
| - --log-level=debug | ||
| - --kafka.consumer.brokers=[KAFKA-IP]:9092 | ||
| - --es.server-urls=http://[ELASTICSEARCH-IP]:9200 | ||
| # - --prometheus.remote.url=http://prometheus:9090/api/v1/write | ||
| ``` | ||
|
|
||
| ### k8s Example | ||
|
|
||
| ```yaml | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: promtail-ingester | ||
| namespace: clymene | ||
| labels: | ||
| app: promtail-ingester | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: promtail-ingester | ||
| replicas: 1 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: promtail-ingester | ||
| spec: | ||
| containers: | ||
| - name: promtail-ingester | ||
| image: bourbonkk/promtail-ingester:latest | ||
| imagePullPolicy: Always | ||
| ports: | ||
| - containerPort: 15694 | ||
| args: | ||
| - --es.server-urls=http://[ELASTICSEARCH-IP]:9200 | ||
| - --log-level=info | ||
| - --kafka.consumer.brokers=clymene-kafka-broker:9092 | ||
| env: | ||
| - name: STORAGE_TYPE | ||
| value: elasticsearch | ||
| securityContext: | ||
| runAsUser: 1000 | ||
| ``` |
Oops, something went wrong.