Skip to content

Commit

Permalink
[klogs] Use maps instead of columns for custom fields
Browse files Browse the repository at this point in the history 
This is a follow up for kobsio/klogs#34 were we
changed the database schema for klogs to use maps instead of arrays for
custom fields.

To be able to query the data ingested by the klogs plugin we had to
adjust the queries to also use maps instead of columns for these fields.
  • Loading branch information
@ricoberger
ricoberger committed Oct 12, 2022
1 parent 26b44dc commit a128579
Show file tree
Hide file tree
Showing 16 changed files with 656 additions and 878 deletions.
12 changes: 8 additions & 4 deletions docs/plugins/klogs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@ To use the klogs plugin the following configuration is needed in the satellites
| options.database | string | The name of the database. | Yes |
| options.username | string | Username to access a ClickHouse instance. | No |
| options.password | string | Password to access a ClickHouse instance. | No |
| options.readTimeout | string | The read timeout for operations. | No |
| options.writeTimeout | string | The write timeout for operations. | No |
| options.dialTimeout | string | ClickHouse dial timeout. The default value is `10s`. | No |
| options.connMaxLifetime | string | ClickHouse maximum connection lifetime. The default value is `1h`. | No |
| options.maxIdleConns | number | ClickHouse maximum number of idle connections. The default value is `5`. | No |
| options.maxOpenConns | number | ClickHouse maximum number of open connections. The default value is `10`. | No |
| options.materializedColumns | []string | A list of materialized columns. See [kobsio/klogs](https://github.com/kobsio/klogs#configuration) for more information. | No |

```yaml
Expand All @@ -27,10 +29,12 @@ plugins:
options:
address:
database:
writeTimeout:
readTimeout:
username:
password:
dialTimeout:
connMaxLifetime:
maxIdleConns:
maxOpenConns:
materializedColumns:
```

Expand Down
32 changes: 16 additions & 16 deletions docs/plugins/sql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) as avg_duration,
avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_ust,
avg(fields_number['content_duration']) as avg_duration,
avg(fields_number['content_upstream_service_time']) as avg_ust,
avg_duration - avg_ust as avg_diff
FROM
logs.logs
Expand All @@ -122,7 +122,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand Down Expand Up @@ -153,7 +153,7 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) - avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_diff
avg(fields_number['content_duration']) - avg(fields_number['content_upstream_service_time']) as avg_diff
FROM
logs.logs
WHERE
Expand All @@ -162,7 +162,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand All @@ -188,8 +188,8 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) as avg_duration,
avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_ust
avg(fields_number['content_duration']) as avg_duration,
avg(fields_number['content_upstream_service_time']) as avg_ust
FROM
logs.logs
WHERE
Expand All @@ -198,7 +198,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand Down Expand Up @@ -230,8 +230,8 @@ The following options can be used for a panel with the SQL plugin:
- name: Log Levels
query: |
SELECT
content.level,
count(content.level) as count_data
content_level,
count(content_level) as count_data
FROM
logs.logs
WHERE
Expand All @@ -241,9 +241,9 @@ The following options can be used for a panel with the SQL plugin:
AND app='myservice'
AND container_name='myservice'
GROUP BY
content.level
content_level
columns:
content.level:
content_level:
title: Level
count_data:
title: Count
Expand All @@ -258,8 +258,8 @@ The following options can be used for a panel with the SQL plugin:
type: pie
query: |
SELECT
content.level,
count(content.level) as count_data
content_level,
count(content_level) as count_data
FROM
logs.logs
WHERE
Expand All @@ -269,8 +269,8 @@ The following options can be used for a panel with the SQL plugin:
AND app='myservice'
AND container_name='myservice'
GROUP BY
content.level
pieLabelColumn: content.level
content_level
pieLabelColumn: content_level
pieValueColumn: count_data
```

Expand Down
18 changes: 12 additions & 6 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/Azure/go-autorest/autorest v0.11.28
github.com/Azure/go-autorest/autorest/azure/auth v0.5.11
github.com/Azure/go-autorest/autorest/date v0.3.0
github.com/ClickHouse/clickhouse-go v1.5.4
github.com/ClickHouse/clickhouse-go/v2 v2.3.0
github.com/DataDog/datadog-api-client-go/v2 v2.3.1
github.com/andygrunwald/go-jira v1.16.0
github.com/coreos/go-oidc/v3 v3.3.0
Expand Down Expand Up @@ -72,16 +72,17 @@ require (
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 // indirect
github.com/ClickHouse/ch-go v0.47.3 // indirect
github.com/DataDog/zstd v1.5.0 // indirect
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/PuerkitoBio/goquery v1.5.1 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/andybalholm/cascadia v1.1.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/docker/distribution v2.8.0+incompatible // indirect
Expand All @@ -94,6 +95,8 @@ require (
github.com/fatih/structs v1.1.0 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/fluxcd/pkg/apis/kustomize v0.5.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.6.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
Expand All @@ -111,7 +114,7 @@ require (
github.com/gorilla/mux v1.8.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.1 // indirect
github.com/hashicorp/go-retryablehttp v0.5.1 // indirect
github.com/hashicorp/go-version v1.4.0 // indirect
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
Expand All @@ -135,13 +138,16 @@ require (
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/openshift/api v0.0.0-20200221181648-8ce0047d664f // indirect
github.com/openzipkin/zipkin-go v0.4.0 // indirect
github.com/paulmach/orb v0.7.1 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
github.com/rs/xid v1.4.0 // indirect
github.com/rs/zerolog v1.20.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/shopspring/decimal v1.3.1 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
Expand All @@ -153,11 +159,11 @@ require (
github.com/xdg-go/stringprep v1.0.3 // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
go.opentelemetry.io/otel/metric v0.31.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.7.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b // indirect
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab // indirect
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect
Expand Down
Loading

0 comments on commit a128579

Please sign in to comment.