Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(observability): OpenTelemetry logs #13291

Merged
merged 5 commits into from
Jul 5, 2024
Merged

feat(observability): OpenTelemetry logs #13291

merged 5 commits into from
Jul 5, 2024

Conversation

samugi
Copy link
Member

@samugi samugi commented Jun 24, 2024 •
edited
Loading

Note for reviewers: commits are split and commented appropriately, I would recommend checking them one by one.

Summary

This PR adds OpenTelemetry formatted logs as a new
signal supported by the OpenTelemetry plugin.

It patches the ngx.log function (and adapts kong PDK log) to collect the logged lines and
enrich them with additional context info:

  • severity
  • introspection details
  • timestamp

And for request-scoped logs also:

  • Request ID
  • Trace ID
  • Span ID
  • Trace flags

The log entries are protobuf-serialized and sent to the endpoint
configured in the OpenTelemetry plugin's config.logs_endpoint field.

Checklist

Issue reference

KAG-4712

@samugi samugi changed the title Feat/otel formatted logs feat(observability): OpenTelemetry-formatted logs Jun 24, 2024
@samugi samugi changed the title feat(observability): OpenTelemetry-formatted logs feat(observability): OpenTelemetry logs Jun 24, 2024
@samugi samugi force-pushed the feat/otel-formatted-logs branch 2 times, most recently from 724a84a to a775404 Compare June 24, 2024 09:05
@samugi samugi added this to the 3.8.0 milestone Jun 24, 2024
@samugi samugi force-pushed the feat/otel-formatted-logs branch 3 times, most recently from 5e5257d to 3621b8f Compare June 24, 2024 17:22
ADD-SP
ADD-SP requested changes Jun 25, 2024
View reviewed changes
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/handler.lua Outdated Show resolved Hide resolved
ADD-SP
ADD-SP reviewed Jun 25, 2024
View reviewed changes
Copy link
Contributor

@ADD-SP ADD-SP left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.

git commit --fixup <commit_id>

@samugi
Copy link
Member Author

samugi commented Jun 25, 2024 •
edited
Loading

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.

git commit --fixup <commit_id>

yes absolutely. I wasn't planning to rebase (for now), which will make syncing (via cherry-pick) with the existing EE PR easier as well.

@samugi samugi requested a review from ADD-SP July 3, 2024 09:57
@samugi samugi mentioned this pull request Jul 3, 2024
Merged
3 tasks
ADD-SP
ADD-SP reviewed Jul 4, 2024
View reviewed changes
kong/globalpatches.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/otlp.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/otlp.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/traces.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/traces.lua Show resolved Hide resolved
kong/plugins/opentelemetry/schema.lua Show resolved Hide resolved
ADD-SP
ADD-SP reviewed Jul 4, 2024
View reviewed changes
kong/plugins/opentelemetry/utils.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/utils.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
@samugi samugi requested a review from ADD-SP July 4, 2024 08:51
ADD-SP
ADD-SP reviewed Jul 4, 2024
View reviewed changes
kong/observability/logs.lua Outdated Show resolved Hide resolved
spec/01-unit/26-observability/05-logs_spec.lua Show resolved Hide resolved
kong/observability/logs.lua Show resolved Hide resolved
@samugi samugi requested a review from ADD-SP July 4, 2024 10:44
@samugi samugi force-pushed the feat/otel-formatted-logs branch 2 times, most recently from 2cd7338 to 6f94ca4 Compare July 4, 2024 15:20
@samugi
fix(instrumentation): active span
b5142eb 
Spans were not set as "active" during their lifetime.
This commit addresses the issue by setting each span as the current
"active" span in instrumentations. This is necessary in order to have
the correct span_id information in the OTel-formatted log entries.
ADD-SP
ADD-SP approved these changes Jul 5, 2024
View reviewed changes
Copy link
Contributor

@ADD-SP ADD-SP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please rebase commits before the merge.

@samugi
feat(dynamic_hook): support non request-scoped hooks + disable
7a2ba89 
When a disabled and non-request scoped hook was called outside of
a request scope, it was producing an error while trying to index ngx.ctx

This commit adds a check to early return from is_group_enabled when
outside of the request scope

This allows for hooks to still work when used in other contexts where
they can be enabled via the always_enable() function

This commit also adds support for disabling hooks.
@samugi
feat(observability): add OpenTelemetry logs
ddbd1fe 
This commit adds OpenTelemetry formatted logs as a new
signal supported by the OpenTelemetry plugin.

It patches the ngx.log function to collect the logged line and
enrich it with additional context info:

* severity
* introspection details
* timestamp

And for request-scoped logs also:

* Request ID
* Trace ID
* Span ID
* Trace flags

The log entries are protobuf-serialized and sent to the endpoint
configured in the OpenTelemetry plugin's config.logs_endpoint field.

feat(opentelemetry): add traceflags to otel logs

This commit collects an additional tracing information (trace flags) in
the internal context and adds it to produced OTel-formatted log entries.
@samugi
refactor(observatility): move tracing and logs in observability
0707ea7 
This commit moves the tracing and logging core code in the observability
folder.
@samugi
tests(opentelemetry): OTel formatted logs
c1f99a9 
This commit adds unit and integration tests for the OTel-formatted logs
feature.
@samugi samugi merged commit 8aea938 into master Jul 5, 2024
29 checks passed
@samugi samugi deleted the feat/otel-formatted-logs branch July 5, 2024 09:55
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13291-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13291-to-master-to-upstream
git checkout -b cherry-pick-13291-to-master-to-upstream
ancref=$(git merge-base 8f9b82d870985cdd5fb417d589ad2f7ddec967e4 c1f99a9ff2a5986dbd72e85d9897167df2124b77)
git cherry-pick -x $ancref..c1f99a9ff2a5986dbd72e85d9897167df2124b77

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 5, 2024
@samugi samugi removed cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention labels Jul 5, 2024
samugi added a commit to Kong/docs.konghq.com that referenced this pull request Jul 5, 2024
@samugi @hanshuebner
feat(OpenTelemetry): support opentelemetry logs
8dac69c 
docs for Kong/kong#13291

Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: Hans Hübner <hans.huebner@gmail.com>
lena-larionova added a commit to Kong/docs.konghq.com that referenced this pull request Jul 23, 2024
@samugi @hanshuebner @lena-larionova
feat(observability): OpenTelemetry logs (#7583)
e512f09 
* feat(OpenTelemetry): support opentelemetry logs

docs for Kong/kong#13291

Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: Hans Hübner <hans.huebner@gmail.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>

---------

Co-authored-by: Hans Hübner <hans.huebner@gmail.com>
Co-authored-by: lena-larionova <54370747+lena-larionova@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanshuebner hanshuebner hanshuebner approved these changes

@ADD-SP ADD-SP ADD-SP approved these changes

Assignees

@samugi samugi

Labels
chore Not part of the core functionality of kong, but still needed core/clustering core/configuration core/db core/docs core/logs core/pdk core/proxy core/tracing plugins/aws-lambda plugins/opentelemetry plugins/zipkin schema-change-noteworthy size/XXL
Projects
None yet
Milestone
3.8.0
Development

Successfully merging this pull request may close these issues.
5 participants
@samugi @ADD-SP @Achiel @team-gateway-bot @hanshuebner