Telemetry correction #55
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
April 23, 2023 18:02
Signed-off-by: Raziel Cohen <rcohen@armosec.io>
…r-time-scope Signed-off-by: Raziel Cohen <rcohen@armosec.io>
|
Summary:
|
matthyx
requested changes
Apr 24, 2023
| } | ||
| } | ||
| } | ||
| logger.L().Ctx(ctx).Info("stop monitor on container - after monitoring time", helpers.String("container ID", contEvent.GetContainerID()), helpers.String("container name", contEvent.GetContainerName()), helpers.String("k8s resources", contEvent.GetK8SWorkloadID()), helpers.Error(err)) |
There was a problem hiding this comment.
we don't attach logs above Warning to spans, please remove .Ctx(ctx) from the line
| @@ -160,10 +176,13 @@ func (ch *ContainerHandler) startRelevancyProcess(contEvent v1.ContainerEventDat | |||
| err = ch.startTimer(watchedContainer, contEvent.GetContainerID()) | |||
| if err != nil { | |||
| if errors.Is(err, droppedEventsError) { | |||
| logger.L().Ctx(context.GetBackgroundContext()).Warning("container monitoring got drop events - we may miss some realtime data", helpers.String("container ID", contEvent.GetContainerID()), helpers.String("container name", contEvent.GetContainerName()), helpers.String("k8s resources", contEvent.GetK8SWorkloadID()), helpers.Error(err)) | |||
| ctx, span := otel.Tracer("").Start(ctx, "dropped events.", trace.WithAttributes(attribute.String("containerID", contEvent.GetContainerID()), attribute.String("container workload", contEvent.GetK8SWorkloadID()))) | |||
There was a problem hiding this comment.
don't create a span just for the error, they should be created when you enter a part of your code that you want to trace all the time (aka. record time spent there and compare with other spans)
added 3 commits
April 24, 2023 18:21
Signed-off-by: Raziel Cohen <rcohen@armosec.io>
Signed-off-by: Raziel Cohen <rcohen@armosec.io>
|
Summary:
|
|
@rcohencyberarmor I corrected some things myself, PTAL |
|
Summary:
|
| @@ -26,7 +26,7 @@ func (aggregator *Aggregator) collectDataFromContainerAccumulator(errChan chan e | |||
| for { | |||
| newEvent := <-aggregator.aggregationDataChan | |||
| if newEvent.GetEventCMD() == accumulator.DropEventOccurred { | |||
| // aggregator.StopAggregate() | |||
There was a problem hiding this comment.
I removed it by purpose, the management decided to continue listen on container when we got dropped events.
main.go
Outdated
| @@ -14,6 +14,9 @@ import ( | |||
| ) | |||
|
|
|||
| func main() { | |||
| context.SetBackgroundContext() | |||
There was a problem hiding this comment.
it must be after the config parser, so we can get account ID etc..
There was a problem hiding this comment.
ok then don't try to attach warnings to the ctx without span
Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
matthyx
force-pushed
the
telemetry-correction
branch
from
fff3e93
to
68b73f2
Compare
|
Summary:
|
matthyx
approved these changes
Apr 28, 2023
| @@ -26,7 +26,7 @@ func (aggregator *Aggregator) collectDataFromContainerAccumulator(errChan chan e | |||
| for { | |||
| newEvent := <-aggregator.aggregationDataChan | |||
| if newEvent.GetEventCMD() == accumulator.DropEventOccurred { | |||
| aggregator.StopAggregate() | |||
| // aggregator.StopAggregate() | |||
There was a problem hiding this comment.
why did you stop accumulating here?
| @@ -117,11 +123,9 @@ func (ch *ContainerHandler) startTimer(watchedContainer watchedContainerData, co | |||
| } | |||
| case err = <-watchedContainer.syncChannel[StepEventAggregator]: | |||
| if err.Error() == accumulator.DropEventOccurred { | |||
| watchedContainer.snifferTicker.Stop() | |||
There was a problem hiding this comment.
was it intended to be removed?
main.go
Outdated
| @@ -14,6 +14,9 @@ import ( | |||
| ) | |||
|
|
|||
| func main() { | |||
| context.SetBackgroundContext() | |||
There was a problem hiding this comment.
ok then don't try to attach warnings to the ctx without span
vladklokun
approved these changes
Apr 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview