Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The timer in MutableSpanMap have not proper handling span data. #247

Open
a14907 opened this issue Dec 13, 2021 · 0 comments
Open

The timer in MutableSpanMap have not proper handling span data. #247

a14907 opened this issue Dec 13, 2021 · 0 comments

Comments

@a14907
Copy link

a14907 commented Dec 13, 2021

I have catch this exception in my product env:

   Unhandled exception. System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at zipkin4net.Tracers.Zipkin.WriterExtensions.WriteList[U](StreamWriter writer, SerializeMethod`1 serializer, String fieldName, ICollection`1 elements, IPEndPoint endPoint, String serviceName)
   at zipkin4net.Tracers.Zipkin.JSONSpanSerializer.SerializeSpan(StreamWriter writer, Span span)
   at zipkin4net.Tracers.Zipkin.JSONSpanSerializer.SerializeTo(Stream stream, Span span)
   at OpenPlatform.AspNetCore.Infrastructure.CusJSONSpanSerializer.SerializeTo(Stream stream, Span span) in /src/PlaformServices/Libraries/OpenPlatform.AspNetCore/Infrastructure/ZipkinStartup.cs:line 88
   at zipkin4net.Tracers.Zipkin.ZipkinTracerReporter.Report(Span span)
   at zipkin4net.Internal.Recorder.MutableSpanMap.FlushOldSpans(DateTime utcNow)
   at zipkin4net.Internal.Recorder.MutableSpanMap.<.ctor>b__6_0(Object _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()

I review the source code . and i know this is the problem of the System.Threading.Timer. This timer won't wait the pre task to finish ,and will start next task. That is the problem.

To verification this, I output the span when exception raise. This is show below:

    "spanState": {
        "traceIdHigh": 0,
        "traceId": -7806682531417472614,
        "parentSpanId": 6104606790270792834,
        "spanId": -3199394417785887997,
        "samplingStatus": 1,
        "flags": 0,
        "sampled": true,
        "debug": false,
        "extra": []
    },
    "annotations": [
        {
            "value": "sr",
            "timestamp": "2021-12-13T09:44:18.4564427Z"
        },
        {
            "value": "flush.timeout",
            "timestamp": "2021-12-13T09:49:14.2332799Z"
        },
        {
            "value": "flush.timeout",
            "timestamp": "2021-12-13T09:49:14.2332922Z"
        }
    ],
.....

There are two "flush.timeout" annotations, their timestamp are very close.

Maybe you should consider when to trigger the timer task.Please fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant