Trace limitations

[honestegg]

Is there a limit to the number of spans (either total or nested) that are shown in the trace view? I've been testing out the Trace feature and I'm not seeing them all in the view -- even though they seem to exist in the logs. When I do a count on the items with the same TraceId there are a total of 65,772 items. In a possibly related issue, I'm seeing a "Missing (2d92...)" at the top of the trace view even though that span clearly exists.

I have attached a screenshot showing that "2d92..." exists:

[nblumhardt]

Hi Brian, thanks for your message. The trace view will show up to 1000 spans or so, so I don't think you're hitting that kind of limitation.

Assuming there's no client side crash going on (you'd see this in the browser's Developer Tools console) my best guess is that there's a data problem, possibly around the trace id.

Would it be possible to post (or email to support@) a redacted version of the raw JSON for the first two spans shown in the log search in the screenshot? (The 55s and 53s ones.) You can get this by clicking into event details for each, and choosing it from the Export menu. Thanks!

[honestegg]

In this example, there are 65,772 spans. I guess I said "items" above, but they are spans. That seems to exceed the 1000 or so you mentioned.

[nblumhardt]

Ah I see 👍

Was it intentional to create a trace with that many spans? Would be interesting to know a bit about the scenario and anything Seq could do to help.

We did take massive traces like this into account in the design of Seq's tracing system, but decided at this level only aggregate information would really be useful, so we've left this to the event view/query language style of interaction, rather than the hierarchical viewer.

[honestegg]

So, do you think that the number of spans is causing the "Missing..."? I suppose if it only loads 1000 or so spans, it would never get to the parent span.

As far as whether I "intended" to create that many spans, I suppose so. I have the auto instrumentation enabled for MongoDB and there are a lot of saves happening. 50,770 of the spans are Mongo operations.

Ultimately, I'm investigating a scenario in production and testing out how we might instrument the code to get useful traces to help narrow down where all the time is being spent. The trace is what I would think of as a single operation -- it's an import of 100 records. There is just a lot of processing that happens during the import. I'm certainly new to setting up instrumentation and am not sure at this point what kind of aggregation or sampling is built into the tools to help make sense of the data.

As far as how Seq could help, it might be nice to filter out spans that don't take much time or limit the number of sibling spans that are shown. Maybe aggregating sibling spans by their name. Maybe this is also not a normal way to use tracing.

[nblumhardt]

Yes, the "missing" message is due to this. We've raised a ticket internally to show a more informative message in this scenario.

I don't think there's a "wrong" way to use tracing, but to me, tracing excels as a high-level map of an interaction, while the kind of detail captured in this trace seems more like what a profile would normally capture.

Without digging into the data myself I could be off-track, but the way I'd tackle this is to turn off/suppress those MongoDB spans, and any others that are this noisy, and see whether the trace reveals any patterns you can drill down into at that level first.

If it turns out you need more detail, logging the timings for your MongoDB operations, and then using classic max/order by etc. to explore them, should show up any outliers.

[DomZZ]

Hello @nblumhardt ,

We have bulk operations that do huge data processing, that generate a lot of spans, and we always have the limit of 1000 spans... that prevent us to diagnostics the issues.

Can we set this limit higher usibg cli tool or Seq.json file ?

If not, can you give us a way to manage this setting.

Thanks

[nblumhardt]

Hi @DomZZ,

We're keen to improve this but wary that the client-side trace viewer (in the JavaScript UI) might not handle massive traces very well - are you looking at a few thousand spans, or is it more like 10s or 100s of thousands?

If it's a few thousand at most I'd guess a setting will improve it (as the expense of load times/page perf) but I think we might need to do more summarization server-side to handle larger ones (also open to doing this but would need more thought/planning).

[DomZZ]

Hi,
actually we switched all our file logs to opentelemetry traces/logs, this is mostly great for our support team but if they fall on the message "1000 spans limit" they are lost and raging about this ...

All you can do to overcome this as soon as possible, will be appreciated

Thanks

PS : If javascript app is the problem, a client side application could be an option also ...

[nblumhardt]

Hi again! Just in case we're able to cook up a short-term solution to keep you going - can you please send some details on the range of large traces (typical and maximum) you need to handle? Starting a thread with support@ might be the best way to coordinate. Cheers!