PendingTrace.write - IllegalArgumentException: Illegal Capacity: -1

[wuservices]

I just updated to 0.67.0 on a Play 2.7 / Scala 2.12 app from 0.46.0, but also turned on profiling and tracing without limits. I'm also testing new code on a transaction than ran for over an 60 minutes with hundreds of thousands or more spans. Not sure if any of that matters, but I'm seeing this new error logged.

IllegalArgumentException: Illegal Capacity: -1
    at java.util.ArrayList.<init>(ArrayList.java:157)
    at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:267)
    at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:255)
    at datadog.trace.agent.core.PendingTraceBuffer.enqueue(PendingTraceBuffer.java:27)
    at datadog.trace.agent.core.PendingTrace.decrementRefAndMaybeWrite(PendingTrace.java:235)
...
(31 additional frame(s) were not displayed)

It seems like this error is crashing my code and preventing it from completing. I ran the code a few times, and it kept crashing at this line. Even if I disabled tracing, it was still crashing here, but once I didn't have the Datadog agent on at all, my code completed running.

Is there a bug that could cause size to be -1 (see source)?

[wuservices]

@tylerbenson could this be related to the new code for isPartial, from 7678312#diff-69b42dab7f340c35c23dd3a4b8a22430147efb4865c0be7c263d28e39e4b42d7R334?

[chkelly]

We are also seeing this as well on a long running (~2 hours) migration task. A dynamodb script executing against ~404k failed on 12 of them

java.lang.IllegalArgumentException: Illegal Capacity: -1
	at java.util.ArrayList.<init>(ArrayList.java:157)
	at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:267)
	at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:255)
	at datadog.trace.agent.core.PendingTraceBuffer.enqueue(PendingTraceBuffer.java:27)
	at datadog.trace.agent.core.PendingTrace.decrementRefAndMaybeWrite(PendingTrace.java:235)
	at datadog.trace.agent.core.PendingTrace.addFinishedSpan(PendingTrace.java:181)
	at datadog.trace.agent.core.DDSpan.finishAndAddToTrace(DDSpan.java:83)
	at datadog.trace.agent.core.DDSpan.finish(DDSpan.java:93)
	at datadog.trace.instrumentation.aws.v0.TracingRequestHandler.afterResponse(TracingRequestHandler.java:49)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.afterResponse(AmazonHttpClient.java:1026)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:774)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4230)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4197)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeQuery(AmazonDynamoDBClient.java:2621)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBAsyncClient$23.call(AmazonDynamoDBAsyncClient.java:1376)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBAsyncClient$23.call(AmazonDynamoDBAsyncClient.java:1370)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I opened a support ticket 418375 which contains an agent flare, APM trace, and additional information

[bantonsson]

Hi @wuservices and @chkelly. Thanks for reporting this issue. This issue is related to the book keeping between the counter completedSpanCount that is used for the size of the array sometimes drifting very shortly if there is a span being added while the current spans are being written.

[ddiekroeger]

Hello, we are using version 0.70.0 and still getting this error occasionally. I'm not sure if this was fully fixed. We are using parallelization in the code that's using datadog

    Caused by: java.lang.IllegalArgumentException: Illegal Capacity: -1
    	at java.base/java.util.ArrayList.<init>(ArrayList.java:158)
    	at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:267)
    	at datadog.trace.agent.core.PendingTrace.write(PendingTrace.java:255)
    	at datadog.trace.agent.core.PendingTraceBuffer.enqueue(PendingTraceBuffer.java:27)
    	at datadog.trace.agent.core.PendingTrace.decrementRefAndMaybeWrite(PendingTrace.java:235)
    	at datadog.trace.agent.core.PendingTrace.addFinishedSpan(PendingTrace.java:181)
    	at datadog.trace.agent.core.DDSpan.finishAndAddToTrace(DDSpan.java:83)
    	at datadog.trace.agent.core.DDSpan.finish(DDSpan.java:93)
    	at datadog.trace.instrumentation.opentracing32.OTSpan.finish(OTSpan.java:203)

[bantonsson]

Hi @ddiekroeger , that stack trace does not line up with the source code for v0.70.0, but matches v0.67.0 perfectly. Could you double check the version that you are running, and the stack trace you are seeing?

[ddiekroeger]

Hi @ddiekroeger , that stack trace does not line up with the source code for v0.70.0, but matches v0.67.0 perfectly. Could you double check the version that you are running, and the stack trace you are seeing?

Interesting thank you.. i will double-check and report back