Allow timestamps to be in epoch ms

[felixbarny]

Currently, timestamps in the intake API have to be a ISO date formatted string. Profiling has shown that this is actually the biggest cause of object allocations when serializing the payloads as JSON. Could we implement support for optionally transmitting the timestamp as a (64 bit) integer in epoch milliseconds?

This applies to Transaction.timestamp and Error.timestamp.

[beniwohli]

Wow, that's surprising. I know that parsing iso datetimes can be a bottleneck (hence why we nailed down the format pretty strictly so we can just parse it with a regex), but creating a datetime string as well? In Python, we more or less just call the libc implementation of strftime, which is probably some of the most tuned code you can find.

To be honest, I'm not a huge fan of accepting multiple datetime formats. Also, an integer epoch doesn't carry any timezone information, so we can't be sure that it's UTC.

[watson]

@beniwohli we could just make it a really clear requirement that it has to be UTC, but I agree that we increase the risk of getting it wrong if we don't state the timezone explicitly

[felixbarny]

Also, an integer epoch doesn't carry any timezone information, so we can't be sure that it's UTC.

An epoch timestamp is always in UTC per definition.

Here is a screenshot of an allocation profile of the Java agent where you can see that one of the biggest cause of allocations is parsing the date string during JSON serialization:

[felixbarny]

This change would reduce the allocation rate overhead by about 30%, which brings allocation rates down to under 1kb per request (including the reporting of the transactions).

[beniwohli]

An epoch timestamp is always in UTC per definition.

yup, you're right, I retract that argument. Sorry about that.

Still, my surprise remains. Compared to all the other things we do, transaction and error timestamps have a relatively low cardinality. Does your test suite do any non-trivial work or is it just serializing more or less empty transactions?

I'd like to avoid complicating the API due to a case of premature optimization, is all.

[felixbarny]

Does your test suite do any non-trivial work or is it just serializing more or less empty transactions?

In this benchmark, I'm measuring recording a HTTP request and a SQL query and reporting the transactions to a mock APM-server over HTTP. So there is no non-trivial work that the benchmark does because the point is to only measure the overhead imposed by the agent. Allocation rates are only one metric of this benchmark, but that simple change would reduce the allocation rate overhead by about 30%.

[felixbarny]

Note that by default, the Java agent only captures stack traces when a span took longer than 5ms. Otherwise that would probably be the dominating factor. Also, I implemented ignore_urls in a way that does not allocate memory by not using regex pattern matching, which would otherwise allocate a lot of memory as well. So there is not much left which does allocate memory and one of those things happens to be the construction of the ISO timestamp String.

[beniwohli]

My point is that if the agent doesn't do much more than creating an almost empty transaction, not doing those allocations looks like it has a huge impact. But if you measure a more realistic transaction with a couple dozen spans with contexts and frames and whatnot, the impact might suddenly look much smaller.

All I'm saying is that we should be vary of taking such drastic steps as changing the intake API for something that might just be a micro-optimization.

[felixbarny]

Keep in mind that we do report all transactions, even when not sampling them. In order for this to work at scale, we have to bring the overhead for creating and reporting a non-sampled transaction down to the bare minimum IMHO.

Also, creating transactions and spans involve close to zero allocations, so it does not really matter how many spans there are in a transaction. Note that in this benchmark, the context is fully propagated with http context in the transaction and db context in the span, including the sql statement, so it is somewhat realistically.

[felixbarny]

Also, creating transactions and spans involve close to zero allocations, so it does not really matter how many spans there are in a transaction.

This is not quite accurate as monitoring SQL queries currently does allocate some memory (see elastic/apm-agent-java#35)

[roncohen]

That is indeed surprising.

Googling a bit, it seems DateTimeFormatter is not very efficient. Could we investigate an alternative implementation?

[felixbarny]

There is a newer date API in Java. Might be that the date parser is more efficient in this API. But it is only available in Java 8+ and we support Java 7 applications.

Not sure what you mean with event.toISOString().

Using an epoch timestamp in an API seems like quite a normal thing to do and I'd even consider it more robust that parsing strings. It would also be a backwards compatible change if we support both in the JSON schema. Also, that doesn't seem to add a lot of complexity. So I can't see a lot of downsides to this approach.

[roncohen]

I think it's important to differentiate between whether generating ISO dates are inherently costly operation or if the Java formatter is inefficient. I'd like to avoid making changes to the API to work around performance problems in different languages' standard libraries. If it turns out more agents have the same problem, then it's another story.

By the way, this is about formatting, not parsing, as that happens on the APM Server, right? @felixbarny

[watson]

For reference here's the difference in speed for generating an ISO string vs getting the current UTC time in milliseconds in Node.js/v8:

new Date().toISOString() x 1,391,237 ops/sec ±1.58% (80 runs sampled)
Date.now() x 8,644,451 ops/sec ±1.19% (82 runs sampled)
Fastest is Date.now()

The test was run on my slow laptop. While it looks like ms is around 6 times faster than ISO, we have to remember that I'm still able to generate over a million ISO strings per second, so it's not worth optimizing for my sake.