New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ActiveRecord SQL spans don't have the same service name as Instantiation spans #3359
Comments
Hello @dannyrife I took a look at your issue but there is two comments in specs that question myself. I am wondering if it was intended.
and
But the comment could also be outdated when I see the f64a90c#diff-480dabeac4cb83aeebfd219880fbbaafff3c0b3a83cef7b2fa0e3819e3b516a5R60-R61 Here is a draft proposal: #3383 |
👋 @dannyrife, @benoittgt , I believe it is behaving as expected, since |
@TonyCTHsu |
Hi @dannyrife, I have been investigating this issue and the related #3184 and #3383. To give a little more detail on @TonyCTHsu’s comment, the service name is intended to describe the application that runs the code in question, not a particular component of the application. The component should be described by the span tags of “component” and “operation”, which are “active_record” and “instantiation”, respectively, for the AR instantiation instrumentation. Specifying another service name for instantiation operation would result in a new service being created (as shown e.g. by the Service Catalog) which wouldn’t have much content, and conversely, the instrumentation of the instantiation would not completely appear under the service which is the application itself. Attached is a screenshot of a Rails request showing both an SQL query (sqlite.query specifically) and an active_record.instantiation: The corresponding configuration is as follows, with ‘foo’ as custom service name for the SQL query: Datadog.configure do |c| c.service = 'hello-world-oleg' c.tracing.instrument :active_record, service_name: 'foo' end The SQL query span represents time spent in the database. It is not 100% accurate because the database client library is instrumented instead of the database server, so part of the “database” time is the time it took for the client to send the query to the database and then to receive and parse the results. The instantiation operation, in contrast, is completely performed by the application, and therefore should have the application’s service name. For this reason, the service name that is overridden for the SQL query operation is not propagated to the instantiation operation. The instantiation operation is not actually a part of the SQL query (and you can see that in the flamegraph attached), it just so happens that both operations are instrumented by the “ActiveRecord integration”. Does this make sense? |
I brought this up in #3184 and can add an additional perspective. In our case:
In other words, for a custom service name "foo", we had spans of the following operations, included via
and then one operation showing up under, in our case, the service name
We are currently working around this limitation with the monkey-patch to def process(span, event, _id, payload)
span.service = "foo"
super
end |
@TonyCTHsu thanks for the look! To address in your specific comment
I think this is only right if you are using the default instrumentation configuration. However, if you are using the supported and documented feature to use a custom service name that is not your application's name, what you get is not the same service as defined for the rest of the application. This is the problem that #3184 attempts to fix. I think #3383 actually misses the mark a bit as it ties the service name to the database service which as you point out it not really correct, as opposed to the configured service name. Let me know if this makes sense! |
Also, for completeness, we initially attempted to see if using the service_name configuration on the active_record, as opposed to rails, instrumentation option would be respected: c.tracing.instrument :active_record, service_name: "foo" vs c.tracing.instrument :rails, service_name: "foo" however this has no different effect than adding |
@ozydingo Thanks for sharing your use case. It's very similar to ours and we're having users asking why the active-record spans are mis-labeled. It took some digging to see it was because of this difference in the instantiation logic. DD Support staff recommended we use the custom service names since without that all components get mapped to a singular service in DD and this doesn't work well with the granularity allowed in the error tracking service (for instance you can't mute errors for only one services active-record component) The issue is so many of the DD APM logic only works with service, version and env tags. So the extra granularity in the span tags isn't always helpful if you can't control the service name. This issue breaks the contract with manual service name instrumentation. |
@ozydingo in your use case, would it be desirable that |
@ozydingo to clarify, when If this is the case, then I think there's a misunderstanding in the documentation. For
This was meaning to communicate that it effectively changes Our intent now is that all spans within the same Ruby application reflect the same
@dannyrife I acknowledge that this has often been recommended in the past. However, doing so will pollute I also acknowledge that without customized service names, the UX lacks the capability of sub-grouping spans within an application, and some features that are tied explicitly to This change will take some time to fully realize within the Ruby library. In the interim, if the trade-offs are worthwhile, you may continue to customize service names by modifying the underlying spans, including through the use of our "Processing Pipeline", until an improved UX is in place. Hopefully this helps clarify things, and I'm happy to take more feedback/questions/suggestions as well. |
@ozydingo In your last comment, can you clarify where "threeplay" is configured? I am not seeing it in the configuration you specified. |
It's set using the DD_SEEVICE env var on the app containers. There might also be something set in our agent (we're using a k8s setup) but I think the pod env takes preference IIRC. |
Thank you for that @ozydingo , I have done some more research and testing and I think that AR instantation instrumentation is indeed different in that it instruments two different services but doesn't permit specifying two service names. If we look at https://github.com/DataDog/dd-trace-rb/blob/master/lib/datadog/tracing/contrib/rails/framework.rb, for some Rails components we forward the service name configured for the To override this, one needs to specify Back to AR - "ActiveRecord" instrumentation instruments two different services:
Currently, service name specified to ActiveRecord instrumentation is used for the SQL queries, which is probably what most users would want and expect, but there isn't a way to specify the service name for the instantiation instrumentation, and therefore the instantiation uses the global service name (either from Datadog.configure block or from DD_SERVICE environment variable). Unfortunately, both of these operations could be considered to exist under or within ActiveRecord, which makes the situation confusing. How could this be fixed? My naive idea is to specify, for example, the following options to ActiveRecord instrumentation:
However, given that the SQL query instrumentation is probably the most common use case, and given that it's already affected by the Alternatively, perhaps AR SQL and AR instantiation could be covered by different instrumentations, at which point they could each have an option called |
@p-datadog |
@ozydingo @dannyrife Could you also clarify why you wish to provide service name to
? |
So we do set the c.service settings but since parts of the DD UI don't allow granularity beyond tags service, env, & version (like muting errors in error tracking) we are prepending all the the instrumented parts with the service name. Ideally we want to use default services and have the various parts of the UI work use all the extra tags that are added. Configuration looks something like this
|
Thank you very much, I think this is great product feedback:
|
Actually I think in our case that would work now. Originally we left the global config as a separate name as we were empirically discovering how service names were generated, as a catch for other misc service names (probably because we were confused by the extra service name from AR, as well as discrepancy between AR's MySQL instrumentation vs the direct mysql2 instrumentation, both of which we have active!) Now that we're stable and have our datadog workflow ironed out, I think using a single globally configured app service name actually does make sense, as long as we can get sensible information between AR, AR's mysql, and mysql2 instrumentation 😄 |
Came across this thread after my team upgraded from ddtrace v0.53 to v1.20 and was surprised by the
From this old issue as well as mentioned the upgrade guide, there used to be a option called I'm always curious when a feature like that is removed between major versions what the driving reasoning is especially when there end up being these kinds of feature/bug requests to create similar functionality. The change wasn't a dealbreaker on the upgrade for us, but it did seem like a weird thing to have lost the ability to do. |
Current behaviour
ActiveRecord SQL calls reflect the configured service name, but instantiations do not
The difference is clear comparing the contrib code: sql gets the settings, instantiation does not:
https://github.com/DataDog/dd-trace-rb/blob/master/lib/datadog/tracing/contrib/active_record/events/sql.rb#L34
vs
https://github.com/DataDog/dd-trace-rb/blob/master/lib/datadog/tracing/contrib/active_record/events/instantiation.rb#L32
Expected behaviour
Since this is part of the active record instrumentations, the service name should match the other instrumentation for active record spans.
Steps to reproduce
Instrument active record and create an instantiation something like below should make something that creates instantiations
Environment
Datadog.configure ...
):The text was updated successfully, but these errors were encountered: