-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
backend/http: allow to force treating all requests as trusted for tracing purposes #19
Conversation
I don't fully understand this. If the client doesn't send the traceID, we still create a new trace per request, don't we? 🤔 |
@dgzlopes The logic for traces and their relationship is a bit convoluted, I'll try to summarize the current behavior here:
This is the default behavior encouraged by opentelemetry, as blindly trusting traceparent headers can have security implications: A client could force your code to sample every request it sends, DoSing your telemetry backend. This PR introduces an env var for blindly trusting all requests: The traceparent header will be honored from external, untrusted requests, and generated spans will use the incoming, untrusted trace as a parent. This is useful for debugging purposes. LMK if this helped made the thing more clear :) |
@roobre I would eliminate
The sampling flag is one thing, but you can also configure sampling at the following levels:
Having link association configured does not help avoid the denial of wallet attack. |
Correct me if I'm wrong, but isn't the typical tracing setup to honor the tracing flag over the SDK configuration? Otherwise you would get incomplete traces, right?
My current understanding is that it does help, as it prevents clients from overriding your sampling choices. I haven't read the full spec, so I might be wrong here.
This is a fair point, but I do not fully agree. I think that as a demo application, it should do things the way they are supposed to be done, or be very obvious in doing them wrong (so everybody can say "ah, that's because it's a demo"). Trusting all traces by default is, in my view, extremely non-obvious for people without prior experience manually instrumenting things. I'm worried about people looking at the code for examples and finding concealed bad practices that they could then get carried over to real projects. As a final point, I think real applications or deployments could be configured with the equivalent of
This is important for me, I would not like to make it hard to use for demos. I'm hopeful that we can figure out a way to make the application easily demoable that without making the code deceptively dangerous to learn from. Some alternatives to getting rid of this behavior are:
@Blinkuu LMK what you think about this. I hope we can find a solution that makes the application easily demoable without making the code deceive readers into bad practices. |
@dgzlopes Any other thoughts on the PR? LMK if you have any remaining questions 😃 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If @Blinkuu is happy, I'm happy. These days, he knows the most about tracing 😄
But yeah, I gave it another look, and generally, LGTM.
Awesome! I will raise a follow-up PR setting this var to |
Trusting client-provided parent trace IDs can be a security risk, as it can open the door to DoS attacks by allowing a client to send a large amount of requests with the sample flag set to true, causing the backend to generate a large amount of tracing data. This can affect the performance of the tracing backend, and sometimes also billing.
However, this could be useful for testing, so this PR introduces the
QUICKPIZZA_TRACING_INSECURE
env var which, when set to a truthy value, will always trust trace parent IDs from external requests.