cli: add cost estimate and success logging to tsdump upload #148575
Conversation
jordanlewis
requested review from
dhartunian,
arjunmahishi,
aa-joshi and
Abhinav1299
and removed request for
a team
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=95b789b1d594ac6b74f377aa7555d8d98d63240d&v=4){kind=avatar}
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
jordanlewis
changed the title
|
Hi @aa-joshi @arjunmahishi, I created this PR to help understand overall costs of tsdump uploading in Datadog. Please take a look. The goal is that we should be able to audit the count of tsdump upload requests, the total number of series, and the estimated cost and compare it against what Datadog charges us for the historical ingest. I also took the opportunity to replace the manually constructed REST API calls with usage of the Datadog Go client. If there's a reason why we weren't using that before, I can certainly back out that part of the change. |
jordanlewis
force-pushed
the
tsdump-upload-cost
branch
4 times, most recently
from
db37ade to
a498385
Compare
This commit adds a cost estimate to the tsdump Datadog upload command to help people understand how much a tsdump upload costs (roughly $.20 per single-node upload, $2 per 10-node upload, or $20 per 100-node upload). Release note: None
Previously, we used raw POST requests in the tsdump uploader. Now, we use the Datadog API client which is already linked in elsewhere in the CockroachDB binary, for use in the cockroach workload command. Epic: none Release note: None
Upload a log message to Datadog upon completion of a tsdump upload, with structured information for the count of uploaded timeseries, the duration, and the estimated cost of the operation. Epic: none Release note: None
jordanlewis
force-pushed
the
tsdump-upload-cost
branch
from
a498385 to
068f386
Compare
|
Thanks for the review! Please take another look. |
There was a problem hiding this comment.
Reviewed 4 of 8 files at r2.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @Abhinav1299, @arjunmahishi, @dhartunian, and @jordanlewis)
This PR adds a cost estimate to the tsdump Datadog upload command to help people understand how much a tsdump upload costs (roughly $.20 per single-node upload, $2 per 10-node upload, or $20 per 100-node upload).
It also adds structured success logging so that it's possible to see historical information about the uploads that were sent to Datadog.
Epic: none
Release note: None