[Celerdata] integration with CelerData #2326

yandongxiao · 2024-03-19T02:24:15Z

What does this PR do?

The integration work between Celerdata and Datadog has been completed.

Motivation

We want to submit an official integration.

Review checklist

PR has a meaningful title or PR has the no-changelog label attached
Feature or bugfix has tests
Git history is clean
If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

Anything else we should know when reviewing?

This is the first version of our integrated work, mainly completing:

Reporting all Prometheus metrics to Datadog
Providing comments in conf.yaml.example explaining the log file paths that need to be collected.

github-actions · 2024-03-19T03:12:25Z

Test Results

2 tests 2 ✅ 1m 38s ⏱️
1 suites 0 💤
1 files 0 ❌

Results for commit 27f512d.

♻️ This comment has been updated with latest results.

Signed-off-by: yandongxiao <yandongxiao@starrocks.com>

celerdata/README.md

celerdata/images/IMAGES_README.md

celerdata/manifest.json

Signed-off-by: yandongxiao <dxyan06@gmail.com>

alai97

Some doc feedback~

celerdata/README.md

alai97 · 2024-03-21T23:35:36Z

celerdata/metadata.csv

@@ -0,0 +1,543 @@
+metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name,curated_metric


Please reformat the description as complete sentences, for example:

Job statistics.
The maximum journal ID of this frontend.

Also noting that there are missing metric descriptions in here.

‌‌‌‌‌‌‌As we do not have documentation to fully describe these metrics, the descriptions of these metrics are missing and may not be immediately fixable at this time.

‌‌The reformatting issue has been resolved.

Hey, it looks like a lot of these have 11 columns instead of 10 and isn't a valid CSV file, could you please fix this?
(Validations have been improved, so future commits on this PR should catch those and report errors back to you)

Yes, have Fixed it

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

thibaultkrebs · 2024-04-23T06:56:02Z

celerdata/assets/logs/celerdata_tests.yaml

+       - "source:LOGS_SOURCE"
+     timestamp: 1712716343937
+ -
+   sample: "I0328 10:22:09.303526 1066 olap_server.cpp:844] try to perform path gc by tablet!"


I have an issue to test this one in staging. There seems there is no date (MMdd) in this log sample, but there is an expected one in the grok parser?

39003729303 for the timestamp means it is definitely wrong

The date (MMdd) does exist, 'I0328' includes the log level and date information.

During the deployment process of Celerdata, different components, FE/BE, will be deployed on different host machines. This type of log is from the BE component.

Thank you for your solid review! I will check why the timestamp is wrong

Yes you are right, I got mislead by believing the number after was just a code. But this is correct, it is just the date MMdd being parsed.

I replicated the grok parser with your sample in Datadog and indeed this is parsing the timestamp well.

We have a problem with our internal tool to generate this result in the CI. I will need to investigate. Is your integration urgent to be deployed, or if we delay for few days the time I find the issue is fine on your side?

I think we can accept to wait few days. During the waiting period, Can you continue to review other parts of this PR? I don't know how long it will take to merge PR.

I will give you an update by this week on the log part. For other files included in this PR, some other people will be in charge to review. I invite you to check with the partner team if you think some areas are still pending reviews.

I have discussed internally with our team, and it appears that we won't be addressing this issue immediately. To avoid holding up this PR, please go ahead and remove this test for now. We plan to revisit this within a few weeks to resolve the issue and reintroduce the test.

I apologize for any inconvenience, especially after I had requested you to increase the test coverage...

OK, thank you for your feedback

Signed-off-by: yandongxiao <dxyan06@gmail.com>

thibaultkrebs · 2024-04-25T01:42:07Z

Looks good for me in our staging environment. I stamp my approval!

Signed-off-by: yandongxiao <dxyan06@gmail.com>

.github/CODEOWNERS

celerdata/README.md

Co-authored-by: bgoldberg122 <ben.goldberg@datadoghq.com>

yandongxiao requested review from a team as code owners March 19, 2024 02:24

yandongxiao force-pushed the celerdata-datadog-integration2 branch from f53931f to a83816c Compare March 19, 2024 03:07

yandongxiao force-pushed the celerdata-datadog-integration2 branch 6 times, most recently from 2ba0375 to 04f47ea Compare March 19, 2024 12:30

jhgilbert added the editorial review Waiting on a more in-depth review from a docs team editor label Mar 19, 2024

[Celerdata] integration with CelerData

88b24be

Signed-off-by: yandongxiao <yandongxiao@starrocks.com>

yandongxiao force-pushed the celerdata-datadog-integration2 branch from 04f47ea to 88b24be Compare March 20, 2024 16:04

emarsha94 added the kind/technology-partner label Mar 20, 2024

datadog-assets bot added agent/review-requested docs/review-requested ecosystems/review-requested product/review-requested labels Mar 20, 2024

emarsha94 requested changes Mar 20, 2024

View reviewed changes

datadog-assets bot added product/requested-changes and removed product/review-requested labels Mar 20, 2024

[CelerData] Fix some documents based on review comments

754c1ce

Signed-off-by: yandongxiao <dxyan06@gmail.com>

alai97 reviewed Mar 21, 2024

View reviewed changes

yandongxiao and others added 6 commits March 22, 2024 15:38

Update celerdata/README.md

d0ee339

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

Update celerdata/README.md

27c8a88

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

Update celerdata/README.md

dcf5cc9

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

Update celerdata/README.md

152fe1d

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

Update celerdata/README.md

842dccd

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

Update celerdata/README.md

c67f067

Co-authored-by: Austin Lai <76412946+alai97@users.noreply.github.com>

thibaultkrebs added assets/deploy-logs-staging ONLY USED BY Logs Backend - Validates that a PR is OK to go to staging and removed assets/no-deploy Prevents APW from deploying this PR in staging labels Apr 22, 2024

thibaultkrebs reviewed Apr 23, 2024

View reviewed changes

yandongxiao added 3 commits April 23, 2024 15:31

fix wrong timestamp in celerdata_tests.yaml

875d8ad

Signed-off-by: yandongxiao <dxyan06@gmail.com>

remove a test case from celerdata_tests.yaml

0bed828

Signed-off-by: yandongxiao <dxyan06@gmail.com>

trigger ci

9c5dd69

Signed-off-by: yandongxiao <dxyan06@gmail.com>

thibaultkrebs approved these changes Apr 25, 2024

View reviewed changes

nmuesch added the assets/no-deploy Prevents APW from deploying this PR in staging label Apr 25, 2024

yandongxiao added 2 commits April 26, 2024 10:03

trigger ci

65afdfc

Signed-off-by: yandongxiao <dxyan06@gmail.com>

Fix: Some lines has too many column(s) detected, found 11, expected 10

1976d45

Signed-off-by: yandongxiao <dxyan06@gmail.com>

emarsha94 added the product/logos-requested label Apr 30, 2024

emarsha94 requested a review from nmuesch May 2, 2024 17:20

emarsha94 approved these changes May 2, 2024

View reviewed changes

datadog-assets bot added product/approved and removed product/requested-changes labels May 2, 2024

emarsha94 assigned nmuesch and bgoldberg122 and unassigned nmuesch May 2, 2024

bgoldberg122 removed the assets/no-deploy Prevents APW from deploying this PR in staging label May 2, 2024

bgoldberg122 requested changes May 2, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

.github/CODEOWNERS Outdated Show resolved Hide resolved

celerdata/README.md Outdated Show resolved Hide resolved

yandongxiao and others added 3 commits May 4, 2024 08:39

Update .github/CODEOWNERS

2dfbcb6

Co-authored-by: bgoldberg122 <ben.goldberg@datadoghq.com>

Update .github/CODEOWNERS

4d99eb6

Co-authored-by: bgoldberg122 <ben.goldberg@datadoghq.com>

Update celerdata/README.md

27f512d

Co-authored-by: bgoldberg122 <ben.goldberg@datadoghq.com>

yandongxiao requested a review from bgoldberg122 May 6, 2024 03:07

bgoldberg122 approved these changes May 7, 2024

View reviewed changes

datadog-assets bot added ecosystems/approved and removed ecosystems/requested-changes labels May 7, 2024

bgoldberg122 added the product/logos-complete label May 14, 2024

bgoldberg122 merged commit 663a7ef into DataDog:master May 14, 2024
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Celerdata] integration with CelerData #2326

[Celerdata] integration with CelerData #2326

yandongxiao commented Mar 19, 2024

github-actions bot commented Mar 19, 2024 •

edited

alai97 left a comment

alai97 Mar 21, 2024

yandongxiao Mar 22, 2024

yandongxiao Mar 22, 2024

nmuesch Apr 25, 2024

yandongxiao Apr 26, 2024

thibaultkrebs Apr 23, 2024 •

edited

thibaultkrebs Apr 23, 2024

thibaultkrebs Apr 23, 2024 •

edited

yandongxiao Apr 23, 2024

yandongxiao Apr 23, 2024 •

edited

thibaultkrebs Apr 24, 2024 •

edited

yandongxiao Apr 24, 2024 •

edited

thibaultkrebs Apr 24, 2024

thibaultkrebs Apr 24, 2024

yandongxiao Apr 24, 2024

thibaultkrebs commented Apr 25, 2024

		@@ -0,0 +1,543 @@
		metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name,curated_metric

[Celerdata] integration with CelerData #2326

[Celerdata] integration with CelerData #2326

Conversation

yandongxiao commented Mar 19, 2024

What does this PR do?

Motivation

Review checklist

Additional Notes

github-actions bot commented Mar 19, 2024 • edited

Test Results

alai97 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thibaultkrebs Apr 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thibaultkrebs Apr 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yandongxiao Apr 23, 2024 • edited

Choose a reason for hiding this comment

thibaultkrebs Apr 24, 2024 • edited

Choose a reason for hiding this comment

yandongxiao Apr 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thibaultkrebs commented Apr 25, 2024

github-actions bot commented Mar 19, 2024 •

edited

thibaultkrebs Apr 23, 2024 •

edited

thibaultkrebs Apr 23, 2024 •

edited

yandongxiao Apr 23, 2024 •

edited

thibaultkrebs Apr 24, 2024 •

edited

yandongxiao Apr 24, 2024 •

edited