Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cloudera integration #13244

Merged
merged 73 commits into from
Dec 23, 2022
Merged

Add Cloudera integration #13244

merged 73 commits into from
Dec 23, 2022

Conversation

yzhan289
Copy link
Contributor

@yzhan289 yzhan289 commented Nov 1, 2022

What does this PR do?

This PR adds support for collecting metrics and service checks from Cloudera via Cloudera Manager.

Motivation

AI-2511

Additional Notes

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have changelog/ and integration/ labels attached
  • If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

@ghost ghost added the documentation label Nov 1, 2022
@github-actions
Copy link

github-actions bot commented Nov 1, 2022

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

github-actions bot commented Nov 2, 2022

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

3 similar comments
@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

cloudera/pyproject.toml Outdated Show resolved Hide resolved
@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

1 similar comment
@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

10 similar comments
@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

1 similar comment
@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@github-actions
Copy link

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

@yzhan289 yzhan289 requested a review from ofek December 20, 2022 22:06
ofek
ofek previously approved these changes Dec 21, 2022
@yzhan289
Copy link
Contributor Author

Looks like the validate-assets job is failing because there's an unused dashboard in ArgoCD which was fixed in #13510. I will rebase my branch to get the changes.

ofek
ofek previously approved these changes Dec 22, 2022
fanny-jiang
fanny-jiang previously approved these changes Dec 22, 2022
Copy link
Contributor

@fanny-jiang fanny-jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Copy link
Contributor

@buraizu buraizu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing those changes. I made a few additional minor suggestions and posted a question about the use of should in one case, as well as a few other minor suggestions, but it looks like the Autodiscovery link in the containers tab is still opening the support page.

cloudera/README.md Outdated Show resolved Hide resolved
cloudera/README.md Outdated Show resolved Hide resolved
cloudera/README.md Outdated Show resolved Hide resolved
cloudera/README.md Outdated Show resolved Hide resolved
cloudera/README.md Outdated Show resolved Hide resolved
Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>
@yzhan289 yzhan289 merged commit 28ad542 into master Dec 23, 2022
@yzhan289 yzhan289 deleted the az/cloudera branch December 23, 2022 15:50
github-actions bot pushed a commit that referenced this pull request Dec 23, 2022
* Cloudera template

* Outline plan for integration

* Set up init metric and test env

* Add common.py

* Outline check and include tests

* Clean up check

* Update cloudera/pyproject.toml

Co-authored-by: Ofek Lev <ofekmeister@gmail.com>

* Drop support for Python 2

* Clean up timeseries metrics

* Add more metrics

* Add more metrics

* Implement host service check and fix style

* Change how cloudera client is created

* Add caddy to mock API output

* Add caddy test API outputs

* Apply Jose's metric and health check collection style (#13423)

* First commit

* Added docker-compose file

* Match current PR

* Match PR

* Fix style

* Refactor

* Revert any test changes

* Spacing

* Revert unnecessary changes

* Spacing

Co-authored-by: Jose Manuel Almaza <josemanuel.almaza@datadoghq.com>

* Apply testing changes (#13424)

* Apply changes from Jose's branch

* Add Jose's new host metrics

* Fix validations

* Add license

* Fix properties

* Change to use caddy docker image

* Start events implementation

* Unit tests working

* Map to 80 local port not recommended

* Added url to debug log

* Remove events implementation

* Fixed integration tests

* Fix license validation

* Remove unused constants

* Add e2e test

* Use master's change

* Fix e2e and integration test style

* Add temporary gitlab change

* Temporarily add default dep

* Wait for Cloudera is up

* Refactor Cloudera (#13460)

* Add native host metrics

* Remove unused function name

* Refactor Cloudera

* Remove native metric support

* Fix style

* Update refactor from feedback

* Update service checks and update metadata.csv

* Fixed missing coverage

* Added 'io' to VALID_UNIT_NAMES

* Added some tag checks to tests

* Fix service checks and add comma

* Add native metrics (#13474)

* Add native metrics

* Fix e2e test

* Move metadata.csv change to separate branch

* Renamed some methods

* Fixed problem merging metrics

* Fixed unit tests

Co-authored-by: Jose Manuel Almaza <josemanuel.almaza@datadoghq.com>

* Clean up

* Add version collection

* Add back service check for can_connect

* Update README and add pictures

* Move images directory

* Add custom tag support

* Fix style

* Add custom tag tests

* Fix test for e2e test

* Run cloudera manager requests in parallel (#13499)

* Added multithread

* Add hidden config for max connection pool size

* Added 'dd_environment' to test_metadata

* Update config

Co-authored-by: Andrew Zhang <andrew.zhang@datadoghq.com>

* Fix config param

* Remove io as metric type

* Apply suggestions from code review

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Apply suggestions

* Apply suggestion

* Fix style

* Update cloudera/README.md

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update cloudera/README.md

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Apply suggestions

* Fix tests

* Fix style

* Fix formatting

* Apply suggestion and move credentials to init_config

* Update Cloudera README for collecting metrics

* Fix README

* Fix README

* Apply suggestions from code review

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

Co-authored-by: Ofek Lev <ofekmeister@gmail.com>
Co-authored-by: Jose Manuel Almaza <josemanuel.almaza@datadoghq.com>
Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> 28ad542
@yzhan289 yzhan289 restored the az/cloudera branch December 23, 2022 19:06
@yzhan289 yzhan289 deleted the az/cloudera branch December 23, 2022 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants