Skip to content

Xena: Add basic monitoring & alerting stack #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Oct 31, 2022
Merged

Xena: Add basic monitoring & alerting stack #179

merged 10 commits into from
Oct 31, 2022

Conversation

cityofships
Copy link
Member

It's meant to be minimal working configuration for all clouds. Draws some inspiration from #18 but it's stripped down version.

@cityofships cityofships requested a review from a team as a code owner October 5, 2022 10:00
@cityofships cityofships mentioned this pull request Oct 5, 2022
Closed
@cityofships
Copy link
Member Author

CI will probably fail due to prometheus-msteams inventory group missing.

@cityofships cityofships force-pushed the xena_monitoring branch 4 times, most recently from 75fef0d to eddc4d2 Compare October 6, 2022 09:40
@dougszumski
Copy link
Member

CI will probably fail due to prometheus-msteams inventory group missing.

@k-s-dean is working on a patch to add it

@cityofships
Copy link
Member Author

stackhpc/kayobe#59

dougszumski
dougszumski reviewed Oct 7, 2022
View reviewed changes
Copy link
Member

@dougszumski dougszumski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cityofships, looks good, these are the most recent dashboards from sms-lab? A couple of minor questions.

@dougszumski
Copy link
Member

dougszumski commented Oct 7, 2022 •
edited
Loading

stackhpc/kayobe#59

Thanks, Kyle has made https://review.opendev.org/c/openstack/kayobe/+/860670

dougszumski
dougszumski reviewed Oct 7, 2022
View reviewed changes
@cityofships
Copy link
Member Author

Thanks @cityofships, looks good, these are the most recent dashboards from sms-lab? A couple of minor questions.

Hmm, let's review https://github.com/stackhpc/smslab-kayobe-config/pull/159 and update here if merged - the rest is up to date

dougszumski
dougszumski previously approved these changes Oct 7, 2022
View reviewed changes
Copy link
Member

@dougszumski dougszumski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, thanks @cityofships

cityofships and others added 5 commits October 10, 2022 18:53
@cityofships
Enable Kolla features
61b5070
@cityofships
Elasticsearch memory tuning
6940e0e
@cityofships @dougszumski
@k-s-dean @priteau
Add standard alerting rules and dashboards
a8deca1 
Co-authored-by: Doug Szumski <doug@stackhpc.com>
Co-authored-by: k-s-dean <kyle@stackhpc.com>
Co-authored-by: Pierre Riteau <pierre@stackhpc.com>
@cityofships
Use sane defaults for basic monitoring stack
5b7330f
@jovial @cityofships
Consider agents that are auto-downed
9a9efd2 
The count of agents down failed to include nova-compute which was disabled by nova automatically because the connection to libvirt was lost. This still represents an error condition.
@cityofships
Copy link
Member Author

Rebased

@markgoddard
Copy link
Contributor

Will try to review tomorrow, but immediate concern is that when using multiple environments the config in etc/kayobe/kolla/config/ won't get picked up.

markgoddard
markgoddard reviewed Oct 12, 2022
View reviewed changes
@markgoddard
Copy link
Contributor

@cityofships could we move this forward so that it's ready for the SMS lab upgrade?

@cityofships
Copy link
Member Author

cityofships commented Oct 20, 2022 •
edited
Loading

@markgoddard Please merge if needed - the inline comments and/or README don't have an influence on the upgrade.

markgoddard
markgoddard reviewed Oct 27, 2022
View reviewed changes
Copy link
Contributor

@markgoddard markgoddard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to see the CI passing. One thing that could be improved is if the aio environment used the custom configuration that's being added here. Currently that requires symlinking it into the ci-aio environment.

@markgoddard
Copy link
Contributor

Nice to see the CI passing. One thing that could be improved is if the aio environment used the custom configuration that's being added here. Currently that requires symlinking it into the ci-aio environment.

It would also be a good opportunity to test the documentation (which I assume still needs writing) on how to use the config.

@markgoddard markgoddard merged commit 190e83c into stackhpc/xena Oct 31, 2022
@markgoddard markgoddard deleted the xena_monitoring branch October 31, 2022 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dougszumski dougszumski dougszumski left review comments

@jovial jovial Awaiting requested review from jovial

+1 more reviewer

@markgoddard markgoddard markgoddard approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cityofships @dougszumski @markgoddard @jovial