Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RRD storage paths do not scale properly #4533

Closed
TheWitness opened this issue Jan 24, 2022 · 4 comments
Closed

RRD storage paths do not scale properly #4533

TheWitness opened this issue Jan 24, 2022 · 4 comments
Labels
boost Boost related issue enhancement General tag for an enhancement resolved A fixed issue
Milestone
v1.2.22

Comments

@TheWitness
Copy link
Member

Describe the bug

On large systems with over 1M RRDfiles, it's very difficult to restructure paths and keep polling operational at the same time. The current CLI script first disables polling, then restructures the paths one at a time which is arguably too slow. Additionally, if you have 20k+ hosts, you still end up with a directory of over 20k files, which though not as bad, is still excessive.

To Reproduce

Steps to reproduce the behavior:

  1. Create a system with 1.7M RRDfiles and over 20k hosts

  2. Allow all RRDfiles to be created in the RRA directory

  3. Attempt to "structure the RRDfile paths"

  4. See error

Expected behavior

Cacti should be able to execute this process with no downtime.
@TheWitness TheWitness added bug Undesired behaviour enhancement General tag for an enhancement labels Jan 24, 2022
@TheWitness TheWitness added this to the 1.2.21 milestone Feb 5, 2022
@TheWitness TheWitness added the boost Boost related issue label Feb 15, 2022
@browniebraun
Copy link
Member

This was the first script for Cacti I originally wrote in 2006. Times are changing...

@browniebraun
Copy link
Member

or was in 2005? I don't know. 🙄🤔😅

@TheWitness
Copy link
Member Author

Where's my cane 😂

@TheWitness TheWitness modified the milestones: v1.2.21, v1.2.22 Apr 14, 2022
@TheWitness TheWitness removed the bug Undesired behaviour label May 5, 2022
@TheWitness TheWitness modified the milestones: v1.2.22, v1.2.23 Jun 19, 2022
@TheWitness TheWitness modified the milestones: v1.2.23, v1.2.22 Aug 4, 2022
@TheWitness
Copy link
Member Author

I've bumped this one to 1.2.22 as it's material for me right now.

TheWitness added a commit that referenced this issue Aug 7, 2022
@TheWitness
Fixing #4882, #4881, #4880, #4533 - Boost and Structured Paths
6c07b04 
This change adds two features, and corrects some minor bugs:

* issue#4880: The MySQL get_lock() function can lead to stalled boost and dsstats processes
* issue#4882: Under rare conditions the boost process can get hung until the poller times out
* feature#4533: Restructuring RRDfile paths takes too long on large systems and will stop polling for potentially several hours
* feature#4881: When boost is running, it's not clear which processes are running and how long they have to complete
TheWitness added a commit that referenced this issue Aug 7, 2022
@TheWitness
Fixing #4882, #4881, #4880, #4533 - Boost and Structured Paths
0d0b272 
Missed one minor change entry for this commit
@TheWitness TheWitness added the resolved A fixed issue label Aug 7, 2022
TheWitness added a commit that referenced this issue Aug 9, 2022
@TheWitness
Add two new methods to #4533 Structured Paths
15b096b 
- In my case, I would still have too many base directories.  So, this change will add a third possible level to spread out files within the file system.
@netniV netniV changed the title Restructuring RRDfile paths takes too long on large systems and will stop polling for potentially several hours RRD storage paths do not scale properly Aug 14, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Nov 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
boost Boost related issue enhancement General tag for an enhancement resolved A fixed issue
Projects
None yet
Milestone
v1.2.22
Development

No branches or pull requests
2 participants
@TheWitness @browniebraun