Span the hardware inventory collection over a period of time.#199
Span the hardware inventory collection over a period of time.#199r0h4n merged 2 commits intoTendrl:masterfrom
Conversation
|
@brainfunked @r0h4n @nthomas-redhat @shtripat Please review |
shtripat
left a comment
shtripat
left a comment
There was a problem hiding this comment.
LGTM.
Comments from others required.
r0h4n
left a comment
r0h4n
left a comment
There was a problem hiding this comment.
Looks good, but how does it affect performance and alerting cc @anmolbabu (please review)
|
@r0h4n None of these impact performance monitoring because that is in total done for us by collectd. But, in future when we add service status monitoring, we might only see a lag in the time of it being noticed and that very very minute and I think it is very much tolerable. And given that it is not delaying the first time update(Its only that interval b/w subsequent syncs has changed) specially of node context, the change looks good to me. |
|
@nnDarshan What is the cpu consumption with this PR? |
| try: | ||
| gevent.sleep(3) | ||
| interval = 10 | ||
| if tendrl_ns.first_node_inventory_sync == True: |
There was a problem hiding this comment.
a simple "if tendrl_ns.first_node_inventory_sync" is enough
| if s.running: | ||
| tags.append(TENDRL_SERVICE_TAGS[service.strip("@*")]) | ||
| s.save() | ||
| gevent.sleep(interval) |
There was a problem hiding this comment.
do not sleep for every iteration of this loop, put this after the loop
Currently the node inventory is synced every 3 seconds which is very aggressive, considering they change very rarely. So Its better to change the sync interval. Also instead of syncing everything after certain interval (say 5 or 10 mins) of time which causes a huge spike in CPU utilization every time this interval is reached, We can span the inventory collection over a period of time. In this patch I have introduced a gevent sleep of 10 secs after collecting every resource. So Its like every resource is updated every 3 minutes once. tendrl-bug-id: Tendrl#198 Signed-off-by: nnDarshan <darshan.n.2024@gmail.com>
|
@r0h4n Earlier in the machine I tested CPU util was continuously 40-50%... now with this fix and Tendrl/commons#151 its always below 1% and when we fetch inventory it spikes a little to around 5-10% |
4 participants
Currently the node inventory is synced every 3 seconds which is very aggressive, considering they change very rarely. So Its better to change the sync interval. Also instead of syncing everything after certain interval (say 5 or 10 mins) of time which causes a huge spike in CPU utilization every time this interval is reached, We can span the inventory collection over a period of time.
In this patch I have introduced a gevent sleep of 10 secs after collecting every resource. So Its like every resource is updated every 3 minutes once.
tendrl-bug-id: #198