Store the global list of buckets under the "buckets:" key

[fatpat]

SUMMARY

For statistics about buckets, oio-exporter needs to have access to the entire list of buckets. Currently, we have two options:

• scan with a match on buckets:* but this is not an option as performances are terrible
• hscan over accounts: and then for each account, zscan over buckets:$account but it's too complicated and does not fit well into oio-exporter code as we wish to separate fetching accounts and buckets metrics.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

Account

SDS VERSION

7.x

ADDITIONAL INFORMATION

Feel free to to use the following script for creating buckets: key from buckets:* key on existing cluster: https://github.com/open-io/oio-toolkit/blob/master/scripts/updates-buckets:.lua

local keys = redis.call("keys", "buckets:*")
for i, key in ipairs(keys) do
	if key ~= "buckets:" then
		local zrange = redis.call("ZRANGE", key, 0, -1)
		for j, bucket in ipairs(zrange) do
			redis.call("ZADD", "buckets:", "NX", 0, bucket)
		end
	end
end

[fvennetier]

Looks good.

I had plans to merge the bucket DB with the account database, but it's not easy.

[codecov-commenter]

Codecov Report

Merging #2072 into 7.x will increase coverage by 0.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              7.x    #2072      +/-   ##
==========================================
+ Coverage   77.62%   78.48%   +0.87%     
==========================================
  Files         376      378       +2     
  Lines       73020    73447     +427     
  Branches     6567     6563       -4     
==========================================
+ Hits        56673    57637     +964     
+ Misses      16131    15603     -528     
+ Partials      216      207       -9     

Impacted Files	Coverage Δ
oio/account/backend.py	70.89% <ø> (ø)
events/oio_events_queue_beanstalkd.c	74.28% <0.00%> (-9.00%)	⬇️
tests/functional/content/test_service_id.py	78.83% <0.00%> (-5.29%)	⬇️
oio/common/utils.py	67.69% <0.00%> (-3.04%)	⬇️
oio/directory/meta0.py	84.07% <0.00%> (-0.92%)	⬇️
meta2v2/meta2_filters_extract.c	82.68% <0.00%> (-0.78%)	⬇️
meta2v2/meta2_filters_action_content.c	83.13% <0.00%> (-0.48%)	⬇️
proxy/transport_http.c	85.36% <0.00%> (-0.28%)	⬇️
server/transport_gridd.c	74.66% <0.00%> (-0.19%)	⬇️
proxy/m2_actions.c	88.32% <0.00%> (-0.13%)	⬇️
... and 65 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7151cf7...fd5e584. Read the comment docs.