Cache last commit to accelerate the repository directory page visit

[lunny]

Will replace #6045, resolve #491

On my local test (macOS ) with default cache setting(memory):

aports (home directory): 4010ms -> 225ms
vdsm (home directory): 1510ms -> 112ms

BTW: aports /main has many files, so I haven't tested it. We need a limitation on files listed. That will be another PR.

[6543]

@lunny instead of a hard lim, could we paginate it somehow?

[6543]

can you add some tests?

[silverwind]

Why the separate options? Is the new cache expected to go big?

[6543]

I like the added Enable option!
@silverwind for me: I'm totaly fine with more options ...
so you can tweek around as instance admin

[lafriks]

I don't think we need separate cache for last commit, just use cache we already have in place

[silverwind]

Yeah, if there is no compelling reason to make it a separate cache, I'd prefer it to be included in the regular cache.

[lunny]

@lunny instead of a hard lim, could we paginate it somehow?

That will be another PR.

@lafriks @silverwind I have sent another commit that will use the original cache config defautly.

I think the default cache time 16h is not a good default value and I also needs some extra config items like commits_count and etc. So I added a seperate cache config.

[lunny]

@6543 I will try to add a test but since it's a page performance optimization for big repository but the test repositories are small. The test will only confirm the results are the same before enable the cache or after.

[codecov-io]

Codecov Report

Merging #10069 into master will increase coverage by 0.01%.
The diff coverage is 81.25%.

@@            Coverage Diff             @@
##           master   #10069      +/-   ##
==========================================
+ Coverage   43.48%   43.49%   +0.01%     
==========================================
  Files         567      567              
  Lines       79067    79104      +37     
==========================================
+ Hits        34381    34409      +28     
- Misses      40440    40447       +7     
- Partials     4246     4248       +2

Impacted Files	Coverage Δ
routers/repo/branch.go	55.72% <100%> (+0.34%)	⬆️
modules/references/references.go	80.32% <80.85%> (+0.41%)	⬆️
services/pull/check.go	54.54% <0%> (-2.1%)	⬇️
services/pull/patch.go	66.03% <0%> (-1.89%)	⬇️
modules/queue/workerpool.go	46.61% <0%> (-1.78%)	⬇️
modules/log/event.go	65.64% <0%> (+1.02%)	⬆️
modules/queue/manager.go	62.05% <0%> (+1.53%)	⬆️
modules/process/manager.go	78.31% <0%> (+3.61%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c15871...300e790. Read the comment docs.

[6543]

@6543 I will try to add a test but since it's a page performance optimization for big repository but the test repositories are small. The test will only confirm the results are the same before enable the cache or after.

i think it is enouth if you add test to test if cache works at all ... (set cache-size to low value .. ; open repo & test if it is in the cash) ?

[lafriks]

@lunny only following settings are needed for [cache.last_commit] section

• ENABLED: true: Enable the cache. - edit: setting ITEM_TTL to 0 will disable cache so no need for this option also
• ITEM_TTL: 16h: Time to keep items in cache if not used, Setting it to 0 disables caching.
• COMMITS_COUNT: 1000: Only enable the cache when repository's commits count great than.

You can reuse same cache and set specific TTL for last_commit keys.

[lafriks]

@lunny I created PR to your PR with what I meant: lunny#4

[lunny]

@lafriks done.

[6543]

without

• aports/src/branch/master/main: 78375ms(first) - 76574ms(snd.)
• aports/src/branch/master/community: 38033ms(first) - 36449ms(snd.)

with

• aports/src/branch/master/main: 77753ms(first) - 591ms(snd.)
• aports/src/branch/master/community: 37208ms(first) - 685ms(snd.)

[silverwind]

Maybe pre-populate cache on startup and push?

[6543]

I think the endresult is just more load on the system ... if we implement this i like to be able to disable this @silverwind

[lunny]

@silverwind I will send another PR to do that. Because I think it's need a queue that will make this PR more complicated.

[lunny]

@lafriks need your approval.