Improve render directory

[zeripath]

This PR represents a small improvement for the native git backend's get
last commit. It reduces the time taken on ports from 13s to 6.5s which is
now faster than the go-git variant.

Signed-off-by: Andrew Thornton art27@cantab.net

[zeripath]

There are still significant slowdowns with subdirectories - but this is also improved - down to 9s

I think I've cracked this somewhat.

[codecov-commenter]

Codecov Report

Merging #15891 (cc6d732) into main (36dce0e) will decrease coverage by 0.01%.
The diff coverage is 56.87%.

@@            Coverage Diff             @@
##             main   #15891      +/-   ##
==========================================
- Coverage   44.10%   44.08%   -0.02%     
==========================================
  Files         682      682              
  Lines       82375    82521     +146     
==========================================
+ Hits        36328    36381      +53     
- Misses      40140    40212      +72     
- Partials     5907     5928      +21     

Impacted Files	Coverage Δ
modules/git/pipeline/lfs_nogogit.go	0.00% <0.00%> (ø)
modules/indexer/code/bleve.go	68.22% <0.00%> (-0.59%)	⬇️
modules/indexer/code/elastic_search.go	1.64% <0.00%> (-0.02%)	⬇️
modules/git/repo_language_stats_nogogit.go	47.56% <40.00%> (+2.98%)	⬆️
modules/git/batch_reader.go	50.63% <51.80%> (-3.12%)	⬇️
modules/git/commit_info_nogogit.go	68.48% <71.87%> (+0.39%)	⬆️
modules/charset/charset.go	71.71% <0.00%> (-6.07%)	⬇️
modules/queue/queue_bytefifo.go	75.14% <0.00%> (-3.56%)	⬇️
modules/git/repo_base_nogogit.go	82.85% <0.00%> (-2.86%)	⬇️
modules/git/utils.go	62.50% <0.00%> (-2.78%)	⬇️
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3720503...cc6d732. Read the comment docs.

[lunny]

Thanks for your effort. Unfortunately, haven't see an obvious increment of the speed of linux kernel repo.

I tested with linux kernel home page,
Before twice time are 37850ms and 39430ms
After twice time are 37796ms and 37421ms

my test env is macOS

[zeripath]

@lunny can you double check that test repo has a commit-graph written?

[zeripath]

(in .git/objects/info/commit-graph)

[zeripath]

It's really weird as I am I get 2s for rendering git.git on my laptop.

Could you try:

https://github.com/zeripath/gitea/tree/use-os-pipe

This switches to use an os.Pipe with all the above changes in place.

[lunny]

I have generated that commit graph file, but from your patch, it seems you haven't read it?

[lunny]

@zeripath
Before generated commit-graph file
Main branch twice time are 32s and 34s
This PR twice time are 32s and 32s

After generated commit-graph file
Main branch twice time are 23s and 23s
This PR twice time are 23s and 23s

[zeripath]

I'm not reading it directly - I'm relying on git reading it - the commit-graph would speed up the git rev-list --format=%T commitid -- treepath call - so having one should exclude that as a cause of the problem.

The main slow down in all of this code appears to be getting information out of and possibly to the git cat-file --batch pipes. Mostly this PR simply re-ordered things in an attempt to reduce the time spent filling the buffer and time spent locked in bufio.fill(...) etc. It was enough for my (quite fast) laptop but clearly not enough for yours and others.

Try the os.Pipes thing - if there's any improvement it would be good to know.

[zeripath]

Ah I wonder if your systems are stdout buffering?

[zeripath]

Closing in favour of #16059 which is faster