Long delay with brws'ing a file in a large repository

[shmup]

Howdy, just did a cargo build of the tool to test out simply browsing a file in the browser on its relevant branch, in Github.

I'm seeing times such as:

$ time git brws Makefile

real    0m4.878s
user    0m4.761s

And of course I tested this on a much smaller repo, and it is "instant"

I'm curious what the underlying mechanism is for I suppose finding which URL to open, and what could be a bottleneck within it regarding large repos?

I'm mostly interested in this tool for this single behavior, as it allows me to remove a vim plugin thats sole purpose is also this behavior, and I use it a lot in aforementioned repo

Thanks :)

[rhysd]

Thank you for reporting this. I think this is because underlying some git command invocations are slow. I should check which invocation is slow. Did you use some opensource repository? If so, please let me know. I'll try to reproduce this with it. Otherwise I'll try some big repository like LLVM or Linux.

[shmup]

It's a private work repository, but I will also check this on something larger and see what I can uncover.

If push comes to shove I'll probably dig into https://github.com/tpope/vim-rhubarb just to see how tpope went about it, and then can learn how you went about it

Will report back

[shmup]

Okay, testing on https://github.com/llvm/llvm-project was a good idea:

$ time git brws README.md

real    0m4.539s
user    0m4.125s

Curious what your experience will be.

And for the record, tpope pretty much cheats making his technique "instant" in all cases, by taking the remote url and converting it into the Github url. He has that liberty because his plugin only cares about Github, but I suppose it seems like a harmless way to go about it in your build_github_like_url fn, tho it maybe feels "dirty"

function! rhubarb#HomepageForUrl(url) abort
    let domain_pattern = 'github\.com'
    let domains = get(g:, 'github_enterprise_urls', get(g:, 'fugitive_github_domains', []))
    call map(copy(domains), 'substitute(v:val, "/$", "", "")')
    for domain in domains
	let domain_pattern .= '\|' . escape(split(domain, '://')[-1], '.')
    endfor
    let base = matchstr(a:url, '^\%(https\=://\%([^@/:]*@\)\=\|git://\|git@\|ssh://git@\)\=\zs\('.domain_pattern.'\)[/:].\{-\}\ze\%(\.git\)\=/\=$')
    if index(domains, 'http://' . matchstr(base, '^[^:/]*')) >= 0
	return 'http://' . tr(base, ':', '/')
    elseif !empty(base)
	return 'https://' . tr(base, ':', '/')
    else
	return ''
    endif
endfunction

[rhysd]

Curious what your experience will be.

I tried llvm-mirror/llvm on my machine (iMac 2013 Late) but it only took 0.063s.

$ time git brws README.txt
git brws README.txt  0.03s user 0.02s system 70% cpu 0.063 total

And for the record, tpope pretty much cheats making his technique "instant" in all cases, by taking the remote url ...

Thank you for digging into. Would you try below commands and check if this issue occurs or not?

• git brws -r 'owner/repo'
• git brws -R origin

As you kindly investigated, getting remote URL might be the cause. There are two functions to get remote URL in git-brws implementation.

• remote_url(): Get remote URL from config. It should be fast because it just retrieves config value
• tracking_remote(): This may be slow because it retrieves current tracking remote using git rev-parse

If tracking_remote is slow, giving -R should not cause this issue.

[shmup]

Hm, well certainly it is much quicker for you than me! I'm running from an X1 Carbon from ~2017

Trying git brws -r 'llvm/llvm-project' is of course fast, as is -R origin, when not specifying a file.

Otherwise, it is:

$ time git brws -R origin README.md

real    0m4.224s
user    0m4.023s

[rhysd]

I listed git commands invoked while running git brws. Since I cannot be reproducing this issue, would you please try following script in your private repo? I want to know which part is slow.

time git config --get remote.origin.url
time git rev-parse --abbrev-ref --symbolic '@{u}'
time git rev-parse --abbrev-ref --symbolic HEAD
time git rev-parse --absolute-git-dir
time git rev-parse HEAD

[shmup]

Well this is something!

$ time git config --get remote.origin.url
git@github.com:nsone/platform.git

real    0m0.003s
user    0m0.003s
sys     0m0.001s

$ time git rev-parse --abbrev-ref --symbolic '@{u}'
origin/fe-323/update-decisions-chart

real    0m0.005s
user    0m0.005s
sys     0m0.000s

$ time git rev-parse --abbrev-ref --symbolic HEAD
fe-323/update-decisions-chart

real    0m0.003s
user    0m0.003s
sys     0m0.000s

$ time git rev-parse --absolute-git-dir
/home/jared/workspace/platform/.git

real    0m0.002s
user    0m0.002s
sys     0m0.000s

$ time git rev-parse HEAD
22b58ad14e7309129d489cb86f28c826a0068f5d

real    0m0.003s
user    0m0.002s
sys     0m0.000s

Oh right, I should probably point these at a specific file though! git brws as it is, runs plenty fast. It's only when specifying a file..

[shmup]

Here's pointing at a file.. same thing, hm:

$ time git config --get remote.origin.url README.md

real    0m0.003s
user    0m0.002s
sys     0m0.001s

$ time git rev-parse --abbrev-ref --symbolic '@{u}' README.md
origin/fe-323/update-decisions-chart
README.md

real    0m0.005s
user    0m0.004s
sys     0m0.000s

$ time git rev-parse --abbrev-ref --symbolic HEAD README.md
fe-323/update-decisions-chart
README.md

real    0m0.002s
user    0m0.002s
sys     0m0.000s

$ time git rev-parse --absolute-git-dir README.md
/home/jared/workspace/platform/.git
README.md

real    0m0.002s
user    0m0.002s
sys     0m0.000s

$ time git rev-parse HEAD README.md
22b58ad14e7309129d489cb86f28c826a0068f5d
README.md

real    0m0.002s
user    0m0.002s
sys     0m0.000s

[rhysd]

Thank you for your help. Ah, I missed one more command which is executed by try_parse_file_or_dir() in page.rs... I'm sorry but would you try following as well?

$ hash=$(git rev-parse HEAD)
$ time git branch --remote --contains $hash

This might be slow when your private repository has so many remote branches since it checks specified hash is included in some remote branch. LLVM repository would not be slow because it has only 32 remote branches. This check is necessary to check if the hash is already pushed to remote. If it is not pushed to remote yet, opening the URL will run into 404. In the case git brws will fall back into branch name instead of hash.

[rhysd]

OK, I finally could reproduce this issue. I've remembered that Kotlin repository has so many 2821 branches.

https://github.com/JetBrains/kotlin

I cloned it and tried to reproduce this issue:

$ cd /path/to/kotlin
$ time git brws ReadMe.md
git brws ReadMe.md  5.51s user 0.17s system 96% cpu 5.871 total

It took 5.871 seconds.

I also confirmed git branch --remote --contains takes most of execution time (5.47 of 5.87).

$ time git branch --remote --contains $(git rev-parse HEAD)
  origin/HEAD -> origin/master
  origin/master
  origin/rr/mg/kotlin-jps-without-java-clean
git branch --remote --contains $(git rev-parse HEAD)  5.47s user 0.22s system 91% cpu 6.230 total

[rhysd]

I will try to fix this issue.

[rhysd]

This issue was fixed at a341398 as follows:

$ cd /path/to/kotlin
$ time git-brws ReadMe.md
git-brws ReadMe.md  0.04s user 0.05s system 63% cpu 0.154 total

[rhysd]

The fix was included in 0.11.5.