Repo structure suggestion

[mattfarina]

Homebrew has run into a problem where you can't even view the formula listing on GitHub. Running ls is even a bad idea. Alternatively, the folks working on rust also use an index on GitHub but found a different structure that's worth looking at. See https://github.com/rust-lang/crates.io-index

[bacongobbler]

Is this because users want to search for formula using github rather than an optimized search engine e.g. brew search? I saw this limitation too, but it doesn't seem to look like a problem with git, but moreso with searching for formulae on github.

Running time ls homebrew-core/Formula clocks in barely over a millisecond for me, for example:

><> time ls homebrew-core/Formula

...

real	0m0.015s
user	0m0.009s
sys	0m0.005s

Would it be better to lean users towards something like the search engine as implemented on https://crates.io/ instead? That might be a good solution to provide users a way to search against fish-food.

[mattfarina]

I couldn't find the reason for the structure when I went searching.

My thoughts were around maintainers for the apps in the repository. I didn't think of end users using it so much as those who need to maintain their apps.

My personal experience (just 1 person) is that many files in one directory become a pain. For example, when I want to edit a formula, in homebrew, it can be a pain. Try this, clone the homebrew Formula repo and then open it in visual studio code. Then try to edit something. Take the formula maintainers experience for a spin. This is where I've run into annoyances.

[bacongobbler]

I found some very interesting data from a github engineer in a comment on cocoapods and having a flat file directory structure, similar to homebrew/fish-food: CocoaPods/CocoaPods#4989 (comment)

4. Finally, the layout of the repo itself doesn't help. Specifically, the Specs directory, which contains 16k+ subdirectories, causes some Git operations to be unexpectedly expensive, further driving up CPU usage.

Longer-term, you should also consider points 1 and 4. Using GitHub as your CDN is not ideal, for anybody involved. I would urge you to consider how CocoaPods could be distributed without using Git operations, which are intrinsically hard to scale. I'm confident that you could come up with a more reliable approach for serving packages. Perhaps a method that is more similar to the approaches used by other packaging systems would work better.

If anyone has data/docs/source code on how cargo indexes their repository, that would be much appreciated, otherwise I'll have to carve out some time and have a look into this.

[bacongobbler]

The CDN piece here isn't necessarily important considering that gofish doesn't host the package's release assets itself (though it might be something I would consider doing in the future), but homebrew uses bintray to host pre-compiled bottles so bintray is their effective CDN.

The takeaways from that thread are:

• no action should be necessary for gofish install since it doesn't rely on git
• gofish update and gofish upgrade are less frequent operations, so we should be well within a reasonable hit rate against github, though perhaps mirroring repositories may not be a bad option/contingency plan
• we should continue to perform a full fetch on gofish init/gofish rig add because

It is usually preferable to pay the price of a full clone once, then incrementally fetch into the repository, because then Git is better able to negotiate the minimum set of changes that have to be transferred to bring the clone up to date.

• figure out a better indexing option for fish-food similar to cargo, as that appears to be the more efficient option for git (unverified, but I'll trust his perspective) 😄

[bacongobbler]

It looks like the original issue around traversing a directory with thousands of files on github's UI has since been resolved. Moving the rest of the repository discussion over to #130. Thanks @mattfarina for your suggestions!