Integration with crystalshards

[zamith]

This would be super helpful for crystalshards, since it would make shard discovery super easy, opposed to searching for crystal project using GitHub's API, which is what we're doing right now.

I had mentioned to @asterite that at some point it would be good to have some sort of registry, if for nothing else, for the metadata of projects, so you can better and easily analyse the libs landscape.

Do you plan on storing the binaries for each version, eventually? Will I have to register a shard somewhere (I'm assuming yes)?

[ysbaddaden]

I've started thinking about the registry design. I think I'd like something between bower and rubygems. I mean to keep the metadata and let the actual code live in their GIT repositories — of alternative SCM.

The metadata being the shard.yml data (name, repository URL, dependencies, authors and license) for each available version.

Also maybe add webhooks to the repositories, so pushing a new tag would have the registry update itself. That would be nice.

[asterite]

With @waj we were thinking how to have a decentralized approach by maybe leaving a shard's metadata in a special branch in the repository. That metadata would contain a special shard.yml (or another bigger file) that contains dependency information for all versions up to the last one, like what Rubygems has. Because it would be in a separate branch, checking it out should be fast so we wouldn't need to checkout each tag to know the dependencies to resolve conflicts.

The language/compiler could be aware of this and have commands to push new versions by creating tags in the repo and updating the special branch metadata.

[asterite]

As for @zamith's concern about a registry, maybe we can have the registry just be a website that scans GitHub/BitBucket for crystal-related projects that have that special branch with the version information, and cache that information. Then the compiler would allow you to search libraries and add them to your project.

(when I say "the compiler" it could be a related tool, but maybe it's simpler to just have one entry point, like "crystal search", "crystal install", etc.)

[ysbaddaden]

git show v0.1.0:shard.yml cats a file for the given refs. This is quick, yet still requires an up-to-date local mirror/clone of the repository (I don't think it applies on remote repositories).

[zamith]

The problem with searching GitHub is that it only searches for projects in which the main language is Crystal. So, if you have a shard that has more of another language, it won't show up.

[ysbaddaden]

Note that a registry + decentralized aren't mutually exclusive.

Some operations like search or info will require the registry, while some operations like install or update could take advantage of the registry (or not) and fallback to clone repositories and extract the metadata from it. To install the dependencies we'll most certainly have to clone the repositories anyway.

[zamith]

I like @ysbaddaden idea of the hybrid registry.