Offline support

[Rich-Harris]

If we can't fetch refs because we're offline, it should still be possible to use a cached archive. Right now they're only stored by commit hash which is unhelpful — we could store some metadata alongside the tar files, indicating that (for example) 1234abcd is the latest master.

[lukeed]

At the risk of plugging shamefully (😅) my recent Gittar can solve this issue, along with #1 and #3.

The module does not ship with a CLI, so there's no overlap in that regard.

A few key differences:

1. All archives will live in ~/.gittar, and cannot be configured.

2. it doesn't lookup or store archives according to the releases' full refs. IMO, this a bit overkill (at least in my use cases), since I'm far more likely to download specific tags or branches.

   • The master branch is always refetched, if online. Otherwise it will revert to the last master.tar.gz file it cached. Since that's probably the only scenario for caring about the more-recent commits, Gittar fetches anew by default.

And with the included tar package, it's fairly easy to decide how/when to keep/trash/compare metadata. Any options to gittar.extract are passed directly to tar.extract, so all the magic happens there. This inches towards a solution for #9.

Let me know what you think. Happy to start working on integrating this into a PR if you'd like. No hard feelings if you don't want to go this route. 😃

[Rich-Harris]

Nice 👏 Since gittar would be doing most of the heavy lifting, what if we just merged these projects? In other words, have the CLI and the library in the same repo. If you reckon that's a good idea, then I could send you a PR to add a gittar CLI?

It occurs to me that master might not be quite such a special case — you could imagine a project template that had different branches that were maintained in parallel (e.g. a React project template that had a preact branch). So I'd advocate for always going network-first, whether that means redownloading or just checking to see if the hash for a given ref has changed — though I realise that adds a decent amount of complexity and isn't something we'd need to solve straight away.

Let me know what you think!

[lukeed]

Hey @Rich-Harris, thanks!

I want to keep the API separate from the CLI -- I kinda like that approach. Even tho mri is super lightweight (I should know 😜), having the API separate enforces a build-for-reuse as opposed to building with a complete project in mind, y'know?

I totally understand that branch example. That's why I made sure to add a force:true option for the fetch.

Originally, I wanted to decipher between a semver-tag & a branch-tag, but in the real world (unfortunately) not everyone is using semver, or necessarily even marks releases with a numerical system. Doing this would allow all semver(-like) tags to be cache-first & the rest (master, preact, feature-foo) be network first.

[Rich-Harris]

The more I think about it, the more I think we have slightly different goals. You want offline-first, I want network-first, but I don't want to redownload tar files that I already have (so force: true doesn't really help). So maybe it's better we keep these projects separate after all.

[lukeed]

Okay! No problem 😃

While I do prioritize offline-first, I still want to be flexible.

Would it help if I added an ~onParse option that decided whether or not to send a network request?

The default would return true for master branches:

opts.onParse = ({ site, repo, branch }) => branch === 'master';

if (opts.force || opts.onParse(info)) {
  // send network
} else { ... }

[Rich-Harris]

It wouldn't quite solve the problem of only wanting to download if the commit hash had changed for a given ref. In my testing, fetching refs is usually quite a bit quicker than redownloading (though obviously it depends on the size of the repo).

[lukeed]

Gotcha. Well if you have any suggestions, feel free to send 'em my way. This was borne out of what I needed, but I'd like it to be flexible for others too.

Thanks!