Trying to boil down a bunch of issues into major themes

[arschles]

I was going through some old issues today and noticed a bunch of them fall into two major categories:

1. Storage drivers
2. Offline mode

I want to post this publicly, but not before I get some feedback and edits. What better place to do it than here!

Storage Drivers

This one is fairly easy to categorize. There are some issues that flat out ask for more storage drivers, some PRs that have tried it, or some issues that storage drivers might fix:

• Plugin architecture for storages #1110
• Implement storage.Backend for Artifactory storage #1130
• Prototype: Implement new storage backend http #1131
• Use cache before running "go list" #1511 (see Use cache before running "go list" #1511 (comment) for background why)
• Add ability to remove modules from the storage #1004 is about removing modules, which might be better done on a storage backend rather than through an Athens API
• pkg/mod storage backend #1459 is asking for a disk-based storage backend that just reads an existing $GOPATH/pkg/mod directory
• [question] What is use of @Delete method in Storage interface? #1547 talks about our Delete method in the storage interface
• Add a configuration for Athens to proxy to upstream proxies #1488 is a pretty generic one about creating a download mode entry for proxying to upstream proxies. we might be better off doing this as a storage driver for the standard module download API

Offline Mode

This one is more nuanced, and also related to storage drivers. Athens has always tried to function while cut off from the internet. That's easier said than done since the proxy protocol has a mix of immutable and immutable endpoints. And adding the checksum DB on top makes it even more difficult.

Right now, Athens can run air-gapped if storage is full of all module versions that someone has in their go.sum. That's a common case for most teams. Lucky for us, if a team doesn't do a go get -u, or any other command that hits a mutable proxy endpoint, then Athens won't have to reach out to the internet for mutable module metadata or checksum information.

Over time, people have wanted to start doing those things though. Not much work has started on making a full "offline mode", so there are mostly issues.

• In Availability vs. correctness in list API #1532 we saw a real use case from Uber. The discussion deals directly with how to handle immutable endpoints when Athens is offline or upstream repos changed/disappeared
• Use Athens as repository in private (offline) network #1506 stalled out a bit but it's another use case. We talk about pre-filling storage and the checksum DB, both in offline environments
• in Use cache before running "go list" #1511 the same Uber engineer talked specifically about the /list API. It's almost a subset of the Availability vs. correctness in list API #1532 discussion, but it has some of Uber's needs and use cases, and that leads into custom storage drivers as well
• could athens support mirror? #1315 talks about server-side aliases. this one is tangentially related, but we could support it in theory through pre-loading
• 404 from @latest and @v/list when pre-filling the disk cache #1416 talks some more about pre-loading with pacmod
• Support for running athens behind a company proxy #1534 is about supporting a company HTTP proxy. we don't know what kind yet
• Proxying sumdb behind a proxy or upstream #1468 is specifically about proxying and/or caching the sum DB
• There's Athens in Offline Mode #1538 of course 😄

Sidenote: Uber has a fork of Athens. They've given us a few good use cases and seem willing to get off their fork if we make some of these changes anyway

That's it

It's late so I didn't have time to make this more coherent. I think that we have enough use cases and desire to build in some kind of support for storage drivers.

Offline mode seems amorphous right now, but I think we're close to having a good idea what we need to do. When we have it, I'm sensing that it'll be a popular feature.

I'm looking forward to hearing what you all think!

[arschles]

Update! Almost all of our external storage stuff is done with #1587 (thanks @marwan-at-work 😄 ). Let's move on to offline mode support