I opened a preliminary PR (#10) for Federation but probably best to go via a issue first, to better enable discussions around it. Here is what I've been thinking so far.
Open-Registry as a crowdfunded registry won't be able to reach the same scale
The model of federation proposed here will decentralize the storage and
Once implemented and used, we can start focusing on research about federated
Ok, so the working plan is the following:
This is the small, MVP version to ensure the idea is viable in the wild.
First step towards federation is having the metadata index centralized with
Plan is to use ipfs-lite by @hsanjuan to start a embedded libp2p node that will
The software will connect to the central registry to find out the latest root
The root hash can be found in multiple different ways, depending on the
The software will basically be a resolver for (packageName, packageVersion) =>
Pointing your package manager to
When the federation software gets started on the users device, it connects to
Once connection has been established, it asks for the latest version of the
Concurrently, it starts a HTTP server locally.
Now the user can point it's client to the local HTTP server
Requests will be proxied via the latest root hash the federation software knows
When the root hash of the main registry changes, it publishes it via the
If the local client makes a request for a package that doesn't exists in the
First step of the federation setup is creating a suitable testing environment
Simulator should start with running the following scenarios:
More elaborate schemes can be created in the future.
Open-Registry will run a couple of bootstrap nodes. These are responsible for
Both the bootstrap nodes and the main registry index should publish metrics in
For the federation nodes, we can offer opt-in metrics in the future, so we can
Existing infrastructure migration
The current Open-Registry is just one instance which is the main Open-Registry
The text was updated successfully, but these errors were encountered:
Wonder how far we can get with cloudflare + cloud storage.
Will be experimenting with this in the coming weeks and will report back :-)
@retrohacker thanks, appreciate it, ping here once you have some results to share :)
That said, I do think that even if we find the fastest CDN, we can make it faster for people by having a federated model. But CDN in front of the metadata registry would still be a good idea.
I've updated the initial issue here with an updated version of the proposed federation, will also bump it on the roadmap.
Old proposal can be found here: https://gist.github.com/victorb/82ace9e6fe7adf578833527b8b94f914
@victorb as promised, circling back to report on cost.
Self hosting on cloud providers turned out to be reasonable. Our GCP mirror ended up costing ~$300 to do the initial mirroring (pulling 5TB of data through cloud functions and into cloud storage).
Once the files are sitting in storage (multi-zone within the US), the cost is ~$6 a day. That includes the instance that is sitting there watching the CouchDB stream from the npm registry to keep the mirror fresh. The breakdown is $3.46 per day for storage and $2.28 for the compute instance.
Cloudflare functions (where we are doing our load balancing) costs $0.50 per million invocations.
BTW the service is up and running if you want to give it a try: https://freajs.com