Virtual IPFS
Nebulus is an IPFS compatible file system that lets you work privately and locally while preserving all the authenticity traits of IPFS, as well as providing the ability to synchronize with the IPFS network.
Nebulus makes use of the encoding scheme of IPFS to represent files, without having to use the IPFS network.
You no longer have to keep running an IPFS node to work with IPFS files. Simply work offline and publish to IPFS only when needed.
Nebulus lets you manage files in a manner identical to how files are stored on the public IPFS network, but privately.
Instead of thinking of IPFS as just a "storage system", we can use IPFS as a data packet for private communication between multiple parties.
This means you can store data in Nebulus, and directly send the data over any network transport mechanism without making it public.
For example, we can think of a Nebula server that listens to IPFS messages. You don't have to use the IPFS DHT. Alice can directly post an IPFS data packet to Bob's server, privately.
Instead of having to share everything on the public IPFS network, Nebulus computes the IPFS hash (CID) of the files, locally.
Once the hash is calculated, a symbolic link is created. The symbolic link points to the original file, and has the file name of the calculated IPFS hash.
We have two folders above:
- src: the source folder for the application
- ipfs: automatically generated folder for Nebulus for maintaining the IPFS bindings
Note that all files under the ipfs
folder are symbolic links pointing to other locations, some of which include the files under the src
folder.
Nebulus is an "Offline First" version of IPFS. Basically it lets you work with IPFS without having to share everything on the IPFS public network.
IPFS is essentially a bundle of two things:
- Content Addressable Storage
- Peer to Peer Network
This means, to use IPFS you need to publish everything to the public network.
With Nebulus, the network and the file system are unbundled and you can use IPFS privately, without having to publish everything to the public IPFS network.
Instead of publishing everything to the public IPFS network immediately, you can work with IPFS files locally, and publish later:
This is similar to how Git works (compared to centralized version control systems like SVN, where everyone needs to publish immediately to the central repository to take advantage of the version control features):
When you use IPFS directly, ipfs.add()
always adds both to local and remote (public IPFS network) since there is no distinction between local and remote:
// adding directly to IPFS means it's immediately shared on the public IPFS network
await ipfs.add(...)
However with Nebulus, you now have the option to use Nebulus as an offline buffer. You can work privately and only publish to IPFS when needed:
// 1. Privately add an IPFS file to the local file system
let cid = await nebulus.add(Buffer.from("Hello world"))
// 2. "Push" to the public IPFS network
nebulus.push(cid)
Essentially, Nebulus unbundles the IPFS file format from the IPFS network.
This unbundling means more flexibility. For example you can use other network transport protocols to replicate your IPFS files (such as HTTP, WebRTC, Hypercore, etc.)
Sometimes you may want to openly publish a hash of a file before revealing the contents of the file. This way you can prove later that you had that file at that point in time. You can use Nebulus for this.
This property can be used for various use cases such as:
Create a Nebulus file, share its hash with someone over any channel to prove it existed at certain point in time (email, messaging apps, social media), and later reveal content.
Use the IPFS encoding format to store files privately and timestamp on the blockchain (like Opentimestamps) without revealing the contents.
Create an NFT collection that does not reveal its contents initially, but later you can easily "upload" to IPFS with one line of code.
You may want to work with a whole archive of files that you'll publish to IPFS eventually.
For example, you may have a folder structure that looks like this:
/
index.html
avatar.png
item.html
If you want to reference item.html
from index.html
using IPFS CID, you will have to keep updating the URL reference whenever you change the contents of item.html
. Same goes for avatar.png
.
And because updating IPFS CIDS means publishing to IPFS, you have no choice but to make every version of your files public whenever you update.
With Nebulus you can do everything locally without publishing.
Nebulus provides a new way to store and serve IPFS files privately.
Instead of serving your files to the public through the IPFS network, you can serve them privately to those who have permission to access the data.
Because Nebulus has unbundled the IPFS encoding format from the IPFS network, we can go further and ONLY use the IPFS for its encoding format.
You can use IPFS only as a data packet for communicating between parties, instead of thinking of it as a public storage. For example,
- Use Nebulus to create a file with IPFS CID filename, and attach it in a private email or private message
- A server for posting IPFS files privately.
Operate IPFS without running an IPFS node
Sometimes you may only want to use the public IPFS network as a way to replicate your file once, but you may not care about the permance of the file.
To be more precise, you may want to use the IPFS network as a replacement for HTTP POST
, instead of thinking of IPFS as a persistent storage.
Here's an example workflow:
- Privately work with files on Nebulus
- Publish to the public IPFS network
- Wait for another node to pin or replicate the files
- Stop seeding
You can use the push
event to achieve this:
await nebulus.connect()
nebulus.on("push", (cid) => {
// do something here
})
Because the push
event fires when a CID is successfully found on an IPFS gateway (ipfs.io), you can be sure that it will be pinned for at least a while.
You can take advantage of this feature when you are certain that whoever the file is intended for will pick it up eventually as long as it's discovered in the gateway.
This way you don't have to run your own IPFS node.
npm install nebulus
const Nebulus = require('nebulus');
const nebulus = new Nebulus()
const run = async () => {
const buffer = Buffer.from("hello world")
const cid = await nebulus.add(buffer)
console.log("cid", cid)
}
run()
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
const run = async () => {
await fs.promises.writeFile(__dirname + "/fixture/hello.txt", "hello world")
const cid = await nebulus.add(__dirname + "/fixture/hello.txt")
console.log("cid", cid)
}
run()
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
const run = async () => {
const cid = await nebulus.download("https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e")
console.log("cid", cid)
}
run()
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
const run = async () => {
const files = [
await nebulus.add("https://raw.githubusercontent.com/skogard/rarepress.js/0ad35d7da27a4d1e5f990a3bcf301e3fa9bae7ec/README.md"),
await nebulus.add("https://raw.githubusercontent.com/skogard/rarepress.js/0ad35d7da27a4d1e5f990a3bcf301e3fa9bae7ec/index.js"),
await nebulus.add("https://raw.githubusercontent.com/skogard/rarepress.js/0ad35d7da27a4d1e5f990a3bcf301e3fa9bae7ec/package.json"),
await nebulus.add("https://github.com/skogard/rarepress.js/raw/0ad35d7da27a4d1e5f990a3bcf301e3fa9bae7ec/press.png")
]
let cid = await nebulus.folder({
"readme.md": files[0],
"index.js": files[1],
"package.json": files[2],
"press.png": files[3]
})
let files = await fs.promises.readdir("storage/ipfs/" + cid)
cnosole.log("files", files)
}
run()
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
const run = async (cid) => {
await nebulus.connect()
nebulus.on("pull", (pulled_cid) => {
console.log("pulled cid", pulled_cid)
fs.createReadWtream("storage/ipfs/" + pulled_cid).pipe(process.stdout)
})
nebulus.pull(cid)
}
const cid = "bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"
run(cid)
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
const run = async (cid) => {
const buffer = Buffer.from("never gonna give you up")
const cid = await nebulus.add(buffer)
nebulus.on("push", (pushed_cid) => {
// check the following pushed URL in the browser
console.log("https://ipfs.io/ipfs/" + pushed_cid)
})
nebulus.push(cid)
}
add
: Add to IPFSdownload
: Download web file into IPFSfolder
: Create a folder from IPFS CIDsget
: Get file contents by CIDstream
: Get file stream by CIDconnect
: initialize and connect to the public IPFS networkdisconnect
: stop and disconnect from the public IPFS networkpush
: Publish to the global IPFS network, wait till it's replicated to public IPFS gateways, and trigger "push" event.pull
: Pull a CID from the IPFS network into Nebulus. trigger "pull" event when download is complete.
You can initialize a Nebulus instance using a constructor:
const Nebulus = require('nebulus')
const nebulus = new Nebulus(<options>)
Where <options>
can have the following attributes:
path
: storage path. if left out, it's.nebulus
(optional)max
: max file size in MB. If left out, no max limit (optional)config
: The IPFS config object for customizing Nebulus IPFS node. You can use this to run multiple Nebulus nodes on one machine without port conflict.
The path
is where the Nebulus file system will be constructed. For example:
const Nebulus = require('nebulus')
const nebulus = new Nebulus({ path: "storage" })
will create a folder named storage
in the current directory, and store all the files there.
If the path
is left empty, it will create and use a hidden folder named .nebulus
. For example:
const Nebulus = require('nebulus')
const nebulus = new Nebulus()
Will create a .nebulus
folder in the current execution folder with unlimited file size.
Also, you can use the max
attribute to limit the max file size:
const Nebulus = require('nebulus')
const nebulus = new Nebulus({max: 100})
will create a .nebulus
folder and allow up to 100MB file storage.
Finally, you can pass in a config
object to customize the IPFS node:
const Nebulus = require('nebulus')
const nebulus = new Nebulus({
config: {
Addresses: {
Swarm: [
"/ip4/0.0.0.0/tcp/4030",
"/ip4/127.0.0.1/tcp/4031/ws"
],
API: "/ip4/127.0.0.1/tcp/5030",
Gateway: "/ip4/127.0.0.1/tcp/9100"
}
}
})
Add local data to Nebulus.
const buffer = Buffer.from("hello world")
const cid = await nebulus.add(buffer)
await fs.promises.writeFile(__dirname + "/fixture/hello.txt", "hello world")
const cid = await nebulus.add(__dirname + "/fixture/hello.txt")
Download external web files to Nebulus.
const cid = await nebulus.download("https://thisartworkdoesnotexist.com")
const cid = await nebulus.download("https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e")
Create a depth-1 folder (Nested folders not yet supported)
const file_cids = [
await nebulus.add(__dirname + "/fixture/aperank/aperank.png"),
await nebulus.add(__dirname + "/fixture/aperank/readme.md"),
await nebulus.add(__dirname + "/fixture/aperank/index.js"),
await nebulus.add(__dirname + "/fixture/aperank/package.json")
]
let root_cid = await nebulus.folder({
"aperank.png": cids[0],
"readme.md": cids[1],
"index.js": cids[2],
"package.json": cids[3],
})
let cid = await nebulus.folder({
"aperank.png": await nebulus.add(__dirname + "/fixture/aperank/aperank.png"),
"readme.md": await nebulus.add(__dirname + "/fixture/aperank/readme.md"),
"index.js": await nebulus.add(__dirname + "/fixture/aperank/index.js"),
"package.json": await nebulus.add(__dirname + "/fixture/aperank/package.json")
})
Get buffer by CID
let buf = await nebulus.get("bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e")
Get file stream by CID
let stream = nebulus.stream("bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e")
stream.pipe(process.stdout)
interact with the public global IPFS network.
Initializing IPFS node.
You must first initialize the node before doing anything.
await nebulus.connect()
Stop IPFS node
await nebulus.disconnect()
Publishing one or more local Nebulus CIDs to the global IPFS network:
await nebulus.connect()
nebulus.push(<cid>)
Triggers a "push" event when the file is successfully replicated to IPFS gateways
await nebulus.connect()
nebulus.on("push", (cid) => {
// do something here
})
nebulus.push(cid)
pull files from the public IPFS network to Nebulus
await nebulus.connect()
nebulus.pull(cid)
Emits a "pull" event when pull is complete
await nebulus.connect()
nebulus.on("pull", (cid) => {
// do something
})
nebulus.pull(cid)
You can use on()
to listen to events. Currently supported events:
pull
: for when callingnebulus.pull()
push
: for when callingnebulus.push()
You can either listen to a global event, or a filtered CID event.
- Global: A global event may be useful when you use Nebulus as a daemon and want to get all the CIDs being pushed or pulled.
- Filtered: If you want a one-off event handler for a specific CID, you can use the filtered event
Push event:
await nebulus.connect()
nebulus.on("push", (cid) => {
// do something here
})
nebulus.push(cid)
Pull event
await nebulus.connect()
nebulus.on("pull", (cid) => {
// do something
})
nebulus.pull(cid)
Filtered events let you listen for a specific event. The event handler only gets triggered once and gets destructed afterwards.
The event looks like this:
- push:
push:<cid>
- pull:
pull:<cid>
Here's an example code for capturing a CID push event:
let cid = await nebulus.add(Buffer.from("hello world"))
let event = "push:" + cid
await nebulus.connect()
nebulus.on(event, (cid) => {
// do something here
})
nebulus.push(cid)
Here's an example code for capturing a CID pull event:
let event = "pull:" + cid
await nebulus.connect()
nebulus.on(event, (cid) => {
// do something
})
nebulus.pull(cid)