Use IPFS as a decentralized hosting platform? #9

Open
reimertz opened this Issue Mar 24, 2016 · 23 comments

Projects

None yet

10 participants

@reimertz
Member

IPFS -InterPlanetary File System seems to have the solutions for

  • decentralized hosting
  • package validation
  • probably a lot more

Ey, @jbenet (and Hi @jbenet ! 😄 ), do you think IPFS would be a good fit for this initiative?

@noffle
noffle commented Mar 24, 2016

IPFS could definitely do the job: it's all about immutability. <3

Check out https://github.com/whyrusleeping/gx, an IPFS-powered lang-agnostic package manager. We'd love a gx-node implementation -- maybe we could collaborate?

@LukeeeeBennett
Member

@noffle Definitely down for building a gx-node implementation. Just reading through gx-go now.

@reimertz
Member

Interesting. What do you think @LukeeeeBennett, does this make ipm obsolete? Should we just focus on helping these guys out?

@reimertz
Member

And hi @noffle and thanks for the intros <3

@whyrusleeping

Hey @reimertz and @LukeeeeBennett I'd love to see a gx-node or gx-js tool :) we should chat sometime.

@LukeeeeBennett
Member

@reimertz If you'd like ipm to be a distributed repo management system as we've outlined in some areas of discussion, then I think going forth with gx is a brilliant idea. IPFS looks like the perfect protocol to use for this and gx is already sitting there ready to be extended.

I've not had as much Golang experience as many but from what I've read from the current gx and gx-go source code, it's all logical and clean code. Yet to read the IPFS origins paper which also looks great.
Most if not all of the network logic is handled by the core gx client which exposes a set of fuss-free git-style hooks to extend functionality for language-specific repo drivers, there's a little bit to build out to get a gx-node on the go but I think it'll be a blast.

However, that may not mean you need to ditch ipm, from what I've seen there are still a few contributors wanting to get involved in some alternative repo management architectures and I'd hate to storm in here with this distributed network nonsense and take the whole project away from others, even if I think a completely immutable package management system without the support of a distributed network is pretty much impossible and/or potentially a bad idea as it creates many more issues to fix. Whether you maintain it or transfer ownership, I'm sure someone would like to do something else here. It's all the buzz!

@whyrusleeping @noffle I think there is a general agreement on going forth with gx-node.
If someone wants to grab the repo, we can start discussions there.

gx init --lang=node
Coming soon to an IPFS node near you.

@xzer
xzer commented Mar 25, 2016

I do not think the decentralized mechanism is good enough, in fact, sometimes the developer need the help from the organization who are operating the repository against the company's lawyer, which is why we are disappointed to NPM.

@AlexanderOMara

One of the neat features of NPM is that it is itself a Node module. You can extend and build off of it if you want, with Node itself. If IPM is not itself a JavaScript Node module, it would be nice if it could still be extended by Node modules somehow.

@jamen
jamen commented Mar 25, 2016

How does IPFS fare on reliability?

I have lots of packages that I (as a 15 year old) could not host anywhere. No money or time (with school) to set something like that up... So with someone like me, would I need to setup some local server just to "publish" my packages unreliably?... Not something I particularly want.

I have no knowledge of IPFS, so try to bare with me.

@jamen
jamen commented Mar 25, 2016

Another thing to bare in mind... If the user serves up their own packages (decentralized), what goes to say that they cannot modify (or in left-pad's case, unpublish) the "published" packages in an illegal way?... Basically destroying the immutability aspect, if they can...

@xzer
xzer commented Mar 25, 2016

@jamen I am not professional about the distributed share mechanism, but if you can trust BitTorrent or edonkey, you can trust there are enough measures to guarantee the immutability.

@reimertz
Member

@LukeeeeBennett I agree with you, I think implementing this using gx is the way to go long-term.
I created a branch called gx-node under this organization for now if you'd like to continue the discussion of using their implementation. I'd make you a owner of that repo. :)

For short-term though, it seems that the majority of people here would like the following things:

  • Javascript implementation
  • 100% support for current npm implementation
  • non-profit organization
  • open-sourced
  • namespacing
  • non-profit organization backing for legal issues
  • almost immutable -> defaults to immutable, if there are legit legal issues, a package should be able
    to get revoked after proper legal actions has been taken through above organization.

People, please correct me if I am wrong or missed something.

I wonder if there is a quick way of wrapping the npm cli and extend this functionality?

@xzer
xzer commented Mar 25, 2016

private repository should be possible.

@LukeeeeBennett
Member

@reimertz Indeed it sounds like you've got a significant amount of support for a JS implementation here that aims to tackle stability and transparency. Thanks for setting the repo.
@jamen IPFS champions a number of protocol-level security features adopted and adapted from distributed transaction technologies like bittorrent and bitcoin that help deal with attack on/from any nodes in the network. Take a look at the IPFS origins paper. If you're interested, pop over to gx-node.

@noffle
noffle commented Mar 25, 2016

I don't think the development of e.g. gx-node precludes or makes obsolete IPM: gx and gx-node could be under-the-covers dependencies of IPM that provide the backbone and immutability semantics desired, while IPM itself has the UI/UX originally imagined, and whatever other features gx-node might not make sense to have.

How does IPFS fare on reliability?

Great question!

Like any system, storage of bits and retrieval of bits costs energy (in the physical sense). IPFS provides the ability to find your content, given someone on the network has it, but it does not guarantee that an IPFS node will always have your data -- that's a tricky property to guarantee, given the physical energy cost involved.

Said differently, the real problem is "how can I make sure there are always IPFS nodes around that have my content pinned?". We've thought about this a lot! Check out this issue -- @jbenet has some good comments on efforts related to this.

what goes to say that they cannot modify the "published" packages in an illegal way?

There are cryptographic means to make this infeasible, much like how modifying the Bitcoin ledger is highly infeasible. (Recommend doing some reading on Bitcoin to understand why it's so resiliant to being hacked.)

I wonder if there is a quick way of wrapping the npm cli and extend this functionality?

I think this is a really good vector to get results quickly. Rather than hacking on npm directly, I suggest writing a custom registry. The registry is the component that responds to requests from the npm CLI client. If you wrote a custom registry that handled immutable data structures and e.g. ignored unpublish requests, you could get an "immutable npm" today!

In fact, @diasdavid has already written https://github.com/diasdavid/registry-mirror, which is an NPM mirror that uses IPFS for storage.

Telling npm to use a custom registry is as simple as

$ npm install foobar --registry=http://localhost:9000
@kazzkiq
kazzkiq commented Mar 26, 2016

@noffle What about Ethereum? I'm just on babysteps into the blockchain world, but their iniciative seems pretty much like what ipm would need for decentralized packages.

And the good thing is that (if I got the idea right) as Ethereum serves several purposes (and publics) ipm packages would be hosted by a large base of people across the world.

@AlexanderOMara

Might be worth a read:

nodejs/NG#29

Referenced as detailing a possible issue with this proposal here:

http://cryto.net/~joepie91/blog/2016/03/23/reflections-on-npm-gate-one-day-later/

Haven't read or thought enough of it to comment yet.

@LukeeeeBennett
Member

@AlexanderOMara - Excellent link. I suggest everyone has a read.

@VictorBjelkholm

There is a very simple way to use IPFS to install and publish dependencies. I'll copy paste some code from EverythingStays/everythingstays-cli

These are both scripts, one for preinstall and one for prepublish

preinstall


  MODULES=$(node -e "$DEPS_CMD")

  if [ $? -ne 0 ]; then
    echo "No esDependencies found in package.json"
    exit 0
  fi

  echo "Downloading dependencies..."

  while read -r line; do
      NAME=$(echo "$line" | cut -d ' ' -f 1)
      HASH=$(echo "$line" | cut -d ' ' -f 2)
      echo "Getting $NAME@$HASH"
      ipfs get $HASH --output node_modules/$NAME
done <<< "$MODULES"

Basically reads package.json for esDepedencies and then downloads them from there. Simple no?

prepublish

  mv node_modules .node_modules 2>/dev/null

  HASH=$(ipfs add . -r | tail -n 1 | cut -d ' ' -f 2)

  mv .node_modules node_modules 2>/dev/null
  echo $HASH

Even simpler, move node_modules temporary so we don't publish it, add the current folder and print that hash. It's not "published"

Maybe this can give you some inspiration for a simple solution

@LukeeeeBennett
Member

@VictorBjelkholm This is certainly the quickest way for decentralized deps and I aim to start promoting gx-node after this sort of level of logic is integrated with Go, I aim to get it moving on Wednesday. There is plenty more we can do beyond that though which is great. :)

@VictorBjelkholm

@LukeeeeBennett not sure why gx-node is needed if there is simpler ways of dealing with it. gx-node also seems to want to override the npm package.json, can you use npm and gx-node in one repository in the same time or you are limited to one of them?

@LukeeeeBennett
Member

@VictorBjelkholm gx-node is an extension for gx, so yes the end goals of gx-node and everythingstays-cli look the same, but that doesn't mean only one should be built. The different implementations will certainly offer different opportunities to each project as time goes on.
gx-node will be able to be used alongside npm in a single repository, gx is only concerned about the dependencies it should manage, just as npm is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment