Deploying on academic clusters + utilization by job schedulers

[lovelaced]

As briefly mentioned in IRC, I'd be very interested in how IPFS can perform in academic clusters running job schedulers such as HTCondor (used in many facilities such as CERN) as an alternative or addition to currently implemented filesystems such as those that are relatively complex to set up (such as AFS).

An interesting use case for it is the OSG, which HTCondor and other job schedulers schedule to as a shared resource between academic institutions and could potentially very adventageously utilize IPFS for large input files.

I would be very interested in testing the performance of IPFS versus other distributed filesystems that are currently in use and by itself to potentially better user experience when attempting to access (very) large files from a shared filesystem (and in general -- not for specfically large files, though that would be the immediate usecase).

[jbenet]

Indeed! We'd love to help set these clusters up and setup some nice ways to get benchmarks.

as a warning, IPFS right now will be bad perf in clusters. we've worked to optimize individual cases, but it would be a surprise to us if it meaningfully competed in raw performance. This is an optimization problem though so we can vastly improve this perf.

What would really help us is a nice way to:

• easily construct large networks (1000 nodes+, distributed over N physical machines). We need programs like iptb but across machines). maybe we can do this with the new hashicorp deployment stuff. (otto/nomad?)
• modify go-ipfs to output a lots more useful stats
• a way to aggregate all the stats and generate great perf graphs.

This is something we need to do, and which would be a huge contribution to the whole effort. The perf at first may be bad, dont be discouraged by this-- we have lots of room for protocol optimization. (Bitswap is very dumb right now, it's actually surprising ipfs works as well as it does considering we havent optimized it in a while yet)

[jbenet]

cc @whyrusleeping @lgierth @ion1 as they're interested in this too.

[whyrusleeping]

@mafintosh and I discussed something like this, and i've started work on making a set of scripts to help us simulate different types of NAT in test environments.

[lovelaced]

So are you thinking something like ceph-deploy eventually, so one could create a network of machines without a current software deployment solution?

Regarding use with job schedulers, would ipfs be more likely to run system-wide, then somehow allow users to 'tap into' the daemon - or would it make more sense for each user to run their own daemon within a slot? If not, how to handle port selection? What about sharing a daemon across slots (whom are owned by the same user)?