Positioning, Vision and future direction of the Dat Project

[aschrijver]

NOTE This thread has been cleaned and moved to dat-ecosystem-archive/datproject-discussions#58

Alternatively jump to Executive summary on this page for an overview what is here.
.
.

I love the dat project! ❤️ And I think it is currently uniquely positioned in the IT landscape.

Designed primarily for distributing scientific documentation, but it could have much broader application.
I am thinking of a dat as a more generic application platform (the core of it, at least) to create decentralized social networks, and particularly well-suited for creating solutions for the emerging sharing economy.
Instead of file exchange, dat would be involved with message exchange, message collaboration and event sourcing.

The opportunity is now.

I've done my research, and found just the tiniest amount of viable technologies / competitors for decentralized solutions. It basically boiled down to this:

• Secure ScuttleBot: more popular than dat currently, but written in less accessible language
• IPFS: still maturing, but will be a big competitor to dat in future
• Replikativ: solves some issues dat has (read/write + consistency), but too small to succeed (IMHO)
• Blockchain-based stuffz: either too immature, or scams, and maybe not viable in the long run
• IRC-, torrent-, darknet-related: too specific, not suited as generic app platform
• Others: numerous abandoned and outdated efforts once started with the best intentions, great vision, but eventually stranded, interest lost

Plus dat project: Written in accessible js, and doing complex things with a (still) relatively small codebase....and here comes the but...

BUT:

From a marketing perspective the naming of the project and its components is just terrible, IMHO:

• dat: too generic, too short (e.g. means 'that' in Dutch)
• hypercore, hyperdrive: named after the hardware products of tech giants
• and any search term with decentralized, etc. leads mostly to blockchain stuff, you are snowed under..

You will never gain good visibility for the project with this naming scheme. And not only visibility, but also traction. A (aspiring) dat developer will have a hard time finding resources, seeing what the community is doing, etc.

Maybe I am ranting, but I think proper naming is essential for the long-term success of the project. I am afraid that without it, you'll risk ending up in that Other category once current people lose interest in a few years.

Keep the ball rolling, you are doing great work!

PS It would also be great if all dat-related project dependencies (e.g. hypercore, hyperdrive) were under the same umbrella, even though still usable independently of each other.
PS2 I cross-posted to replikativ project to make discussion more interesting: replikativ/replikativ#8

[blahah]

Personally I think the naming is not a problem. Right now, dat does not need more growth than it is seeing - people are finding it organically and building infrastructure, tools and systems around it. Detailed naming of the modules etc. is mostly about being able to find them once you're already in the ecosystem, but the ideal way for people to engage is to come and ask questions. If everything were much easier to discover it would lower the barrier to entry which, right now, would probably lead to a huge increase in support requests and decrease in general progress.

The layer of things above dat - tools and platforms built around it - will include things that make dat and the ecosystem of tools more discoverable and accessible imo.

[yoshuawuyts]

Perhaps we should use "dat protocol" more often - ranks #1 for its search thingy https://duckduckgo.com/?q=dat+protocol&t=hq&ia=web

…

On Fri, Jul 14, 2017 at 11:57 AM Richard Smith-Unna < ***@***.***> wrote: Personally I think the naming is not a problem. Right now, dat does not need more growth than it is seeing - people are finding it organically and building infrastructure, tools and systems around it. Detailed naming of the modules etc. is mostly about being able to find them once you're already in the ecosystem, but the ideal way for people to engage is to come and ask questions. If everything were much easier to discover it would lower the barrier to entry which, right now, would probably lead to a huge increase in support requests and decrease in general progress. The layer of things above dat - tools and platforms built around it - will include things that make dat and the ecosystem of tools more discoverable and accessible imo. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#824 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACWlelDZujDjnONxkUd8LmnYXQhZxmQIks5sNzuegaJpZM4OX433> .

[aschrijver]

I guess underlying my naming objection is the question how ambitious the dat project is? Dat could be one of the large projects on github, with much useful development and good community steering..

[aschrijver]

Some interesting discussion on the #dat irc on the topic:

[12:03] <barnie> you are currently about 4 active developers, am I right?
[12:03] <barnie> isn't that a bit light?
[12:03] <barnie> for such groundbreaking tech?
[12:03] <barnie> or should dat always be small, targeted to science community?
[12:07] <blahah> barnie: I think probably about 50 people developing things with dat right now
[12:07] <blahah> at a conservative estimate
[12:08] <blahah> but distributed around lots of projects and repos, so not easy to measure
[12:09] <barnie> yes, there you have it.
[12:09] <barnie> distributed, dispersed
[12:09] <blahah> but that's OK, it's how it should be
[12:10] <barnie> but looking at main dat project commits 4 core devs
[12:10] <blahah> we don't all need to know each other - the community and the technology reflect the philosophy
[12:10] <blahah> a small number of committers is a sign of stable repo with small scope imo
[12:11] <blahah>  in this case at least - there could be other factors in mnay situations
[12:11] <barnie> its not so much about knowing, but finding resources, being productive, etc.
[12:11] <blahah> barnie I think a lot of people's contributing to dat things will  be at the higher level, beaker browser, sciencefair etc
[12:12] <blahah> and ideally higher level than that
[12:12] <barnie> so many similar projects have failed using this philosophy
[12:12] <blahah> well, yes, but the same is true for the alternative - most projects of all kinds fail
[12:12] <barnie> especially in this new field
[12:12] <barnie> true
[12:13] <blahah> I agree that making it easier to understand or find things is good
[12:14] <barnie> with such small set of core developers it is hard to develop stable base for all the complexity that is yet to be tackled
[12:14] <blahah> but it also has to grow with the project - I've seen a lot of projects fail because they attracted users/contributors faster than they could scale the technology or (human) support system
[12:14] <barnie> yes, that is true of react-native, I think, having tried to setup my first projects with it
[12:14] <blahah> barnie: I think that's the opposite of the case - the stability comes from having small modules that do one thing well and need very little maintenance
[12:15] <blahah> (referring to the complexity/stability issue)
[12:15] <barnie> but are you all 4 full-time available, or is dat a side-project
[12:15] <blahah> I don't work for dat, it's not even a side project
[12:15] <blahah> but I do make a project that depends on it
[12:16] <barnie> any job assignment can take you away for 2 years and the project may die
[12:16] <blahah> there is a dat core team of full-time people
[12:16] <barnie> ok, that's good
[12:16] <blahah> I am funded 50% of my time to work on the thing that depends on dat
[12:16] <blahah> and we are all working on longer term stable funding for dat and related projects
[12:17] <blahah> but until that is in place, trying to scale rapidly will more likely lead to failure than prevent it
[12:17] <barnie> btw, I am not critizising, just worried for your futur ;)
[12:17] <blahah> no problem, it's useful to discuss
[12:18] <blahah> for sciencefair, part of our sustainability plan is to provide a stable % of our funding for projects we depend on like dat
[12:19] <barnie> if, say, IPFS would quickly mature and dat users walk away to it, would you mind?
[12:19] <blahah> they aren't really the same thing, so it doesn't matter to me
[12:19] <blahah> I hope IPFS succeeds
[12:19] <barnie> me too, but IPFS is just an example, losing user base is the danger
[12:20] <blahah> well, the user base is likely to not even know about dat really
[12:20] <blahah> dat is a layer with a developer base, and will have users
[12:21] <blahah> but then a layer of stuff built on dat will have users that don't care whether it's dat or whatever underneath, but the features enabled by it will matter to them
[12:21] <blahah> so they can't just switch to IPFS unless someone builds the same thing on IPFS
[12:22] <barnie> yeah, that's my issue.
[12:22] <barnie> this leaves dat as entirely steered by the applications that grow on top of it
[12:23] <barnie> no knowledge upfront on the direction it 'll go
[12:23] <blahah> guided by the needs of the users, yes
[12:23] <blahah> that's necessarily true
[12:24] <blahah> but the stability again comes from the modular ecosystem
[12:24] <blahah> for example, sciencefair actually depends on hypercore and some other things, not dat directly
[12:24] <barnie> user guidance is good, but if one wants to start a big initiative, like the social platform I mentioned..
[12:24] <blahah> even if dat itself went in a different direction, the small modules that make it up would not - the new direction would come with developing and switching to new modules
[12:24] <barnie> ..its good to know more of its vision and future direction (generally)
[12:25] <blahah> so people depending on those things could carry on
[12:25] <barnie> yea
[12:26] <blahah> I would say that right now, the entire ecosystem including all p2p/distributed projects other than dat is not stable enough to guarantee you that features you depend on will be included in the project in a few years
[12:26] <blahah> that's just because it's nascent and exploratory by nature
[12:26] <barnie> that's exactly the unique opportunity dat now has
[12:26] <blahah> as it matures, the things that survive and don't will become clear naturally
[12:26] <barnie> too slow, I am afraid
[12:27] <blahah> it's much more right now to do with whether we can convince funders to provide stability
[12:27] <barnie> you'll end in the dustbin
[12:27] <blahah> well, that's the gamble we are all taking - we believe these things will work and succeed if we do them right, and are trying to make that happen
[12:28] <blahah> slow is good
[12:28] <barnie> hmm.
[12:28] <barnie> take replikativ for example
[12:28] <blahah> moving faster than the resources can support leads to failure imo
[12:28] <barnie> i find it fantastic what they mention in their site
[12:29] <barnie> but the latest project activity is 9 days old
[12:29] <barnie> nothing much to find with googling
[12:29] <barnie> i would not decide to integrate that in my solution
[12:29] <blahah> I agree, dat is like the opposite of replikativ
[12:30] <barnie> there are a lot of successful github projects having much more activity. Dat could be like that too
[12:30] <blahah> stable growth over time, spawning many stable, well-scoped modules as it goes, not outgrowing its resources or over-stating its claim
[12:30] <barnie> I totally agree with that tendency, but one does not rule out the other
[12:31] <blahah> github activity is not a measure of success :) it's a function of popularity, how big the codebase in a single repo is, and how buggy the thing is
[12:31] <barnie> no, i mean active + successful github projects. they exist
[12:31] <blahah> generally the most active things are hype-driven projects with huge corporate backing
[12:31] <barnie> like blockchain, ha ha I know what you mean
[12:31] <blahah> e.g. the facebook stuff like React, yarn, jest
[12:32] <barnie> yep
[12:32] <blahah> and blockchain
[12:32] <blahah> yup
[12:32] <blahah> that's unhealthy in every way
[12:32] <barnie> I have open issues with Jest :(
[12:32] <blahah> React will be obsolete in no time, same with jest and yarn
[12:32] <blahah> their success is not the kind dat would want
[12:32] <barnie> well, in general whole js community is going too fast now
[12:33] <barnie> agreed
[12:33] <blahah> carefully building out things that solve the problems incrementally and stably is the way
[12:33] <blahah> and avoiding commerical interests gaining control
[12:34] <barnie> that I fully, totally agree with
[12:34] <barnie> but that is also why many dat-like activities die
[12:34] <barnie> they don't have a good model/strategy to keep the community alive in the long run
[12:36] <barnie> blahah: do you mind if I copy the thread to the github issue?
[12:40] <barnie> btw, the whole commercial infuence on the internet should be a motivating driver to develop dat faster
[12:41] <barnie> internet freedoms are rapidly encrouched by private interests
[12:42] <barnie> a good decentralized application model would directly compete with the large IT moguls' products
[12:42] <barnie> governments in general also do not stimulate growth of decentralized solutions
[12:43] <barnie> bitcoin/blockchain doesn't make it easier for dat to flourish.

[joehand]

Thanks for starting the great discussion! It seems like there were a few separate points you expanded on in IRC:

• Discoverability/Visibility & Success
• Sustainability
• Community growth

I can address a few things on each of these points:

Discoverability & Success

As @blahah said in IRC, discoverability can be a good and bad thing. We need to be careful not to conflate success with various measures of visibility (e.g. being the top Github project). Two years from now, if we are using a module that hasn't been updated for two years - I'd say that module is successful regardless of it's popularity.

The other issue is that we do develop in a very modular approach. That this repository has only 4 core contributors means nothing for how many people have contributed to it's development or the community behind it. We can do a better job of highlighting this but it has not been a priority for us.

Finally, there are use cases we're interested in where popularity is a negative. Human rights defenders in countries with heavy surveillance need tools built on Dat or similar tech. But popularity will make it more likely Dat is blocked or measures put in place to make it more difficult to use (see bittorrent).

Sustainability

Sustainability is a major concern - both in how development is funded and how users use our tools. I'd agree, we do need to do a better job of being transparent about Dat's funding and how we plan to make it more sustainable. As a small team of focused developers we've let a few of the communications aspects slide. I've been trying to prioritize this more lately and hopefully will have some blog posts and other communication about this out soon.

Community Growth

We've been focused on growing Dat through development of (hopefully) funded applications built on Dat. We like when people building Dat or applications on Dat are being paid to do so whenever possible. If folks are contributing in their free time that is great, but it is not something we want to rely on. Frankly, if Dat were to explode in popularity with developers and have a bunch of contributors adding directly to this repository, we do not have the capacity to manage that.

We do want our community to grow but most of our work is eyeing users in academia, government, or journalism. We feel that by prioritizing interaction with these users, we can develop tools that have a large positive impact but are also generally useful in other applications.

Speaking for myself, these are the core properties of how we can be successful:

• Steady growth through funded application development that allows us to reach new users while maintaining underlying technology.
• Open and modular development that demonstrates how to build on the Dat protocol without prescribing solutions.
• Transparency around how Dat is funded, plans for future financial support, and developers doing important work on Dat for free that we need to help support.
• Community engagement that focuses on engaging users in industries that exist for public benefit, such as academia, government, journalism.

p.s. I usually recommend users search for dat project, dat data, or dat protocol, depending on their interests.

[aschrijver]

And thanks for your elaborate response :)

Discoverability & Success

I agree with your arguments here.

But popularity will make it more likely Dat is blocked or measures put in place to make it more difficult to use (see bittorrent).

Valid point. I hadn't thought about this being such an issue, but bittorrent proves it is.

As a small team of focused developers we've let a few of the communications aspects slide. I've been trying to prioritize this more lately and hopefully will have some blog posts and other communication about this out soon.

Who has not? If you have these blog posts ready I'd be happy to see if I can contribute albeit just my own 2cts worth.

We do want our community to grow but most of our work is eyeing users in academia, government, or journalism. We feel that by prioritizing interaction with these users, we can develop tools that have a large positive impact but are also generally useful in other applications.

Your current positioning is clear and has well-defined boundaries. My ideas evolve around local sharing economies, sustainability (social, economical, ecological) and sustainable development / growth (of the social networks), which make it an outlier in this sense but having the same high values as dat project (or its derivatives, like sciencefair).

Open and modular development that demonstrates how to build on the Dat protocol without prescribing solutions.

Yes, important point. Right now to me the project feels overly scattered and fragmented, and for many things you'll have to dive in the code instead of reading the doc (like finding all opts that you can pass hypercore as a parameter).
Of course this is that time issue, but with documentation the community could be a good help.

Ideally I could just take the protocol specs and write e.g. a Java port that is compatible with the core modules. Don't know how far dat is from this point. But I know in general getting to this point requires much effort.

[blahah]

@aschrijver you could implement https://github.com/mafintosh/hypercore in java right now (but for the love of all that is small and modular, I urge you to consider other ways to expend your energy 😄)

[aschrijver]

Ha ha thanks, I was not considering it. Was just examplary 😄

[aschrijver]

Hi @joehand @blahah

I understand the reasoning behind your current strategy, but I still think you miss out on a big opportunity. Allow me to elaborate..
.

Current positioning

Dat et al - in its current state - is actually a generic technology framework for creating decentralized solutions of any kind. But it is not presented nor positioned as such!
The landing page on datproject.org quickly states: "Powerful data sharing from your desktop" and later "Dat is the package manager for data. Share files with ..."

If I would want to use Dat for a different use case, say an event collaboration framework, and not distract you from your science community focus, then I would have to seriously untangle and recompose current modules, add different glue and logic. This because your focus is on file exchange, including hyperdrive, etc. No problem, but it would lead to extra work if later on you would like to incorporate the cool modules I have created.

I might have missed Dat Project altogether in my technology research given its application focus, just like I may have missed another viable candidate because it was positioned solely as an IRC app.

Broader vision

Broadening your horizon and calling it a technology framework (or platform, or whatever is fitting) and explicitly positioning as such will cost you nothing in terms of how the core modules are developed and the pace in which that occurs. You can still have the dedicated core team you have now.

And it would alleviate the feeling spin-off initiatives might have of being more or less out in the cold, taking big risk in diversifying from the overall direction.

On your landing page you would have to extract the application parts and place them somewhere separately, but more on that later..

Benefits

There are a number of benefits to be gained:

1. Spin-off (legitimate) decentralization initiatives that are using Dat have - by the nature of the technology - usually a similar set of high moral, ethical values and goals. While they can operate completely independently and not burden your team, they will also have an inclination to fund you, or make donations

2. These spin-offs will broaden the ecosystem and bring unexpected additions that are useful to your cause (e.g. in security, mobile, networking, social, deep learning, etc.)

3. More (useful) contributions to the core, like documentation, more contributors. More ability to delegate support to the community

4. Having a successful technology in the field of Decentralized Computing (instead of a successful application) will be much more of an incentive for newcomers, competing technologies and players to emerge, which is a healthy thing, leads to cross-pollination, more creativity, etc.

Current threats

That last point is especially important. As @joehand already pointed out there are unusual forces out there against the success of decentralized computing solutions, which you don't find in other technology areas. Besides oppressive governments there is the commercial aspect. The field is not only not so commercially attractive (a good thing probably), but could also become a threat to current status quo (internet monopolists, etc.). And if applied to local sharing communities any government is not so happy about the taxes the miss out on with all the non-intrinsic bartering and doing each other favours.

From about 2001 the internet is strewn with the corpses of 'the decentralized web is coming'- and 'we must act now before regulation strikes'-type initiatives. They did not survive, but I think for most of them it was their positioning, not the 'evil' forces that led to their demise.
But these forces are steadily growing and will become more and more truly felt.

Repositioning ideas

Just doing a bit of brainstorming and ideation here, since restructuring needs a well thought-out plan. Names are just indicative.

• I would make the main entry point / landing page to 'Dat world' be targeted to the community ecosystem at large

• The main entry could be: datfoundation.org (or with a dash)

  • Has better findability in search engines than 'project'
  • Comparable to dat-awesome, but much better structured
    • (PS. Remove second, outdated awesome list that exists, confusing)
  • Both technical and non-technical information source
  • Support, but not necessary provided by you, but also by the trusted community members
  • Introduction, reference to partners and affiliate initiatives
    • Affiliates operate completely independently, but now have a central focus point, a base
  • Blog, News, Articles, Video, Forums, Community, About, etc.

• The datfoundation.org goals:

  • Bring awareness, attract people to protocol, technology and ecosystem
  • Foster the community, central community hub
  • Attract funding! --> now broadly positioned/exposed to everyone involved

• Have datproject.org be a sub-site of the foundation

  • Explain Vision, Mission and Values, Philosophy
  • Explain importance of Funding, Sponsoring, Donations
  • Present your Sponsors in more detail
  • Present the Team and major co-operators
  • But nothing else ...

• Have datprotocol.org as another sub-site

  • Remove the datprotocol.com or have it as a mirror (why did you choose .com?)
  • Sections like 'Using Dat from the command line` do not belong here. No deep-dive in code, implementation details

• The datprotocol.org goals:

  • Protocol specification, technical concepts explained, architectural guidance
  • Insurance to developers regarding compatibility, quality of their own projects and contributions
  • And thus avoid having developers doing reverse-engineering on scattered code to find their way, waste time

• Have a dattechnology.org sub-site (or datplatform.org or similar)

  • Provides the 'deep-dive' information for Dat core technologies
  • Has pointers to core modules (hypercore, hyperdrive, discovery-swarm, etc.)

• The dattechnology.org goals:

  • Explain module structure and philosophy
  • User and API documentation
  • Tech-stack, versions, release notes, breaking changes, known issues
  • Technical in-depth articles, howto's, implementation details
  • Technology roadmap, plans, hurdles, technical challenges
  • Coding guidelines, licensing, contributing, code of conduct

• Have a datecosystem.org sub-site (or datcommunity.org)

  • Only for community contributors to present their works (compare 'marketplace')
  • Has no forums, articles, etc. These are accessible from the central foundation hub
  • (It would be really cool if this was a Dat application itself, e.g. a Blendle- or twitterboard-like dashboard where presentation cards automatically pop up when a contributor creates a local Dat with some html and then syncs it)

• Have a datsharing.org or whatever fittingly named sub-site

  • For your desktop-based file sharing solution that is now prominently advertised as 'the Dat'
  • Fine to target it exclusively to scientific community
  • Fine to hide the underlying technology for reasons of sensitivity (tip: also choose an unrelated name altogether)
  • Could position it as an officially 'supported' application in the Dat ecosystem

• Have similar branding and style on all sites and pages that constitute the dat foundation (not the community ones, of course)

  • Visual consistency is very comforting to people. It radiates trust, professionality and stability.

• Sanitize the repositories that exist in github.com/datproject

  • Only have relevant repositories that correspond to foundation concepts, structure and/or modules
  • Remove orphaned, irrelevant and outdated repositories like
  • Rename repositories where it improves consistency, findability and branding (public branding, not commercial)
  • You could create a new datfoundation organization and use datproject as your 'attic' for old projects

• Have at least the core module repositories under one github organization

  • Really this would be so much better!
  • It took me hours to figure out how you are structured, what was important or related, and - more importantly - what was 'supported', considered part of the core rather than a 3rd party lib dependency

• Consider hosting a private Gitlab instance to which trusted community members can enter to work on the more 'sensitive' parts of the technology base

Required effort

Now this looks like a lot of work, and part of it will certainly be. But most changes can be adopted gradually on a clean migration path.

• The high-level restructuring can be done without too much effort.
• The foundation website can be wholly, or mostly community maintained
  • But it probably requires a different setup (e.g. some CMS) to achieve ease of maintenance
• The protocol and datproject sites should be entirely controlled by you
  • But they are static and do not frequently change
• The technology site is controlled by you and trusted community members
  • Alleviating you of much of the burden in merging documentation improvements, etc.
  • Site is also static

Well.. quite a post this became. I hope you'll find it useful!

[aschrijver]

BTW Just bumped into a confused potential community member: https://stackoverflow.com/questions/44859200/what-are-the-differences-between-ipfs-and-hyperdrive

(Apparently the first one asked, related to dat project, I can't find more. Someone having 1500+ reputation could add SO tags for dat-project, decentralized-computing, etc.)

[pfrazee]

In addition to some pretty good ideas, I think @aschrijver has a point. The tech itself is a little bit buried in the dat homepage.

[ralphtheninja]

I have a profile that I haven't used in a long time, but in the process of recovering it, since I can't remember the details :) Anyway, it has 50k something in reputation and if I get the account back I could use that as 'support' for dat.

[aschrijver]

Hi all,

Some more input, but on a slightly more technical level, but very much related to the above ..

Technical considerations

If I it understand correctly, simply put, with Dat you now have:

    stream exchange --> file exchange --> dat exchange

This by using modules from the ecosystem. AFAIK you currently have a modular decomposition, but you do not yet have a full modular design. or call it framework, if you prefer.

This because transitive dependencies require me to untangle, recompose and add my own glue logic, in order to realign modules in such way that it supports my new cool use case.
But the modularization is not what I wanted to talk about here. That's for a future topic maybe.

Refactoring proposal

Instead consider this: Wouldn't it be better if instead Dat was modelled as follows:

    stream exchange --> message exchange

So as a

decentralized message-based system on top of raw streams

In this design:

• hypercore-protocol defines wire format (nothing changes)
• dat-protocol defines file archive exchange (nothing changes)
• dat-messaging defines allowed messaging formats, data types, core message types, etc.

I think this constitutes a change that requires only little effort, yet comes with tremendous benefit:

• First of all, technology-wise you'll have immediately repositioned yourself according to the 'Broadening vision' objective

• Message-based systems are incredibly modular, decoupled by nature. Actors are only triggered by messages they understand and that are addressed to them

• New applications need only to plug into the - here decentralized - message bus and everything just works. Outdated apps can run in tandem and in peace with their newer versions. Applications can appear, disappear without disruption, etc.

• Want to have Event Sourced application? CQRS? DDD? File exchange? Live video editing? IRC? What have you? Sure, no problem. Define the right message types, pub/sub handling and message processing (with much help from Dat ecosystem)

• You can define typed data models with relationships that correspond to your messaging design and use e.g. GraphQL to help minimize wire traffic and payload sizes

• You would be clearly positioned with regards to IPFS. You would no longer be more or less competing standards. You serve different purposes. And Dat would have the unique selling points now. While any IPFS feature could also be modelled with Dat, the reverse is not true. You are now the more general-purpose technology.

Overhead and effort

The overhead of wrapping data payloads into messages is only very small, depending on the message format you choose. So would code changes to existing modules, I imagine (but I'm not the one to say)

I might be off in some cases, or stuff already exists; I know still little about Dat. But it is the overall idea that matters :)

Cheers!

PS. I'll follow up with yet another one. About vert.x this time and how I think it can progress your cause.

[jondashkyle]

Good feedback, @aschrijver!

This isn’t directly related to any of the points you made, but I suggest watching this talk by Alan Kay on the power of simplicity. From my perspective, there is a lot of overlap with how the Dat community approaches (philosophically) some of the problems you’ve mentioned.

https://www.youtube.com/watch?v=NdSD07U5uBs

You could say the people working in this space of distributed, p2p systems are today sort of like the astronomers of Copernicus’ era :)

Also, this part when going over scaling at PARC is particularly relevant:
https://youtu.be/NdSD07U5uBs?t=1749

[pfrazee]

We have an out-of-date site at https://www.datprotocol.com/ which we could turn into a technology & developer-focused site. Tara and I can contribute some time & energy to help with that.

[jondashkyle]

With that developer site, I think documentation is kind of an art form, so this is relevant!

“Good art should elicit the response of “Huh?” and then “Wow!” As opposed to “Wow!” And then “Huh?” Whenever I feel an instant “Wow!” I always question when the “Huh?” is going to come. I always think this is a good tool to use when I’m trying to digest something new.”
— Ed Ruscha

[aschrijver]

@jondashkyle saw 1st video. entertaining! @jondashkyle Huh? ;)
.

One last chunk to my raw input stream, before I'll await pushback and response data :)

I wanted to address some concerns @blahah and @joehand came up with, what I mean with 'ambitiousness' (hint: its not github stars) and to introduce you to vert.x in the process. You might want to take a quick peek at their landing page before you read on..

Let's start with vert.x..

Vert.x is interesting for 2 reasons: one, it's a good case study for good project organization and community building, and two, their technology and/or ideas + concepts can be quite relevant to you.

Comparing vert.x

@blahah The Vert.x project, invented by Tim Fox, came along quite smoothly, with a steady 1.x, 2.x release chain. Tim as custodian had a very practical approach, hammering on KISS, LEAN and steady, healthy growth. Very comparable to want you want to see with Dat. Also their project organization was very similar. Things were added as needed, by people having time and opportunity. Everything fine!

But then - despite their careful approach - traction happened. People started taking note. They couldn't help it. And quickly problems started to amount. Code that shouldn't be there crept into vertx-core, architecture needed an overhaul, and more importantly Tim Fox - being the ultimate expert - was drowning in community support, which didn't help (I'd imagine e.g. @mafintosh already having quite a workload on these things). In short: Vert.x was going up quickly on Kay's 'Yikes' curve @jondashkyle

So with community involvement they decided on a clear strategy for a total reorganization, to keep their dedicated focus, productivity, high code quality, and guarantee useful community involvement. In short, they did their job well, and even survived Tim Fox leaving the project, without as much as a hitch. The result is there for all to see.

Some interesting bullet points (I won't digress too much, hopefully):

• appealing web presence, narrow profile, everything branching out from landing page
• well-guarded core, under Eclipse guidance, but without most of its strictures
• toolkit with a comprehensive set of officially supported modules (meaning: compatibility, stability guarantees, follows release cycles, not customer/dev support)
• clean, consistent, relevant set of repositories in vert-x3, no pollution
• if you are officially supported you must be in vert-x3
• community can be promoted to official modules based on popularity, use, stability and quality. The core team is the ultimate decider here
• ~800 questions on SO. Modest, yes, but not requiring core team attention at all
• very active google group and freenode channels for the dev team
• and with all this vert.x didn't change its identity. It still felt like the project as it was before the Jykes! curve started mounting

In terms of broader exposure to the public Vert.x is still quite low-profile, being silently integrated and embedded in a number of large projects, for dedicated purposes mostly. A kind of exposure the Dat team would feel comfortable with as well, I think.

Concluding: "Be prepared for traction when it comes"

Vert.x technology

Not only is vert.x an excellent example of a message-based application framework, it also puts the properties that come with such architecture design to good benefit. For example by adding polyglot programming support.

Now this should resonate with the scientific community! I would bet that given N programming scientists, you get N + 1 programming languages being used. Vert.x polyglot programming allows teams / projects to cooperate seemingly using their (jvm-based) language of choice. And this goes further than just having decoupled modules, or microservices written in different languages.

[Note: Regarding JVM I should probably write another post sometime, on a number of interesting cultural observations I made, as a newcomer to your community, like religious superstitions. Let me know if you're interested]

If Dat were to be polyglot in the future it would attract a broader range of scientists to both dev and user community - people that already have the proper mindset and moral + ethical values (in general).

Then there is the vert.x toolkit itself. Even though vert.x is running on the JVM it can be a useful technology for running/combining with Dat. I don't know the terminology, but I hear about various background services, storage, blind servers, repeaters/forwarders, elevated services, etc.

Concluding: "Be open-minded towards all technology"

Traction and exposure

Now @joehand besides the 'How much traction can we handle? How much should we handle?' discussion.. You rightfully say 1. 'Traction leads to exposure, and exposure leads to unwanted restrictions'. An interesting subject.

And vitally important, I would argue.

So are tickling questions like these:

2. Can we afford to be relatively small and under the radar for long?
3. How much exposure is required to survive?
4. What is our moral responsibility towards the technology in that regard?
5. What if our technology does not become the success we imagined?

Let's address them in turn..

Negative exposure

@joehand wrote:

"Finally, there are use cases we're interested in where popularity is a negative. Human rights defenders in countries with heavy surveillance need tools built on Dat or similar tech. But popularity will make it more likely Dat is blocked or measures put in place to make it more difficult to use (see bittorrent)."

I agree. But, unless you guys really mess up, negative exposure eventually coming to your project is inevitable. Its a given. And just like asteroids, tsunami's and earthquakes you should plan in advance, make preparations so you are ready when the moment is there.

Having a stable, theoretically unstoppable software at that time just doesn't cut it, as I'll explain next.

Concluding: "Be prepared for restrictions, they will come"

Minimum exposure

Consider that while you go steady and slow, technology in general and encroachment of internet freedoms may well be evolving at an altogether much more rapid pace.
And with this comes the same increase in ability by government and commercial powers to control and suppress ever smaller initiatives with relative ease.

And when dark powers strike, you'll be much weaker as a small community of color-bearded ;) good-hearted dev guys, than with a broad, vibrant community with representatives from throughout society, who are willing to defend you tooth and nail.

Another point. The above doesn't matter, right? "Our decentralization magic, once stable, is like self-replicating nanotech. Virtually unstoppable once we open the vial.". You may think something along these lines.

But you'd be dead wrong! Other than the nanotech, people have to physically switch you on first, on a peer-by-peer basis. And if they don't like your technology, or there is a more viable candidate around they will choose that technology over yours. I know I would at least.

Sure, there will still be a number of scientists using your tech in 5 to 10 or even 30 years, just like now with the many old techz - sometimes in archaic languages from the sixties - still floating around.
And you will apply all your hardly gained knowledge and experience to that other technology that became the standard (maybe IPFS), so not much is lost. If its for the best interest of the scientific community.

But it would be a damned pity, would it not? You want sow seeds that prosper even when you go for greener pastures.

Concluding: "Exposure is vital to survival"

Optimum exposure

We've established there is a minimum required amount of exposure. What about the upper bounds.. Is there a maximum? Or, better stated: What is the best amount of exposure at any specific point in time?

Well, I would argue that that should be the maximum amount you can possibly deliver successfully. Strive for maximum exposure! But that's entirely besides the point.
Yes, strive for maximum, but.. reach only the right people! So follows:

Concluding: "Optimum exposure requires targeting"

And the efforts required to get the right kind of exposure and build the right community, should put little to no unnecessary burdens or distractions to the core team. Therefore:

Concluding: "Optimum exposure requires clever strategy"

You could say, good exposure requires marketing, but I know that is a dirty word :)

** Responsibility**

I just recently got interested in the field of Decentralized Computing (after being annoyed how the term 'Serverless' was hijacked by the big server moguls). And I observed following:

• An entire IT field that is mostly still unbroken ground
• A field where there are as of yet hardly any players
• A field where there is 0 (as in zilch) generic standardization effort
• A field where there are about 4 to 5 viable technology candidates for standardizing
• But only to sub-domains of the field, like file sharing
• And only one of these candidates has potential for more generic application. That would be Dat!

Ooff! That puts the Dat project in a unique position!

In any other IT field being a first adopter in such an empty field would lead someone to sink to the knees and thank the Big Bang, or whatever God you preach.

The special characteristics of this IT area make this position both a gift and a curse.

Given this situation can you afford to let the technology fail? Can you afford not going to the utmost in healthy competition to IPFS and others? Can you effort not jumping to grab unique selling points? Or can you afford not to broaden your horizon, and reposition yourself so that IPFS and Dat can coexist happily together?

The answer is ... yours to make

Concluding: "New technology comes with moral responsibility"

Success

I am not going to spend much time here. I am not going to talk about 'Failure'. The gist of my rants should be clear by now:

The moral of the story: "Consciously craft the formula to your own success"

(probably already said by someone, I'm not laying claim :)

--

Pfff, that was it, I'm done for now, but created for you with much pleasure! As Richard Quest would rumble, I exclaim: "I hope you'll find it profitable, wherever you may go!"

Arnold Schrijver.

[okdistribute]

@ralphtheninja that would be awesome. Thanks for volunteering that. Let us know if you get that account back anytime soon :)

[okdistribute]

@aschrijver thanks for your input, i think that it's good to talk about these things. User facing work isn't good work unless it's in a constant state of change and reflection, hopefully fast enough to keep pace with a rapidly changing world :)

As far as traction and exposure, yes I agree that it's important for the protocol to have exposure, but right now we don't have a ton of resources to spend on that. There is already a large community working on dat and the ecosystem, with lots of input and feature requests hanging unworked on. So this leads me to believe that yes, more developer-focused documentation and marketing ought to be available front and center.

The site as it is now was designed before beaker approached 1.0 and other apps started being built on Dat. I think now I'd approach it similarly to what you're talking about, as a data sharing tool as well as a stack for building decentralized apps, with the suite of compatible apps and the logos of organizations currently using dat.

I'm going to read through that issue thoroughly and take folks' suggestions to come up with a new front page concept. Thanks a bunch!

[aschrijver]

@Karissa thanks for your feedback! And remember I am not criticizing anyone in any way. You've done a tremendous job getting where you are now.

Nice to hear you are like-minded on so many points.

Besides the more cosmetic and informational aspects, I hope you are equally triggered by the more strategic, disruptive and sensitive point I am laying out here.

(For that matter I may have distracted you myself from the real essence of this thread, by providing too much additional eye candy and side paths.)

I think in my comments you'll find plenty of hints to strategies on finding more funding and obtain it more easily. If funding is an issue, you should fix it ;)

"There is already a large community working on dat and the ecosystem, with lots of input and feature requests hanging unworked on."

The more reason to go the vert.x route (or some other successful initiative of your liking)!

[creationix]

FWIW, I would love to implement DAT in other languages/frameworks, but am stopped mostly by a lack of clear documentation on the protocol and internal data structures.

In particular, I'd probably implement it in Luvit, C, and Rust as I have time.

[okdistribute]

@creationix have you read the dat paper? https://datproject.org/paper If that isn't good enough reference then could you perhaps @mafintosh know what you think might be helpful? He has expressed wanting to make it clear enough to reimplement dat.

[creationix]

There wasn't enough detail in there last time I looked, but I see more now thanks.

[aschrijver]

@creationix IMO it would be fantastic if there were more language implementations, that could interoperate in the Dat ecosystem!

That would make the technology so much stronger, more capable to survive
It would lead to cross-pollination, increased innovation, and a much larger, more diversified developer base, which is also a good thing.

I've seen this recently with GraphQL, where besides the reference implementation now has stable implementations in Javascript, Ruby, PHP, Python, Java, C/C++, Go, Scala, .NET, Elixir, Haskell, SQL, Lua, Elm, Swift, OCaml, Rust, R and yes @whilo also in Clojure, but you were probably already aware of that..

@Karissa Explicitly promoting your spec and stimulating implementation initiatives is yet another way to acquire more / easier funding

[aschrijver]

BTW @Karissa @joehand I would love to hear the opinion of some of your sponsors on the more strategic topics I've touched on above.

Would it be possible to involve them?

[aschrijver]

On request of @dominictarr I created an executive summary, which I copied here to lead the discussion back to essentials, not implementation details.

Executive summary

• Decentralized computing holds the promise of The Decentralized Web
• The field is still largely empty, undiscovered
• There are only a handful of players
• This puts these early-bird players in a very special position
• Usually where such Google-sized niches exist every party would jump on the bandwagon
• But there exist unique forces working against the success of DC: government and commercial powers
• Looking at history most of preceding DC initiatives have failed.. but not due to evil powers
• They failed mostly due to unrealistic vision, combined with bad organization / community building

--> They failed because did not properly define the roadmap to their own success (maybe some even didn't define success at all)

As a newcomer (interested in building a social network) I made following observations on some currently viable candidate technologies (i.e. the players, and specifically IPFS, SSB, Dat and Replikativ):

• All candidates, except maybe IPFS, are still very fragile, in their infancy
• IMO they incorporate many of same mistakes of their failed predecessors
• Only real standardization efforts are IPFS and Dat. Of these IPFS is the most likely to succeed
• But both IPFS and Dat target just a small DC sub-domain: file sharing
• This makes IPFS unsuitable to me, but current Dat design allows it to be easily repositioned
• Expanding vision and positioning as a message-based system would be high value, low impact
• Same holds for SSB: lose application focus, broaden horizon, define your success

Now, where previous failures had mostly themselves to blame, you are of the first generation of DC software initiatives that will be challenged by the dark powers
And this time the stakes are much higher! Very comparable to climate change. Action is required now to avoid future disaster. Instead of climate its about the internet

Slightly exaggerated, but in essence true, the choice is for a future internet where there is:

• either freedom of expression and people control their own data and privacy
• or controlled thought in a 1984 + cyberpunk (without the magic) type of society

Recommendations

• Analyse and understand your unique position
• Then be conscious of the moral responsibility that comes with it
• And what that means to how you approach the 'market'
• Foster grass-roots and open-minded culture, cooperation
• But be much more strategic in your approach
• Define the recipe to your success with all this in mind
• And reorganize accordingly, so you come prepared

Quick reference

• Traction, exposure, moral responsibility (+ case study for good organization)
• Dat repositioning, expanding vision and benefits (+ reorganization brainstorm)
• Dat repositioning as a message-based system, the technical aspect

[joehand]

Thanks, this was interesting. I'm going to close this, feel free to continue discussion in our discussion repo: https://github.com/datproject/discussions.

[aschrijver]

Feels like its all being archived :(
But I'll move it.

(UPDATE And severely cleaned it up, so it was probably for the best. Please take a look: dat-ecosystem-archive/datproject-discussions#58)

[joehand]

It is being archived. We use this repository to manage our work on the command line Dat tool and determine what to work on. We understand it is often used as a catch-all for the project but always try to move them to the proper repository.

Keeping issues open that aren't related make addressing the actual issues more difficult. We actively manage issues in this repo and try to close issues regularly if they are not related to the command line tool.

These points are made in our contributing guide which we ask you to read before opening an issue:

Any new issues should be actionable. If your issue cannot be solved (and thus closed) please reconsider opening it in our discussion repository or rewording it.

And:

We prefer to be able to close issues in this repository, which does not lend itself to discussion type questions. Open discussion type issue in the datproject/discussions repository.

Please keep this in mind when opening future issues. We can move all of the comments here to another issue in that repo, if you'd prefer.

[aschrijver]

Well written and understood. Thanks. I would move this to a Contribution Guidelines section if you hadn't already.

[arni077]

@aschrijver The most important aspect which is security wasn't addressed properly by Dat. Dat network still has to face same problems other DHT adopters had with Sybill attacks,node adversarial fraction and still possible churn. IPFS approach with Coral DSHT+ S/Kademlia is much better and still missing in Dat.