Overhaul and complete update of awesome page

[aschrijver]

This PR is related to:

• Improve dat-awesome page datproject-discussions#68

and (to lesser extent):

• Overview of the Dat ecosystem datproject-discussions#70

[joehand]

Thanks for starting to organize all of this!

I appreciate the effort to label these but I really don't understand the labeling system or how it was applied. For example, why is discovery-channel an "external module" when discovery-swarm is a "core module" (discovery-swarm depends n discovery-channel and they were both written by max/mathias). Or why is dat-colors a community project. In my mind there are a few categories that are clear:

• User Applications that you can use to manage dats (dat cli, dat desktop, beaker)
• Modules used inside Dat applications (but not hyper[drive|core])
• Modules used inside hyper[drive|core]
• Utilities for managing hyperdrives/cores
• Things built on top of hypercore feeds or hyperdrive archives

My biggest confusion is the mix of modules we use to build hyper[core|drive] and modules built on top of hyper[core|drive], for example in the streams section. It'd be cool if that was made clearer with the icons, but right now the icons seem to confuse me more in that section.

And a few general comments/questions:

• A fair number of the modules listed are out of date (specifically older than hypercore v6 and hyperdrive v9 which was a major breaking change). Do we want to label them? Or move them to a section asking folks interested to updated them? This also goes for a few of the links in the technology section.
• We used the 📔 icon to pull in docs from modules to our docs page. We either need to restore that or identify modules we have in our docs page another way.
• Some of the categories and groupings need clarity. Perhaps adding a description would be useful. For example, the group of "data access" doesn't seem coherent and many of these don't have to do with accessing data in my mind.
• I'm not sure why some modules like pump are listed. They have nothing to do with Dat besides the fact that we use them.

[aschrijver]

but right now the icons seem to confuse me more in that section

I gave the labeling my best effort as a newcomer to the dat community. I spend lots of time to determine what is what. Every newcomer will have the same confusion.

As you rightly state you and the rest of the dat team are the one which know best which icon belongs where, and what is outdated! That's the issue: its now in your heads only and we have to bother you with needless questions on #dat :)

It would be great if you incorporated this PR (because the old awesome was really outdated) and work from there.

[aschrijver]

PS Pump could be removed, but others are so essential that they add information to people that e.g. want to create ports of dat technology to different languages / runtimes and are working from the specs / protocols.

[aschrijver]

We used the 📔 icon to pull in docs from modules to our docs page. We either need to restore that or identify modules we have in our docs page another way.

I think currently there are a number of better solutions to aggregating the docs, and also allow some richer formats in docs.datproject.org (like 'warning' yellow box formatting, etc.)

PS was an ugly icon anyway, and unclear what it meant.

[aschrijver]

PS2 I got some really really positive feedback on the layout of this list on the #dat channel.

[aschrijver]

A fair number of the modules listed are out of date (specifically older than hypercore v6 and hyperdrive v9 which was a major breaking change).

Exactly, but are they still active part of the ecosystem or should they have a clear 'Outdated' warning (or go to datattic, see: dat-ecosystem-archive/datproject-discussions#71)?

(this all makes it really hard to discover the dat ecosystem, and I then I am only talking about the dat project side of things, not even the wider community)

[aschrijver]

On the current labeling system:

• orange, core module = transitive dependency when using 1 of the 4 top-level modules (dat-node, dat-cli, dat-js, dat-desktop)
• blue module, dat-related = any project created by dat core team members that is reasonably up-to-date (up to a year old)
• black square, community module = interesting, non-trivial (but not necessarily production-ready) community modules, I found
• red icon, vital dependency = modules that implement core concepts of the dat technology, like bittorrent-dht, multicast-dns, merkle-tree, etc.

[joehand]

It would be great if you incorporated this PR (because the old awesome was really outdated) and work from there.

Ya, I'd agree, here is what is blocking the merge:

• This will break our docs builds immediately. There are probably better solutions for aggregating docs but we have something implemented that is working for us.
• I do not understand enough about the categorization to maintain it. Even with your explanation, you have inconsistent labeling (you said that red modules "implement core concepts... like merkle-tree" but the merkle tree module is marked orange).
• The section labels do not properly communicate their intent. A description of each will help us and others know where to contribute new modules.

I gave the labeling my best effort as a newcomer to the dat community. I spend lots of time to determine what is what. Every newcomer will have the same confusion.

This is my fundamental concern with the labels. If every newcomer will be similarly confused, how are they supposed to contribute back to this list? And then if the Dat team members also do not understand it, we cannot maintain it.

I really appreciate the intention of the labels but this system causes confusion without communicating more information.

[aschrijver]

• Yea agree the document aggregation should not be disturbed. I can look for an easy solution.

This is my fundamental concern with the labels. If every newcomer will be similarly confused, how are they supposed to contribute back to this list?

• Yes, we are saying the same, I think. I gave it my best with regards to labelling and categories but will certainly have made a number of wrong guesses and strange categories. No one better than a dat project member to give these a quick scan and change to something more appropriate. And only then the list is 'final'
• In the labelling the red icon entries can be left out, if you will. They are a deviation of a standard awesome list anyways.
• In terms of labeling you could also have just blue and orange for 'dat project'-project and 'community'-project

[aschrijver]

@joehand I have created a documentation proposal: dat-ecosystem-archive/datproject-discussions#73

[aschrijver]

@joehand The latest commits solve most issues you addressed

• simplified labelling: only you + the community
• improved categorisation: added explainer text, changed data access to data exchange
• mentioned important modules in separate paragraphs (clearly not as part of ecosystem)

Not addressed: Outdated modules

• I don't know what is outdated
• Options for handling outdated in the awesome:
  1. add a '(outdated)' text after the description, but leave them in main categories
  2. add an 'archived' category or paragraph, and move all outdated projects there
  3. do not mark as outdated on awesome list, but clearly on the project README itself
  4. remove outdated entries (off-topic: preferably also move project to a datattic org, or something)

Option 3 is not so bad, IMHO:

• outdated entries are bad marketing-wise, and not all that awesome on an awesome page
• discourages people to check out the project and decide to contribute to make it actual again

[aschrijver]

@joehand the PR on Docs fixes the documentation aggregation, after which this PR can be merged. See dat-ecosystem-archive/docs#79

[max-mapper]

@aschrijver question, is there a reason for using the    extensively in this PR vs more normal markdown?

[aschrijver]

Its to create a list with the icons. Otherwise it looks visually quite bad. I put a lot of effort in making the list look as clean as possible.

[max-mapper]

a good rule of thumb for github PRs is to not spend lots of time on something that the maintainers haven't agreed to. this makes it harder to maintain, as it has made the markdown more complicated.

i also don't find the icons to be useful, the distinction between community and dat team modules is arbitrary IMO.

otherwise i appreciate the effort to organize this list. it's just hard to make big unsolicited opinionated changes and expect everyone to be on the same page.

[aschrijver]

Yea, thanks. It is extremely hard to know what is a Dat Project project and what is not. I find that all the time. The same goes who is exactly part of the core team.

I started completing these changes after getting very positive reaction on #dat after my first draft. Should maybe not have done that.

I've also given a lot of unsolicited opinionated feedback on Discussions, which is largely an exercise for myself, it appears.

It not easy volunteering one's valuable time trying to improve dat project in ways that are not hardcore-coding related.

[joehand]

Thanks for updating the list @aschrijver. I did some reorganization. I tried to separate out modules that a user may need for building stuff on top of dat vs what we used to build dat. I also moved the outdated modules to their own section as you suggested (since we do not own many of them, adding outdated to all of their readme files isn't feasible).

Yea, thanks. It is extremely hard to know what is a Dat Project project and what is not. I find that all the time. The same goes who is exactly part of the core team.

We understand it's difficult to tell what is part of Dat and what isn't, that is why we feel making that distinction is not necessary (I've not seen any other awesome list with that distinction). It was also not something I'm willing to maintain as it'd require more curation of every PR.

It not easy volunteering one's valuable time trying to improve dat project in ways that are not hardcore-coding related.

Thanks for putting all the effort in to updating this list, it was something we'd put off since the last big release in May. We recognize that non-coding contributions are hard at this point, and I am sorry for that. I'd wish we could work to improve that immediately. But with a current team of four (some part time) trying to ship new features, fix bugs, and find additional funding is about all we can manage. It is something we'd like to prioritize but unfortunately not feasible right now so you'll have to continue to be self-guided.

[aschrijver]

Thanks a lot @joehand You did a lot of work on it still. It was the reshuffling that a non-insider could never do. Really appreciate it!

We understand it's difficult to tell what is part of Dat and what isn't, that is why we feel making that distinction is not necessary

It is okay without the icons, but you understand my need for them. For a technology to be trusted people should be able to find the work of the core team, be it core modules, reference impls, etc.

We recognize that non-coding contributions are hard at this point, and I am sorry for that. I'd wish we could work to improve that immediately. But with a current team of four (some part time) trying to ship new features, fix bugs, and find additional funding is about all we can manage.

No need to be sorry, you, @mafintosh @maxogden @Karissa @juliangruber are all doing your utmost. Look, when I started investigating Dat Project I became very worried on a number of weaknesses and threats I perceive, that may lead you to fail.
That worry motivated me to write all those Discussion pieces; to offer a different perspective.

The biggest threat currently, IMHO, is you are already swamped in technical work and may suffer to some extent from 'developer myopia' (as I described in my small culture SWOT). I've seen this clearly already with sciencefair and hashbase where I gave feedback on nonsensical, technical texts on their landing pages; not targeted to audience.

In this way Dat Project risks becoming 'just' a set of cool projects and apps, but not a good, established technology.

I would strongly advise you to have at least one team member dedicate half his/her time on the non-technical stuff, or acquire some more funding for and additional part-time member!

Besides, having clarity of vision, clear strategy, community approach and concrete action plans should make it much easier to acquire funding, I would argue.

[joehand]

I would strongly advise you to have at least one team member dedicate half his/her time on the non-technical stuff, or acquire some more funding for and additional part-time member!

Yes, we'd agree =). Let us know if you know foundations interested in funding us. You can also donate: https://donate.datproject.org. We can make sure your donation goes towards that position.