Documentation: Tribler, IPv8, and Trustchain developers portal

[synctext]

The Developers Doc Portal

Goal: enable starting developers an immediate broad grasp of Tribler
Approach: single webpage with pointers to all documentation, low-maintenance, no sub-pages.

• Tribler basic functionality with screenshots: http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_61x/lastBuild/
• Tribler architecture: modules and their functions
• Database structure: Tribler tables, Dispersy core and Dispersy communities
• Tribler API docs: Tribler download core, Libswift and Dispersy core
• Continous build system: Jenkins performance, unit test, and system tests
• Tribler, Dispersy and Libswift testing framework: DAS4 tests (Effort / Allchannels)
• Dispersy wire protocols: Walker, Allchannels, Search, BarterCast3
  https://github.com/Tribler/dispersy/blob/master/doc/wireprotocol_2.x
  SHA: b8eab4fb501213ae151372cd83a6131582e2ad6b
• Dispersy access control, security and certificate validation
• Dispersy core: event handling, internal data structures and fault behavior

How much can we autogenerate? (zero-maintenance..)

[boudewijn-tribler]

As for the Database structure, we can easily create diagrams from the .schema data using:

sudo apt-get install sqlfairy
sqlite dispersy.db .schema > dispersy.sql
sqlt-diagram --color -f SQLite -o dispersy1.png dispersy.sql
sqlt-graph --color -f SQLite -t png -o dispersy2.png dispersy.sql --show-datatypes --show-constraints

Unfortunately this does not include comments. Nor is the parser able to understand the Tribler database, but I suspect a bit of database cleanup should be able to fix this. Regardless, dumping the .schema data every night can give us up-to-date database documentation.

[NielsZeilemaker]

@synctext seems like a great addition to the new website.

[synctext]

Historical Dispersy stuff:

tribler.sql sadly does not parse. It gives this error:

Error: translate: Error with parser 'SQL::Translator::Parser::SQLite': Unable to parse line 32:
''
'CREATE TABLE MyInfo (
'  entry  PRIMARY KEY,
'  value  text
');'

Full listing:

CREATE TABLE MetadataMessage (
  message_id             INTEGER PRIMARY KEY AUTOINCREMENT,
  dispersy_id            INTEGER NOT NULL,
  this_global_time       INTEGER NOT NULL,
  this_mid               TEXT NOT NULL,
  infohash               TEXT NOT NULL,
  roothash               TEXT,
  previous_mid           TEXT,
  previous_global_time   INTEGER
);
CREATE TABLE MetadataData (
  message_id  INTEGER,
  data_key    TEXT NOT NULL,
  data_value  INTEGER,
  FOREIGN KEY (message_id) REFERENCES MetadataMessage(message_id) ON DELETE CASCADE
);
CREATE TABLE BarterCast (
  peer_id_from  integer,
  peer_id_to    integer,
  downloaded    numeric,
  uploaded      numeric,
  last_seen     numeric,
  value         numeric
);
CREATE UNIQUE INDEX bartercast_idx
  ON BarterCast
  (peer_id_from, peer_id_to);
CREATE TABLE Category (
  category_id    integer PRIMARY KEY NOT NULL,
  name           text NOT NULL,
  description    text
);
CREATE TABLE MyInfo (
  entry  PRIMARY KEY,
  value  text
);
CREATE TABLE MyPreference (
  torrent_id     integer PRIMARY KEY NOT NULL,
  destination_path text NOT NULL,
  progress       numeric,
  creation_time  integer NOT NULL,
  -- V2: Patch for BuddyCast 4
  click_position INTEGER DEFAULT -1,
  reranking_strategy INTEGER DEFAULT -1
);
CREATE TABLE Peer (
  peer_id    integer PRIMARY KEY AUTOINCREMENT NOT NULL,
  permid     text NOT NULL,
  name       text,
  thumbnail  text
);
CREATE UNIQUE INDEX permid_idx
  ON Peer
  (permid);
CREATE TABLE Torrent (
  torrent_id       integer PRIMARY KEY AUTOINCREMENT NOT NULL,
  infohash         text NOT NULL,
  name             text,
  torrent_file_name text,
  length           integer,
  creation_date    integer,
  num_files        integer,
  thumbnail        integer,
  insert_time      numeric,
  secret           integer,
  relevance        numeric DEFAULT 0,
  source_id        integer,
  category_id      integer,
  status_id        integer DEFAULT 0,
  num_seeders      integer,
  num_leechers     integer,
  comment          text,
  dispersy_id      integer,
  swift_hash        text,
  swift_torrent_hash text,
  last_tracker_check    integer DEFAULT 0,
  tracker_check_retries integer DEFAULT 0,
  next_tracker_check    integer DEFAULT 0
);
CREATE UNIQUE INDEX infohash_idx
  ON Torrent
  (infohash);
CREATE UNIQUE INDEX Torrent_swift_torrent_hash_idx
  ON Torrent
  (swift_torrent_hash);
CREATE TABLE TorrentSource (
  source_id    integer PRIMARY KEY NOT NULL,
  name         text NOT NULL,
  description  text
);
CREATE UNIQUE INDEX torrent_source_idx
  ON TorrentSource
  (name);
CREATE TABLE TorrentStatus (
  status_id    integer PRIMARY KEY NOT NULL,
  name         text NOT NULL,
  description  text
);
CREATE TABLE TrackerInfo (
  tracker_id  integer PRIMARY KEY AUTOINCREMENT,
  tracker     text    UNIQUE NOT NULL,
  last_check  numeric DEFAULT 0,
  failures    integer DEFAULT 0,
  is_alive    integer DEFAULT 1
);
CREATE TABLE TorrentTrackerMapping (
  torrent_id  integer NOT NULL,
  tracker_id  integer NOT NULL,
  FOREIGN KEY (torrent_id) REFERENCES Torrent(torrent_id),
  FOREIGN KEY (tracker_id) REFERENCES TrackerInfo(tracker_id),
  PRIMARY KEY (torrent_id, tracker_id)
);
CREATE VIEW CollectedTorrent AS SELECT * FROM Torrent WHERE torrent_file_name IS NOT NULL;
CREATE TABLE UserEventLog (
  timestamp      numeric,
  type           integer,
  message        text
);
CREATE TABLE BundlerPreference (
  query         text PRIMARY KEY,
  bundle_mode   integer
);
CREATE TABLE _Channels (
  id                        integer         PRIMARY KEY ASC,
  dispersy_cid              text,
  peer_id                   integer,
  name                      text            NOT NULL,
  description               text,
  modified                  integer         DEFAULT (strftime('%s','now')),
  inserted                  integer         DEFAULT (strftime('%s','now')),
  deleted_at                integer,
  nr_torrents               integer         DEFAULT 0,
  nr_spam                   integer         DEFAULT 0,
  nr_favorite               integer         DEFAULT 0
);
CREATE VIEW Channels AS SELECT * FROM _Channels WHERE deleted_at IS NULL;
CREATE TABLE _ChannelTorrents (
  id                        integer         PRIMARY KEY ASC,
  dispersy_id               integer,
  torrent_id                integer         NOT NULL,
  channel_id                integer         NOT NULL,
  peer_id                   integer,
  name                      text,
  description               text,
  time_stamp                integer,
  modified                  integer         DEFAULT (strftime('%s','now')),
  inserted                  integer         DEFAULT (strftime('%s','now')),
  deleted_at                integer,
  FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE
);
CREATE VIEW ChannelTorrents AS SELECT * FROM _ChannelTorrents WHERE deleted_at IS NULL;
CREATE INDEX TorChannelIndex ON _ChannelTorrents(channel_id);
CREATE INDEX ChannelTorIndex ON _ChannelTorrents(torrent_id);
CREATE INDEX ChannelTorChanIndex ON _ChannelTorrents(torrent_id, channel_id);
CREATE TABLE _Playlists (
  id                        integer         PRIMARY KEY ASC,
  channel_id                integer         NOT NULL,
  dispersy_id               integer         NOT NULL,
  peer_id                   integer,
  playlist_id               integer,
  name                      text            NOT NULL,
  description               text,
  modified                  integer         DEFAULT (strftime('%s','now')),
  inserted                  integer         DEFAULT (strftime('%s','now')),
  deleted_at                integer,
  UNIQUE (dispersy_id),
  FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE
);
CREATE VIEW Playlists AS SELECT * FROM _Playlists WHERE deleted_at IS NULL;
CREATE INDEX PlayChannelIndex ON _Playlists(channel_id);
CREATE TABLE _PlaylistTorrents (
  id                    integer         PRIMARY KEY ASC,
  dispersy_id           integer         NOT NULL,
  peer_id               integer,
  playlist_id           integer,
  channeltorrent_id     integer,
  deleted_at            integer,
  FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE,
  FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE
);
CREATE VIEW PlaylistTorrents AS SELECT * FROM _PlaylistTorrents WHERE deleted_at IS NULL;
CREATE INDEX PlayTorrentIndex ON _PlaylistTorrents(playlist_id);
CREATE TABLE _Comments (
  id                    integer         PRIMARY KEY ASC,
  dispersy_id           integer         NOT NULL,
  peer_id               integer,
  channel_id            integer         NOT NULL,
  comment               text            NOT NULL,
  reply_to_id           integer,
  reply_after_id        integer,
  time_stamp            integer,
  inserted              integer         DEFAULT (strftime('%s','now')),
  deleted_at            integer,
  UNIQUE (dispersy_id),
  FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE
);
CREATE VIEW Comments AS SELECT * FROM _Comments WHERE deleted_at IS NULL;
CREATE INDEX ComChannelIndex ON _Comments(channel_id);
CREATE TABLE CommentPlaylist (
  comment_id            integer,
  playlist_id           integer,
  PRIMARY KEY (comment_id,playlist_id),
  FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE
  FOREIGN KEY (comment_id) REFERENCES Comments(id) ON DELETE CASCADE
);
CREATE INDEX CoPlaylistIndex ON CommentPlaylist(playlist_id);
CREATE TABLE CommentTorrent (
  comment_id            integer,
  channeltorrent_id     integer,
  PRIMARY KEY (comment_id, channeltorrent_id),
  FOREIGN KEY (comment_id) REFERENCES Comments(id) ON DELETE CASCADE
  FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE
);
CREATE INDEX CoTorrentIndex ON CommentTorrent(channeltorrent_id);
CREATE TABLE _Moderations (
  id                    integer         PRIMARY KEY ASC,
  dispersy_id           integer         NOT NULL,
  channel_id            integer         NOT NULL,
  peer_id               integer,
  severity              integer         NOT NULL DEFAULT (0),
  message               text            NOT NULL,
  cause                 integer         NOT NULL,
  by_peer_id            integer,
  time_stamp            integer         NOT NULL,
  inserted              integer         DEFAULT (strftime('%s','now')),
  deleted_at            integer,
  UNIQUE (dispersy_id),
  FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE
);
CREATE VIEW Moderations AS SELECT * FROM _Moderations WHERE deleted_at IS NULL;
CREATE INDEX MoChannelIndex ON _Moderations(channel_id);
CREATE TABLE _ChannelMetaData (
  id                    integer         PRIMARY KEY ASC,
  dispersy_id           integer         NOT NULL,
  channel_id            integer         NOT NULL,
  peer_id               integer,
  type_id               integer         NOT NULL,
  value                 text            NOT NULL,
  prev_modification     integer,
  prev_global_time      integer,
  time_stamp            integer         NOT NULL,
  inserted              integer         DEFAULT (strftime('%s','now')),
  deleted_at            integer,
  UNIQUE (dispersy_id),
  FOREIGN KEY (type_id) REFERENCES MetaDataTypes(id) ON DELETE CASCADE
);
CREATE VIEW ChannelMetaData AS SELECT * FROM _ChannelMetaData WHERE deleted_at IS NULL;
CREATE TABLE MetaDataTypes (
  id                    integer         PRIMARY KEY ASC,
  name                  text            NOT NULL,
  type                  text            NOT NULL DEFAULT('text')
);
CREATE TABLE MetaDataTorrent (
  metadata_id           integer,
  channeltorrent_id     integer,
  PRIMARY KEY (metadata_id, channeltorrent_id),
  FOREIGN KEY (metadata_id) REFERENCES ChannelMetaData(id) ON DELETE CASCADE
  FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE
);
CREATE INDEX MeTorrentIndex ON MetaDataTorrent(channeltorrent_id);
CREATE TABLE MetaDataPlaylist (
  metadata_id           integer,
  playlist_id           integer,
  PRIMARY KEY (metadata_id,playlist_id),
  FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE
  FOREIGN KEY (metadata_id) REFERENCES ChannelMetaData(id) ON DELETE CASCADE
);
CREATE INDEX MePlaylistIndex ON MetaDataPlaylist(playlist_id);
CREATE TABLE _ChannelVotes (
  channel_id            integer,
  voter_id              integer,
  dispersy_id           integer,
  vote                  integer,
  time_stamp            integer,
  deleted_at            integer,
  PRIMARY KEY (channel_id, voter_id)
);
CREATE VIEW ChannelVotes AS SELECT * FROM _ChannelVotes WHERE deleted_at IS NULL;
CREATE INDEX ChaVotIndex ON _ChannelVotes(channel_id);
CREATE INDEX VotChaIndex ON _ChannelVotes(voter_id);
CREATE TABLE TorrentFiles (
  torrent_id            integer NOT NULL,
  path                  text    NOT NULL,
  length                integer NOT NULL,
  PRIMARY KEY (torrent_id, path)
);
CREATE INDEX TorFileIndex ON TorrentFiles(torrent_id);
CREATE TABLE TorrentCollecting (
  torrent_id            integer NOT NULL,
  source                text    NOT NULL,
  PRIMARY KEY (torrent_id, source)
);
CREATE INDEX TorColIndex ON TorrentCollecting(torrent_id);
CREATE TABLE _TorrentMarkings (
  dispersy_id           integer NOT NULL,
  channeltorrent_id     integer NOT NULL,
  peer_id               integer,
  global_time           integer,
  type                  text    NOT NULL,
  time_stamp            integer NOT NULL,
  deleted_at            integer,
  UNIQUE (dispersy_id),
  PRIMARY KEY (channeltorrent_id, peer_id)
);
CREATE VIEW TorrentMarkings AS SELECT * FROM _TorrentMarkings WHERE deleted_at IS NULL;
CREATE INDEX TorMarkIndex ON _TorrentMarkings(channeltorrent_id);
CREATE VIRTUAL TABLE FullTextIndex USING fts3(swarmname, filenames, fileextensions);
CREATE TABLE 'FullTextIndex_content'(docid INTEGER PRIMARY KEY, 'c0swarmname', 'c1filenames', 'c2fileextensions');
CREATE TABLE 'FullTextIndex_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'FullTextIndex_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));

<\details>

[devos50]

Actually, this is partially implemented now: http://tribler.readthedocs.org and http://dispersy.readthedocs.org.

[synctext]

Great documentation example: https://www.toptal.com/python/beginners-guide-to-concurrency-and-parallelism-in-python
Really cool stuff..
{ Remember, please fix issue "1", The Token Economy in 2018, then this fancy stuff for 2019. But I agree we need to think and plan ahead; just no overengineering. }

[synctext]

The cool kids at the Javascript corner make quite nice documentation: https://github.com/amark/gun/blob/master/README.md

We need a GUI .gif demo and tutorial + Dev pointers.

[synctext]

We need to visualise the Trustchain.

Create a movie out of it growing.
Cool tool to use: https://asciinema.org/a/35597

[synctext]

Revisiting #52 from 2003. Mostly duplicate issue from 2017: #2928. After over 14 years of effort, all of our initial research code has been replaced with improved code or even production-quality code. In the near future we need to start documenting {more}. Could we find somebody within this universe who wants to bug-hunt, code cover, and document full-time?

Another great example of online documentation: https://wyag.thb.lt/ This is a complete explanation of GIT by creating a fully compatible client. It consists of only 588 lines of very simple Python code.
A IPv8 / Trustchain stack with minimal features in tutorial form would be amazing. For instance, a crawler written from scratch. Idea for a BEP project? Minimal viable Trustchain agent, fully stand-alone, without any non-default imports.

[synctext]

Documentation of related work.
We need to be aware of the diversity of initiatives in our area. For instance, with Clio, you can develop modules and host them, so you can import them via network from another server. You can essentially build microservices as clio modules and build up decentralized systems. All NodeJS and JSON based, leaning firmly towards minimizing developers time, not raw performance.
IPv8 communities need the same abstract API docs and tooling.
https://docs.clio-lang.org/learn/official-tutorial/hosting

[synctext]

Documentation of related IPv8 work. Content for Phd thesis intro? @qstokkink

Governance entity	Delft University	Protocol Labs	real-logic.github.io
Top layer	Tribler/VLC	Filecoin	Agrona data struct Lib for finance
Middle	Trustchain/Libtorrent	IPFS	Simple Binary Encoding for any format support in financial markets
Networking	IPv8 {modules}	Libp2p pub/sub, STUN/TURN, I2P,	Aeron efficient reliable UDP unicast, multicast with low latency and high throughput

[synctext]

blockchain-building framework

https://polkadot.network/technology/ Another full-stack, but almost really works now approach. Stunning documentation! Includes all our main ideas of parallel interlinked chains. But then explained clearly. Quite complex approach. Plus identity and file sharing incentives. Well know solution and used by several teams.

[synctext]

Related work: "Azimuth is a general-purpose public-key infrastructure (PKI) on the Ethereum blockchain, used as a decentralized ledger for what are known as Urbit identities, or simply identities."