Atlas Product Onboarding [Q4 2021]

[bedeho]

Document Purpose

The purpose of this document is to attempt to summarise the current status of work on Atlas, in a broad sense, for the benefit of all recent incoming and near term incoming team members, and in particular a new product manager to be onboarded very soon.

Background

The Atlas product size and team has grown considerably. The product as matured significantly feature wise, and is now being used by a small but passionate community of project contributors, who primarily use it as an information sharing and community building platform for the Joystream DAO. The product has, until now, focused quite heavily on reaching a certain baseline level of familiar video platform functionality, which has served as some level of insulation from the errors that inevitable come from not being more closely connected to the user base we are building for, however, going forward, we expect to build more and more totally new and unfamiliar product features. These factors have raised the need to make sure

• we have greater emphasis on representing the current and future users we are building for in our processes.
• we are executing on our deliveries in a timely and effective manner.
• we maintain internal team processes that ensure effective and pleasant process for our all members.

Atlas

Introduction

Atlas is the web based consumer product exposing the content consumption, publishing, monetisation experience made possible by the Joystream system. A production version of this application is available here

https://play.joystream.org/

Reflecting the version of our system as reflected by the Sumer network, described here

https://www.joystream.org/sumer

Assets

The product is built entirely in the open, so all of our assets are public and transparent, and all assets exists under permissive licenses.

• Handbook: https://joystream.gitbook.io/joystream-handbook/
• Source code:
  • Atlas app: https://github.com/Joystream/atlas
  • Orion: https://github.com/Joystream/orion
  • Query Node: https://github.com/Joystream/joystream/tree/master/query-node
  • Hydra: https://github.com/Joystream/hydra
  • Colossus: https://github.com/Joystream/joystream/tree/master/storage-node
  • Argus: not yet on master
  • Blockchain: https://github.com/Joystream/joystream/tree/master/runtime
• Designs: https://www.figma.com/files/team/988759271589293564/Atlas?fuid=632538047582453662
• Planning
  • Gantt: https://prod.teamgantt.com/gantt/schedule/?ids=2757545#&hide_completed=true&ids=2757545&user=&custom=&company=&date_filter=&color_filter=
  • Tech Kanban: https://github.com/orgs/Joystream/projects/42

Terminology

When the name Atlas is used in this document without additional qualification, it refers to the core digital product artefacts, including source code and design assets, which are open source. It does not refer to a particular instantiation, such as the version running on Jsgenesis infrastructure and distributed through the domain https://play.joystream.org/.

Purpose

Atlas is primarily meant to show current and prospective community members what Joystream can enable as a platform. It, and in particular the deployment of it on the joystream.org domain, is not intended to be a value capturing asset for Jsgenesis the company, unlike the role of youtube.com or the YouTube mobile app for Alphabet. This is a very important, and perhaps unfamiliar, distinction between products in the Joystream project, and traditional startups.

This purpose is served in two distinct ways at two distinct time scales.

1. Demonstrating Capabilities and Community

Now, and in the immediate future, Atlas shows community members and creators what the system can enable in terms of familiar and unique Web3 features. The Jsgenesis instantiation also shows the state of the current creator community and content selection.

2. Template for Innovation

Over the somewhat longer term (mid 2022+), it supposed to serve as a base template for others to build their own product offerings on top of Joystream. This innovation is uniquely unlocked by the open permissionless infrastructure of a blockchain system, and a forthcoming subsystem called the gateway system which provides a built in business-model for developers and entrepreneurs to build their offerings on top of the Joystream system. The innovation will initially likely be in terms of

• user acquisition models,
• verticalised content focus and
• access gating through different monetisation schemes (advertising, subscriptions, etc.),

but then our bet is that people will start introducing deeply differentiated video product experiences that starts to work very differently to the initial baseline Atlas.

Technology

The Atlas web app itself has a relatively conventional web stack for purpose, perhaps with the exception of relying on an external extension for key management, as is customary for web applications in Web3. The much more important aspect worth understanding is all the different infrastructure pieces and roles interact to make Atlas a working product. The architecture diagram below gives a succinct summary of this, and subsequent sections dive into some further detail on each of the pieces outlined in the diagram.

Overview

Joystream Blockchain

Maintainers

• Shamil
• Ignazio
• Mokhtar
• Arsen
• Gabriel

The Joystream blockchain is the technical centrepiece, serving as both the data, social and economic infrastructure layer for the overall system. It is a standalone Proof-of-Stake (PoS) blockchain (hence not a set of smart contracts, as is often the case for other systems). The native token, or currency, of the system is called JOY, is often denoted by the currency symbol $JOY. A distinguishing feature of the system is that it is upgradable, meaning that it has a built in capability for the community to upgrade the rules that govern the system. It has many interconnected subsystems, but the most important parts w.r.t. Atlas are

• Membership System: Described here https://joystream.gitbook.io/joystream-handbook/
• Data Directory: Described here https://joystream.gitbook.io/joystream-handbook/subsystems/storage/data-directory
• Content Directory: Described here (old) https://joystream.gitbook.io/joystream-handbook/subsystems/content-directory

The Joystream blockchain is written on top of a blockchain framework called Substrate, which is a Rust based SDK for developer to build their own blockchains. This means we do not need to deal with very low level concepts, such as peer-to-peer networking, the consensus algorithm or state database. We only need to define the business logic of our specific use case, and this is done in Rust.

Query Node

Maintainers

• Andy
• Leszek

When applications need to read the state of the blockchain in some way, for example to see all video channels that exist, this mostly does not work well if you attempt to do so my directly connecting to a validator node. Validator nodes, who are involved in the business of validating the integrity of the system as it evolves, do not have any search indexes that allow for efficient queries of the current state of the system. For this reason, the query node is an intermediate query node layer which maintains various explicit query state that can be accessed through a GraphQL API. Our particular query node is built on top of a framework we built, called Hydra, described here

https://www.joystream.org/hydra

A critical issue with the query node infrastructure is that any changes to the blockchain require that infrastructure developers have to define the APIs, through schemas called input schemas, and processing code that takes blockchain ledger events and transactions, and updates the underlying query state of the node, these are called mappings. Writing such input schemas requires understanding both the underlying business logic of the part of the blockchain you want to expose to applications, and also what the requirements are of the particular application(s) you have in mind. This capability currently does not exist within the Atlas team itself, and so far they have relied on infrastructure developers to support them.

Currently, there is only a single query node deployed in every test or production deployment for Joystream networks, and it is operated by Jsgenesis in those cases. The likely future scenario is that running a query node will get bundled into the activities of a Gateway, as what particular schemas and queries are relevant to a given product instance may not apply to another. An alternative could be to make it a paid DAO role, or to try to rely on a third party middleware protocol like Subsquid eventually will become.

Orion

Maintainers

• Klaudiusz

Orion is a service for maintaining viewing statistics for content in Joystream, it is still in its very early stages, and likely will become the future Gateway node. Its also highly likely that it stops becoming a primary data store for the this viewing data, as this really needs to live in a shared public data layer, rather than a trusted server. Read more about this in The Missing Shared Data Layer.

Storage Nodes - Colossus

Maintainers

• Shamil

Storage nodes are responsible for long term archiving of user assets, such as videos, avatars, etc. The reference implementation for the Joystream storage node protocol is is called Colossus. Importantly, these assets are not stored on-chain, that is by validating nodes. A given asset is stored by a small subset of storage nodes, for redundancy, and each storage node only stores a small share of the overall set of assets. They are operate by community members through on-chain roles in the storage working groups, and are policed and directed by on-chain authorities, i.e. the lead and the council. These nodes also accept initial uploads from users when they publish these assets.

Bandwidth Nodes - Argus

Maintainers

• Leszek

Bandwidth nodes are responsible for distributing user assets to end-users at scale, similar to how a CDN works. The reference implementation for the Joystream bandwidth node protocol is called Argus. Bandwidth nodes replicate data as needed from storage nodes, and maintain a local cache based on local request statistics. Currently there is no authentication layer, hence anyone can download anything, without payment or credentials. They are also operated by community members through on-chain roles in the distributor working groups.

Note: These nodes are actually only going into production in Giza release, which is imminently going to be release.

Polkadotjs Extension - Key Manager

Maintainers

• External

A key design requirement of Atlas was that the user had full custody of all digital assets, meaning that no third-party (even the future Gateway) should have access to any of the keys that control the $JOY, NFTs, Social Tokens, channels or memberships of the user. This is not the only way to build apps on top of Joystream, someone could - and likely will, build a fully custodied and friction minimised version.

With this constraint in mind, the only robust way to allow users to manage keys, while also using the app in the browser, is to rely on a browser extension for key management. For the Ethereum ecosystem, Metamask plays this role. Since Joystream is built on Substrate, we use the extension Polkadotjs extension, which is the equivalent for this ecosystem. It is not a full wallet, for example it does not have the ability to speak to a validator node, so ti cannot send transactions or display balances, it only allows for holding keys and signing transactions.

Product

Status

The main product features that currently work in production are

• Viewing content.
• Searching for content.
• Browse for content organised by major categories.
• Following creators and receiving.
• Creating, publishing to and managing channels
• Featured content: there are certain parts of the application where the operator of the application can choose what content should be displayed, and we are going to allow the community to select what content will be displayed in the instance of Atlas operated by Jsgenesis.

Storage and Bandwidth

The core tech infrastructure that powers the storage of all media assets, such as images and video media, as well as distribution of that media to end-users, happen through systems that are operated purely by community members who are coordinated by the testnet blockchain and some incentive schemes from Jsgenesis. Those operators run custom written node software that performs the required tasks, and they select what infrastructure to run those nodes on (self-hosted, AWS, Google, their house!). This means that this infrastructure regularly experiences various performance issues, both because the technology is immature, and also because it is today operated by enthusiasts with limited time, experience and incentives. Right now are we in the process of releasing a new network, the Giza network, which will be the last major overhaul of the tech side of this infrastructure before the mainnet, described here

https://www.joystream.org/giza

Gateways

Gateways are a future infrastructure service, operated by what will be called gateway operators. They are responsible for the last mile between the blockchain economics and infrastructure and a mainstream consumer audience. This is not a part of the system which is currently fully designed or built. Briefly put, the role of a gateway is to achieve the following goals

1. Tokenomics: Connect the native token model of the blockchain of the DAO with value derived from a consumer audience that cannot be expect accept onboarding costs and complications of dealing with $JOY the asset as a precondition to using Joystream. This is done by having the gateway finance the cost of access on behalf of users on the gateway, and then being free to monetise their user base however they see fit (ads, cc payments, cc subscriptions, other crypto)
2. Confidentiality: Deliver last mile product features which critically depend on using confidential user data, such as recommendations, search, viewing history, follower relationships, etc.
3. Glue: Deliver the product features which are either too hard to prioritise facilitating through decentralised infrastructure accountable to the DAO directly. So this is stuff like post-processing of videos, like transcoding or preview image extraction, translation, etc.

Currently, the Orion node, which holds viewership statistics and content featuring policy information, is expected to become the precursor for the gateway node. A good introduction to some of these topics can be found in this video

https://play.joystream.org/video/1266

The Missing Shared Data Layer

An open question which remains is what the open shared data layer for traffic information should look like. Currently, a very naive approach is taken to this with the Orion node, but it has obvious problems around verifiability, permissioning, fault tolerance and availability.

Roadmap

Here we attempt to summarise features which have been considered, or are, possibly within scope to be built by Jsgenesis, but without any prioritisation or timeline.

Mainnet

It is yet to be determined what is an absolute must feature set for mainnet launch, which is expected Spring 2022.

Baseline

These features are considered baseline, in the sense that they are familiar from existing offerings, such as YouTube.

• Consumer accounts: There is no notion of any user accounts on a server, as is conventionally the case with platforms, which would hold private user data. Private data includes things like who you follow, your viewing history, search logs any metadata about your search
• User playlists & series: There is no ability for channel owners to publish playlists, as ordered compositions of videos from any channel, and series, which also have seasons.
• Transcoding: There is no ability for uploaded videos to get converted into a range of different resolutions and encoding formats, making it easier for users to consume the content in a way which is suitable for their device and product.
• Scrubbing Previews: There is no ability to generate image extracts from regular intervals in a video so as to provide content previews while scrubbing.
• Recommendations: There is no way to recommend to the user what they should watch which is based on either their own or other consumption statistics. This is required in order to for example suggest new a new video when playback of one ends, or to show related videos.
• Better search: There is no way to search based on anything except a very bad full text matching of the search string against, video titles and descriptions. Sophisticated search will be sensitive to many more parameters, including popularity and consumption statistics of candidate matches.
• Subtitling: There is no way to display subtitles in content, or allow control of. As a result of the lack of post-processing, there is also no way to have subtitles automatically extracted.
• Comments & Likes: There is no way for users to comment on videos or like videos.
• Creator Dashboard: There is no way for a creator to get rich information about how their content is consumed.
• Embeddability: There is no way for users to embed playback of a video on third party websites or social media.
• Analytics: There is no way for Atlas site operators to monitor basic traffic information, or collect rich usage information based on tracking individual users or cohorts.
• Image Safety: Image Safety joystream#2835
• 1-Click deployment: There is no way to deploy a full Atlas site with a single click.
• Managing Categories: Idea: Make it super easy for Atlas operator/community to manage set of categories in Discover view #1659

Web3 Specific

These are features uniquely unlocked by Web3 infrastructure.

• Consumer memberships: These are currently under development
  these allow consumers to hold digital assets, like NFTs and Social Tokens, as well as use public social features, like
• Video NFTs: The ability of creators to issue single edition NFTs for their videos, as well as having a built in market place for buying and selling such NFTs. Initial work has already started here Video NFT Design Brief #1329.
• Creator Social Tokens: The ability of creators to monetise their channel by sharing their future channel revenue with their most passionate fans, as well as give them access to unique content.
• Transparent moderation The ability for anyone to see the moderation history of any content, and for moderation to occur in public.

This video has some useful information on these topics

https://play.joystream.org/video/275

Team 🧑‍💻

Name	Role	Association
Kuba Mikołajczyk	Design Lead (DL)	Netguru
Adam Ruthendorf-Przewoski	Design System Lead (DSL)	Netguru
Nina Ryńska	Product Design	Netguru
Tomasz Czechowski	Product Design	Netguru
Klaudiusz Dembler	Front-End Tech Lead (TL)	Jsgenesis
Diego Cardenas	Front-End Engineer	Jsgenesis
Rafal Pawlow	Senior Engineer	Elpassion
Bartosz Dryl	Front-End Engineer	Elpassion
Dmitry Meltsov	Product Manager (PM)	Jsgenesis
Bedeho Mender	Product Owner (PO)	Jsgenesis

Processes

https://github.com/orgs/Joystream/projects/42

Background

For reference, it's worth consulting and old product management proposal which we have tried to follow with mixed levels of commitment

#777

Meetings

Weekly Planning

There is a Monday morning meeting, typically at 11 am CET, where PM, PO, TL and all designers join. Agenda is prepared in advance by PO, anyone is free to submit agenda points over RocketChat, and a post-meeting summary, prepared by PO, is added to this long-running public issue

#818

Tech Synch

There is a Monday morning tech synch meeting for only the tech team, lead by TL.

Work Management

Overall management of ongoing work, which is almost always organised into sprints that last a few short weeks, including who does what, is done in this Gantt which is updated during the weekly planning meetings

https://prod.teamgantt.com/gantt/schedule/?ids=2757545#&hide_completed=true&ids=2757545&user=&custom=&company=&date_filter=&color_filter=

Low level tasks, to be performed by a single individual, are tracked using Github issues, and everything is tracked in one Kanban

https://github.com/orgs/Joystream/projects/42

Communication

Everyone on the team is supposed to be available on RocketChat on short notice during normal working hours.

• atlas@RocketChat: the primary communication general topic & coordination discussionc hannel, on Jsgenesis RocketChat.
• atlas-design@RocketChat: the primary design discussion channel, on Jsgenesis RocketChat.
• atlas-dev@RocketChat: the primary development discussion channel, on Jsgenesis RocketChat.
• atlas-standup@RocketChat: the developer async standup channel, on Jsgenesis RocketChat.
• atlas-testing@Discord: the community testing channel, on public Discord. Link here

Product Design Process

The development process for new major functionality in Atlas typically follows something like this kind of linear process

1. High level requirements are written by PO for blockchain or other infrastructure: Requirements: Video NFT joystream#2481
2. Detailed specifications are written by blockchain or infrastructure developer: vNFT: Auctions description joystream#2610
3. Feature is implemented: VNFT auction joystream#2512
4. Atlas design brief is prepared by PO: Video NFT Design Brief #1329
5. Design work is broken down into one or more sprints, and work within sprints are assigned to a designer. Each designer iterates on work in public on a long running design issue, like this
   Member Account : Design #1496
   Iterations are communicated primarily through Loom videos that can be consumed asynch, replayed and shared with community.
6. Feedback is provided on each iteration by PO and TL, focusing primarily on UX/UI and tech feasibility, respectively.
7. Designer takes finalised designs and collaborates with DSL to create fully documented designs for developers.
8. TL plans execution, possibly restricting scope, estimates delivery date and assigns work to developers.
9. Progress is tracked and reestimated weekly during weekly planning meetings.
10. During development, and finally before delivery, designer provides quality assurance on final product.

Notice: there is no systematic involvement of users, before or after delivery, not even using passive methods, like analytics or server-side error logging.

[dmtrjsg]

@dmtrjsg to find a place on knowledge base to host such docs. Notion is a candidate