I will introduce the distribution of project source code files, and then introduce the main core modules one by one.
The CYFS project directory is expanded as follows:
/--CYFS
|--doc // formal design documents
|--scripts // some scripts for build/publish...
|--src // the source code, We will specifically introduce the source code directory
|--... // other files for solution: .gitignore, .eslintrc, package.json, .etc
All modules are divided into several categories and collected into corresponding directories in src
:
-
3rd The main code comes from an external project, but with a few custom modifications.
-
component Basic functional modules, most of which belong to this type, each implements its own functions, is referenced by other runnable instances, and has interdependence internally.
If your module will be depended on by other modules, it should be collect in
component
. If part of your module will be depended on, you should split the part into a separate module.As examples:
cyfs-base
: It's the most basic module,Almost all modules depend on it.cyfs-core
: It defines all normalized objectscyfs-util
: Some auxiliary modulesI will introduce them in other issues.
-
meta The modules for
MetaChain
, it's just a conceptual version? I haven't dabbled in this part yet, I will introduce it in future. -
service Services in
CYFS
system, They run as independent processes on variousCYFS
devices.I will introduce them in other issues.
-
tests
Some test cases and sample programs. We can help to supplement and improve, it's a good entrance to enter the
CYFS
world, and contribute to the cause of Web3. -
tools
Some tools to help users/developers to use the
CYFS
. We can design more useful tools. -
misc
Others.
- Devices
As introduced here, there are 2 types of devices in CYFS for users and developers. They are OOD(gateway) and runtime.The relationship is as follow:
Zone A | <--> | Zone Other | ||||
---|---|---|---|---|---|---|
runtime A1 | <--> | OOD(gateway) A | OOD(gateway) O | <--> | runtime O1 | |
runtime A2 | <--> | <--> | runtime O2 | |||
runtime ... | <--> | <--> | runtime ... | |||
runtime An | <--> |
- Services
In a zone, several services is running on the OOD always. These services are running depend on the gateway
, the gateway
is also a service, It is equivalent to nginx for http, receive and forward the requests to the target services.
gateway | <--> | app-manager |
<--> | chunk-manager | |
<--> | file-manager | |
<--> | ... |
- app-manager: It manages the DecApps running on CYFS, install,uninstall,update,start,restart,stop,deamon.
- chunk-manager: The chunk-manager binary is the first experimental NDN implementation (completed in April 2020), which is quite different from the current NDN implementation. Currently this component is only used for Service and DecApp package download.
- file-manager: Similar to chunk-manager, it is the first experimental NON implementation (completed in March 2020), and is currently only used for Service and DecApp package downloads.
- ood-deamon: It will restart the gateway when the process is dead. It is an independent service that does not depend on gateway.
- ood-control: It is not a standalone binary service and is used by programs like ood-daemon or ood-installer. The purpose is to complete stack activation-related functions via the http interface.
By the way, the server of a DECApp
is also running as the services on OOD. They listen to requests from clients through the gateway and respond.
Thanks @weiqiushi for the correction
This is the most basic component, and almost all other components depend on it. It implement the follow functions:
- The basic types or structures:
-
AccessString: Structures for ACL.It refers to the design of file system permission in linux.It divides users into 6 groups:
- CurrentDevice
- CurrentZone
- FriendZone
- OthersZone
- OwnerDec
- OthersDec
These groups can be combined arbitrarily to get the target group that the user wants to configure permissions.
And we can configure 3 permissions:
- Call
- Write
- Read
-
Base36: Transform the BLOB with A human-readable string ignoring case. The string will be used case-insensitively, .eg:
- Address Bar in browser
- FileSystem of windows
-
channel: This name is used in many occasions. Here it defines the operating environment of the CYFS system. Currently, the CYFS system has 3 operating environments:
- nightly: The integrated environment for developers.
- beta: The public test environment for users.
- stable: The formal version release environment. Different environments are independent of each other, they are connected to different MetaChains and SNs, and cannot be interconnected.
-
endpoint: Encapsulation of IpAddr, It's used to connect into the network.
-
BuckyError: Define the Results for CYFS.
-
Name: As the domain for http, we can register a readable nick name for any object.
-
paths: The req-path(similar as the URI for http) or the path in
RootState
-
ports: The listening ports when the
CYFS
running. -
protocol_fields: The
HTTP
header fields,HTTP
is used atRPC
between thestack
(runtime/gateway) and the other services orDECApp
. -
time: Defines various standard timestamps, and the methods to convert between them
- The framework for encode/decode:
- RawEncode/RawDecode: Encode with binary.
- Protobuf: Encode with protobuf
- JSON: Encode with JSON
- Encapsulation of the crypto library:
- AES
- Hash: Only for SHA256?
- KeyPair(PrivateKey/PublicKey): RSA1024/2048/3072, and SECP256K1.
- Signature/Verify: Signature with the KeyPair
- The 14 standard objects:
- Action: It seems a empty object, Maybe the design draft may not be finalized yet.
- AppGroup: It seems a empty object, Maybe the design draft may not be finalized yet.
- Chunk: The piece in any binary data. but there is only the
Hash
of the content is included in theObject system
without the content itself. - Device: The object describing a device,it includes all the parameters used to connect each other:
_ Endpoint
_ SN
_ PN
_ Protocol version * And we can give it a readable name.
All the fields is mutable, so, they are saved in
Body
.TheDevice
will be publish on theMetaChain
, Therefore,we should query the latest version fromMetaChain
when we realized that it was no longer effective. - Dir: The object describing a directory in filesystem.It includes some files in different sub-path, the file(even chunk) list will be included in the
desc
if the size is little enough, otherwise the content(file/chunk list) will be included in the body or in another file.If the content is not saved indesc
, there should be a hash to verify the authenticity of the content. - File: The object describing a file in filesystem.It includes 3 fields:
- File length, immutable, saved in
desc
- File hash, immutable, saved in
desc
- The contained chunks, immutable, saved in
body
, So,we must verify the file hash instead of the chunkids.
- File length, immutable, saved in
- Diff: The delta between 2 file, it's usefull to describe different versions of the same file.
- Group: It's used to describe an organization,The main fields is the administrators and members, they are all mutable,we should update it through the
MetaChain
or other synchronization methods.There is a issue for it. - People: Describing a natural person, There are only 3 fields currently:
- name
- icon
- ood_list
All the fields is mutable,and the new version will be pushed to the
MetaChain
.
- Relation: It seems a empty object, Maybe the design draft may not be finalized yet.
- Tx: It is the concept of tx in the
blockchain
, describing a transaction itself, the main field includes:- caller
- message: transaction details
- gas
- UnionAccount: Joint account of both parties(Provider and Consumer) to the transaction.They transfer the initial coin and set their respective shares,As the contract progresses, the share is revised.All the updates must be signed by both, and executed on the
MetaChain
.Both parties are free to withdraw their share of coins.TheUionAccount
object only describes the account itself, not the current state of the account. It has 3 fields only:- left: party to the transaction
- right: another party to the transaction
- service type: type of the transation
How to update/query the state of the account?Maybe we can find it on the
MetaChain
- Contract: Describe the commitment of both parties to the transaction (price, quality, etc.).
- ProofOfService: Describe the confirmation of the final transaction quantity and quality.Provider and consumer must sign on the contract and receipt.
When there is a dispute in the transaction, one party presents the
ProofOfService
signed by the other party and submits it toMetaChain
for arbitration, and the transaction can be completed according to the contract. The design of the specific service contract is complicated and different from each other, so I need to elaborate on it in another document.By the way,The contract seems a temporary design. - ObjectMap: The container for objects, The
ObjectId
is calculate with all items, it will change when any item is updated, and we can regard the two versions before and after modification as two different objects.TheObjectMap
can express three structures:- Map<Key, ObjectId>
- Set
- Diff: The delta between 2
ObjectMap
, it can be stored in afile
.
- Other structures for object
- Area: Every object should belong to a different
Area
, and accept the legal supervision of the correspondingArea
. TheArea
field is set when the object is generate, and is encoded in header 40 bits of theObjectId
. - NamedObject: The basic trait for any
Object
. - TypelessObject: It's a shell for any Object.It collects the common fields,and it includes the object content in the form of buffer.
- AnyNamedObject: It's a abstract shell for any Object. It can be generate from any object, and we can decode it to get the original object.It is often used with
TypelessObject
. - NDNObject: It's a shell for all unstructured data(
Dir
,File
,Diff
) - ObjectLink: **I don't understand its exact design intent.**Maybe it's a reference to another
Object
. - ObjectSigns: It describes the signature information of an object, and the current signature independently signs
Desc
andBody
respectively.I think there will be some risk,You are welcome to join the discussion. - UniqueId: Binary with a length of 16 bytes, We can fill the
ObjectDesc
with the randomUniqueId
when we need to distinguish the object with same fields inObjectDesc
:- No fields in
ObjectDesc
- The fields in
ObjectDesc
is same, but theObjectBody
will be update in different time.They are 2 different objects.
- No fields in
cyfs-base-derive
Some usefull macro is implement in this crate:
- RawEncode
- RawDecode
- ProtobufEncode
- ProtobufDecode
cyfs-base-meta
The basic structures for MetaChain
.
I mentioned 4 object categories in previous part. The Core
object is collected in the crate cyfs-core
.Many object types have been collected, most of those support the functions of the current CYFS
system.
- common
-
Text
It's a common
Object
to describe aString
.You can save a short(<=31Bytes) string in theObjectId
immediately building with theObjectIdDataBuilder
.Otherwise, you can use theText
object.It contains 3 fields:
- id: immutable
- header: immutable
- value mutable I can't tell the design intent from the field name, maybe it need to rename or redesign? You can flexibly use these three fields according to your needs, for example:
- Make a mutable
Text
,fill theid
with key, and file thevalue
with the content,header
for other immutable annotation. - Make a immutable simple
Text
,fill theid
with the content,header
for immutable annotation, andvalue
for mutable annotation. - Make a immutable large(>=64KBytes,Actually less for the header of
Desc
)Text
, fill thevalue
with the content, and fill theid
with the hash of content to verify thevalue
,header
for immutable annotation. - Other usage
-
Storage
It's a common
Object
to describe a binary buffer(Vec),it can describe both the immutable and mutable buffer. It contains 3 fields:- id: immutable,set by the creator.
- hash: immutable,the hash of the buffer,set
None
if the buffer is mutable to avoid theObjectId
is changed when the buffer is changed. - value: mutable, the content of the buffer.
-
app
It supports the running of the system service
AppManager
,Various related entity objects and behavior objects are defined here.
- DecApp: Describe App Details, the main fields as follow:
- id: immutable, UniqueId?
- source: mutable, what?
- icon: mutable
- desc: mutable, the description for the App
- source_desc: mutable, what?
- tags: tags for the App
- AppCmd: Describe the control commands for APP, it has 2 fields:
- app id
- command code:
- Add: add the app to the app list.
- Remove: remove the app from the app list
- Install: install the app in my zone
- Uninstall: uninstall the app from my zone
- Start: start the app
- Stop
- SetPermission: update the permission for the app
- SetQuota: config the limit amout of resource for the app
- SetAutoUpdate
- AppCmdList: Use this object when we need to execute multiple commands sequentially.
- AppList: APP list organized by category, the main fields as follow:
- category: immutable, the name of the APP category
- id: immutable, set by user to manage the APP category when the AppList is created, What's the different with category field? It's unique?
- list of applications: mutable,The latest version is maintained by OOD in the zone.
- AppLocalList: What does the
Local
mean? - AppLocalStatus: What does the
Local
mean? - AppManagerAction: Is it related to the implementation logic of AppManager?
- AppSetting: Configuration for the app.
- AppStatus
- id: immutable,DecAppId.
- version: mutable
- status: mutable, Why is it a boolean? The latest version is maintained by OOD in the zone.
- DefaultAppList APPs that are automatically installed after the CYFS system is installed.
- group
There is a issue for it.The core objects for group
is implemented in cyfs-core
.
We use the blockchain structure to record the state evolution process of the Group
.The consensus algorithm is Hotstuff
.
- GroupProposal
The object that prompts the infomation of group to update.It will answer the
DECApp
3 questions:- Who want to update?
- What does he want to update?
- How to update? The fields is follow:
- rpath: immutable,
rpath
the blockchain is running on.there is a blockchain is running for eachrpath
of thegroup
. - method: immutable,the method provided by the
DECApp
,it will be called with theproposal
. - params: immutable, the parameters for the method, it's defined and parsed by the
DECApp
. - meta_block_id: immutable,the highest block on
MetaChain
currently,Indicates that this proposal was made after the block onMetaChain
. - effective_begining: immutable, when the proposal will effective.
- effective_ending: immutable, when the proposal will invalid, The proposal should be discarded if it has not reached consensus after effective_ending.
- payload: mutable, A buffer reserved for
DECApp
, theDECApp
guarantees its correctness. It's usefull when the params is large.
- GroupQuorumCertificate
The certificate to the
blocks
forHotstuff
.It has 2 types:- QC: the quorum certificate for the previous block.
- block_id: immutable,current blockid.
- prev_block_id: immutable,the previous block will be proven.
- round: immutable,the round of the previous block.
- votes: immutable,the set of signatures for the
QC
.- voter: the signer
- signature: the signature
- TC:the timeout certificate for last round.
- round: immutable,the timeout round number
- votes: immutable,the set of signatures for the round
- voter: the signer
- high_qc_round: the max round of the block which has collect the
QC
. - signature: the signature
- QC: the quorum certificate for the previous block.
- GroupConsensusBlock
GroupConsensusBlock
is used to describe the content of each block. The fields is follow:- rpath: immutable,
rpath
the blockchain is running on.there is a blockchain is running for eachrpath
of thegroup
. - body_hash: immutable, the hash of
GroupConsensusBlock
, is used to verify thebody
. - result_state_id: immutable, the
ObjectId
of the result calculated by theDECApp
, - height: immutable, the height of the block.
- meta_block_id: immutable,the highest block on
MetaChain
currently,Indicates that this block was constructed after the block on MetaChain. - round: immutable,the round to vote for the block,used by
Hotstuff
. - group_chunk_id: immutable,the version of
Group
the block is depend on. - proposals: immutable,save in
body
, verify bybody_hash
,the set of the proposals(proposal/result/receipt/context) collected by the block. - qc: immutable,save in
body
, verify bybody_hash
,the quorum certificate for the previous block. - tc: immutable,save in
body
, verify bybody_hash
,the timeout certificate for last round.
- rpath: immutable,
-
nft
NFT is a popular application scenario in recent years.CYFS collect and defined its own NFT objects.In theory,any
named data
isNFT
inCYFS
,but most NFTs are files in fact.So NFT is limited to file types.- NFTList Is a file list:
- nft_list: immutable, file list.
-
trans
I don't know this part?
- TransContext
- zone It's the manager of the devices belonging to the same owner.
- Zone
The field is follow:
- owner: immutable
- ood_list: mutable
- known_device_list: mutable
How to maintain the latest version of the
Body
?
CyfsStack integrates all functional modules provided by the CYFS system.Any service or DECApp
will call the CyfsStack
to use the CYFS
.
There are 2 solutions:
- All services and
DECApp
are executed as plugins in the same process as CyfsStack. This has several disadvantages:- A dynamically loaded plug-in system, the developers have to spend more to learn it.
- All services and
DECApp
s will be affected by exceptions thrown by anyDECApp
, that is a disaster. - A
DECApp
can easily access the memory of another.
- Every service or
DECApp
is executed in a new process, they call theCyfsStack
through theRPC
. This solution is selected, and it's more friendly:- The developers can develop
DECApp
with their familiar technology. DECApp
s can run independently in separate processes without interfering with each other.They can even run in separatedocker
s to keep resources isolated. It also has several disadvantages:RPC
in different programming languages is needed.but we can use it immediatly while anyone has complete it.- Performance will be lost in
RPC
.
- The developers can develop
- Protocol for
RPC
- The communication protocol of RPC adopts
HTTP
based onWebSocket
orBDT
. - There is some common headers defined in
cyfs-base/src/base/protocol_fields.rs
.
- Framework of
RPC
There are client and service for RPC
, the service is CyfsStack
, and the client is SharedCyfsStack
.
client | service |
---|---|
Implement in SharedCyfsStack | Implement in CyfsStack |
Multiple instances | Only one instance for one device |
Open in any DECApp process | Start with gateway(for OOD) or cyfs-runtime(for other devices) |
Public for developers of DECApp | Invisible for developers of DECApp |
Module named by cyfs-${function-name}-lib | Module named by cyfs-${function-name} |
- Data flow diagram
The data will flow from the DECApp
to the processor in service.
- It will go through an indeterminate count of forwarding processes..
- The forwarding target may be other devices, or different modules on the current device.
- Each forwarding process has a input to receive the data, and a output to send the data except the initiator with only output and the ending processor with only input.
- The inputs are usually named in the form of
${function-name}InputProcessor
,and the parameter withdata
is named in${function-name}InputRequest
. - The outputs are usually named in the form of
${function-name}OutputProcessor
,and the parameter withdata
is named in${function-name}OutputRequest
. - The forwarding process are usually named in the form of
${function-name}InputTransformer
or${function-name}OutputTransformer
.
- The inputs are usually named in the form of
It's flow as follow:
graph TB
subgraph DECApp
User-->Interface[SharedCyfsStack: function with FunctionOutputRequest]
end
Interface-->FindTheTarget
subgraph SharedCyfsStack
FindTheTarget[find target device]-->RPC.send[RPC.send]
end
RPC.send.->RPC.recv
RPC.send.->OtherDevice
subgraph CyfsStack
RPC.recv-->ObjectHttpListener
ObjectHttpListener-->FunctionRequestHandlerEndpoint[FunctionRequestHandlerEndpoint.call]
FunctionRequestHandlerEndpoint-->FunctionInputProcessor[FunctionInputProcessor.call_function]
FunctionInputProcessor-->FindNextProcessor[output_processor = get_forward]
FindNextProcessor-->InputOfOtherModule[input_processor]
InputOfOtherModule-->NextN[next...]
NextN-->Ending[ending_input_processor.call_function]
end
FindNextProcessor.->OtherDevice
subgraph OtherDevice.CyfsStack
OtherDevice[Other CyfsStack in different device]
end
OtherDevice.CyfsStack-.eq.-CyfsStack
We can integrate our own modules into the CYFS protocol stack according to the above framework.and you can also do it follow the issue step by step.
- cyfs-stack
It implements the CyfsStack
and all processor
s to route the data for each module.It's inaccessable immediatly for DECApp
, so it will never be depended on by any DECApp
.There are some directories named in ${function-name}
or ${function-name}_api
.They are implemented with the function module:
${function-name}
: implement some basic structure, it's accessable for other modules.${function-name}_api
: it is the specific call routing logic of the function module.
- cyfs-lib
It implements the SharedCyfsStack
and some interface for the default function modules.It's the entry interface for DECApp
to call the CyfsStack
.So,most DECApp
will depend on this crate.
If you are extending new functionality to CYFS,It's suggested to make a new crate named in cyfs-${function-name}-lib
to instead of implement in cyfs-lib
,you can export a interface to initialize your module with the SharedCyfsStack
.
- cyfs-stack-loader
There are so many parameters to initialize the CyfsStack
, You are suggested to initialize the CyfsStack
with cyfs-stack-loader
if you want to initialize yourself, for example:
- Debuging your extended functions for CYFS.
- Make a tool like the
zone-simulator
. But you should only open theCyfsSharedStack
if you are making aDECApp
based onCYFS
. it will connect to yourCyfsStack
initialized bycyfs-runtime
orgateway
.