Skip to content

Latest commit

 

History

History
551 lines (360 loc) · 25.5 KB

4.Run_and_debug_your_own_message_service.md

File metadata and controls

551 lines (360 loc) · 25.5 KB
Raw

Introduction

In this chapter, you will learn how to run and debug your own messaging service and master the principles behind it.

How DEC Service works end-to-end

The CYFS protocol is an overall upgrade to the HTTP protocol, and will try to maintain the approximate semantics of HTTP. The core process is:

CYFS DEC App <--cyfs@http--> cyfs-runtime <--cyfs@bdt--> gateway <--cyfs@http--> CYFS DEC Service

The cyfs@bdt protocol actually running in the network will not be used directly by the DEC App client and server. From the developer's point of view, both the front-end and the back-end only deal with the HTTP protocol with special headers. This design makes the implementation details of the cyfs@bdt protocol transparent to application developers, BDT can have the opportunity to continue to iterate (BDT is still a young protocol), and it also reduces the learning and use threshold for developers.

The cyfs@http protocol will be used directly by DEC App developers, so its design is concise and easy to understand and relatively stable.

Further understand the GET, PUT, CALL series protocol semantics of CYFS

GET protocol process

The GET protocol is mainly divided into 3 steps in the data flow process:

  1. Step1: CYFS DEC App (browser) <-> cyfs-runtime@local (cyfs@http)

This is the most common request, so its interface logic is a trivial HTTP GET request to the local cyfs-runtime. According to this design, when cyfs-runtime is bound to the local port 80, if the user configures o in HOST as 127.0.0.1 (or the local virtual IP bound by cyfs-runtime), it can be used directly in traditional browser http://o/$ownerid/$objectid opens.

  1. Step2: cyfs-runtime <-> gateway (cyfs@BDT or BDT)

The BDT protocol is currently transparent to applications, so we reserve the opportunity to improve performance based on application practices. For example, a special BDT protocol message can be customized for NamedObject GET. Using HTTP@BDT Stream is currently the most stable implementation.

According to the above design, after this layer establishes the correct BDT Stream, it only needs to forward the HTTP request as it is. Because BDT has its own identity, the cyfs-from field in Req and the cyfs-remote field in Resp are deleted to reduce traffic usage.

  1. Step3:gateway <-> DEC Service (cyfs@http)

Under normal circumstances, DEC Service should not HANDLE NamedObject's GET request. The default behavior of gateway is to automatically do a NamedObject lookup and return the result. Under the default behavior, the permission control idea for GET NamedObject is as follows

  • All requests are released in the Zone, if there is no OOD in the Zone, it will try to obtain it from other places. (The ownerid in the request does not have to be equal to OOD's Owner).
  • For requests outside the zone, if the requested Object is not available on OOD, 404 will be returned directly. If there is, judge the Owner of the Object, and return if the Owner is not the OOD's Owner. If yes, the request that meets the following conditions is released: from "Friend Zone"; NamedObject is Public; Valid ContextId (see Context Management for details)

DEC Service can set the GET NamedObject Handler according to the interface in the gateway part of the SDK. The basic process after setting is similar to nginx upstream, the process is as follows:

gateway-req->data_firewall->DEC Service->data_firewall-resp->gateway The gateway will go through the data firewall before forwarding the request to the DEC Service and after the DEC Service completes the processing and generates the Response.

  • We haven't opened GET NamedObject Handler yet.

PUT protocol process

The PUT protocol is mainly divided into 3 steps in the data flow process:

  1. Step1: CYFS DEC App (browser) <-> cyfs-runtime@local (cyfs@http) Reqeust
PUT http://o/$ownerid/$objectid HTTP/1.1
[cyfs-from:$deviceid] // If filled in, it means that the App wants to initiate a request with the specified identity
[cyfs-target:$deviceid] // If filled in, indicate the specific device to be reached
[cyfs-decid:$decid] //Decid of the request.
[cyfs-cache-time:$time] //The time you want to cache
(Body) is the binary code of NamedObject

Response

HTTP/1.1 200 OK //The NamedObject has been cached
[cyfs-remote:$remote-device-id]
[cyfs-cache-time:$time] //Determine the cache time
  1. Step2 cyfs-runtime <-> gateway or cyfs-runtime (cyfs@BDT) After establishing the correct BDT Stream, forward the HTTP request to the target device as it is. Because BDT has its own identity, the cyfs-from field in Req and the cyfs-remote field in Resp are deleted to reduce traffic usage.
  2. Step3:gateway or cyfs-runtime <-> DEC Service / Named Object Cache (cyfs@http) DEC App Set Handler is usually not allowed in cyfs-runtime. Here we discuss the case of Gateway. The general HANDLE logic of gateway is as follows:
  • PUTs from within Zone are accepted by default
  • PUTs from outside the Zone are rejected by default

POST(CALL) protocol process

The POST(CALL) protocol is mainly divided into 3 steps in the data flow process:

  1. Step1: CYFS DEC App (browser) <-> cyfs-runtime@local (cyfs@http) Request
POST http://r/$ownerid/$decid/$dec_name?d1=objid1&d2=objid2 HTTP/1.1
[cyfs-from:$deviceid] // If filled in, it means that the App wants to initiate a request with the specified identity
[cyfs-decid:$decid] //Decid of the request.
[cyfs-dec-action:exeucte | verify]

The Body of a POST Call can carry a set of named objects of the package. But in most cases, it is prepared by DEC Service itself.

Response

HTTP/1.1 200 OK
cyfs-dec-state: complete | prepare | running | wait_verify| failed //This dec is completed, preparing, working, waiting for verification, failed
cyfs-dec-finish : $time //dec completion time (dec will not be executed repeatedly, if it has been completed before, the previous time will be used)
cyfs-prepare : objid1,objid2,objid3 ... // if in the prepared state

Response Body: If the action is to execute, it will return Result ObjectIds. If it is verification, the verification will return the signature of the DEC triplet.

  1. Step2 cyfs-runtime <-> gateway (cyfs@BDT)

  2. Step3:gateway <-> DEC Service (cyfs@http)

The whole process is basically the forwarding of HTTP POST Request and Response as it is, and it is also the main Handle request of DEC Service

Deep dive into the starter of the DEC Service

Getting to know AppManager

AppManager is one of the basic services in the CYFS protocol. Mainly used to install and run DEC App. AppManager runs the process of the DEC App through the docker container, which can run the DEC App more securely, allowing the protocol stack to authenticate the DEC App request and check whether the dec id it carries is forged.

DEC Service starter

Before running the Service, let's take a closer look at the Service Launcher.

  • See src/service/entry/app_startup.ts for the complete code

The entry of the Service startup program is the main function, which mainly completes three steps:

  1. Open the Service log.
  2. Open and wait for Stack to come online.
  3. Register the routing module on Stack.

Enable Service log

Logs allow us to quickly discover and locate problems, which is very helpful for solving online problems. Different operating systems have slightly different storage paths for application logs;

  • mac: ~/Library/cyfs/log/app/<dec_id>
  • windows: C:\cyfs\log\app<dec_id>
  • linux: /cyfs/log/app/<dec_id>

It is very simple to open the Service log based on the CYFS SDK. The code is as follows:

import * as cyfs from "cyfs-sdk";

cyfs.clog.enable_file_log({
	name: APP_NAME,
	dir: cyfs.get_app_log_dir(APP_NAME),
});

Open and wait for Stack to go online

By introducing the waitStackOOD method in cyfs_helper, we can easily open and wait for the Stack to go online. The code is as follows:

import { waitStackOOD } from "src/common/cyfs_helper/stack_wrapper";

const waitR = await waitStackOOD(DEC_ID);
if (waitR.err) {
	console.error(`service start failed when wait stack online, err: ${waitR}.`);
	return;
}

Register the route on the Stack

Use the addRouters method to register routes in batches to improve development efficiency. The addRouters method completes the function of batch registration of routes by traversing the routers object that encapsulates all routing modules. In each round of the loop, mainly complete 2 tasks:

  1. Dynamically set access permissions for the request path req_path, here we set Only open all permissions to OwnerDec (Read/Write/Call) for each request path.
  2. Use the add_post_object_handler method to mount the routing module to the specified request path on the Stack.

The addRouters code is as follows:

import * as cyfs from "cyfs-sdk";

export type RouterArray = Array<{
	reqPath: string;
	router: postRouterHandle;
}>;

async function addRouters(
	stack: cyfs.SharedCyfsStack,
	routers: RouterArray
): Promise<void> {
	for (const routerObj of routers) {
		// set access permission for req_path
		const access = new cyfs.AccessString(0);
		access.set_group_permissions(
			cyfs.AccessGroup.OwnerDec,
			cyfs.AccessPermissions.Full
		);
		const ra = await stack
			.root_state_meta_stub()
			.add_access(
				cyfs.GlobalStatePathAccessItem.new(routerObj.reqPath, access)
			);
		if (ra.err) {
			console.error(`path (${routerObj.reqPath}) add access error: ${ra}`);
			continue;
		}
		console.log("add access successful: ", ra.unwrap());
		// Mount the routing module to the specified req_path
		const handleId = `post-${routerObj.reqPath}`;
		const r = await stack
			.router_handlers()
			.add_post_object_handler(
				cyfs.RouterHandlerChain.Handler,
				handleId,
				1,
				undefined,
				routerObj.reqPath,
				cyfs.RouterHandlerAction.Pass,
				new PostRouterReqPathRouterHandler(routerObj)
			);

		if (r.err) {
			console.error(`add post handler (${handleId}) failed, err: ${r}`);
		} else {
			console.info(`add post handler (${handleId}) success.`);
		}
	}
}

Run and test DEC Service

We recommend developing DEC apps by separating the front and back ends. So far, the DEC Service of the message board has been developed. Next, we publish DEC Service to OOD and debug each function of DEC Service independently.

Compile and package the project

In the project root directory, open a terminal and run the following command:

npx tsc

After the command is executed, you can see the new deploy folder in the project root directory.

  • deploy: the folder published to OOD, which contains all the ts files and compiled js files in the project

Next, copy the src/common/objs/obj_proto_pb.js file to deploy/src/common/objs/obj_proto_pb.js

Publish DEC Service to OOD

DEC App Packaging and Publishing Principles

Packaging an app is a local process. The Service part, Web part, and various configuration files of the DEC App are copied to the dist folder and organized in a specific format. The location of the dist folder is specified by the dist field of the project configuration file.

You can execute the command cyfs pack separately to manually perform the packaging process, check the packaging process, and check whether there are errors in the packaged folder. The packaged dist folder is organized in the following way:

├─acl
│ └────acl.cfg
│
├─dependent
│ └───dependent.cfg
│
├─service
│ ├───x86_64-pc-windows-msvc.zip
│ └───x86_64-unknown-linux-gnu.zip
│
└─web

acl: Store the acl configuration file of the service. The packaging process will copy the file you specified in the service.app_acl_config.default field to this folder and rename it to acl.cfg dependent: It is designed to store the CYFS protocol stack dependency configuration of the service. Currently this function is invalid service: The binary file that stores the service. According to the configuration of service.dist_targets, package the {target}.zip file for each platform respectively. When developing service with ts, the content of the zip file is the folder specified in service.pack, plus the app_config file of the corresponding platform web: Store the web content of the app. During the packaging process, the content in the web.folder folder will be copied to this If service.pack is empty, the acl, dependent, service folders will not be generated; if web.folder is empty, the web folder will not be generated

If an app's service.pack and web.folder are both empty, the deploy command has no effect. Will not publish an empty app

Upload DEC App

Currently, the cyfs-client tool in the CYFS browser is used here to upload the dist folder to the owner's OOD. Due to some historical and stability reasons, there is no standard upload method using the CYFS protocol stack. The upload here is different from the upload using the cyfs upload command.

Publish DEC App information

After the DEC App is uploaded, add the information of this version to the local DEC App object, and then upload the object to the chain.

Generate DEC App link

After the information is published successfully, the link is generated according to the following rules: cyfs://{owner_id}/{dec_id}. Since all version information is already contained in the DEC App object, you will notice that this link remains the same every time a DEC App is released

Publish service

We first open the CYFS browser, then, in the project root directory, open the terminal, and run the following command:

  • mac
npm run mac-deploy-pre
npm run deploy

If the following error occurs during the process:

[error],[2022-09-14 19:39:09.175],<>,owner mismatch, exit deploy., cyfs.js:389

This error indicates that the current owner does not match the owner of the application. We need to manually modify the owner of the application, open the terminal in the project root directory, and enter the following command:

cyfs modify -o ~/.cyfs_profile/people
yes

Execute the command and print out the words save app object success, which means the modification is successful.

Next, we open the cyfs.config.json file in the project root directory and find that the app_id has changed. Therefore, we need to change DEC_ID_BASE58 in src/common/constant.ts to the latest app_id.

After the modification, we can go through the process of compiling and packaging the project again.

  • windows
npm run deploy

Finally, the terminal will display the uploaded information. After the upload is complete, the terminal will display the following information:

Upload DEC App Finished.
CYFS App Install Link: cyfs://5r4MYfFbqqyqoA4RipKdGEKQ6ZSX3JzNRaEpMPKiKWAQ/9tGpLNnbNtojWgQ3GmU2Y7byFm7uHDr1AH2FJBoGt5YF

Congratulations, this represents our DEC Service has been successfully published to OOD.

Please copy the link corresponding to CYFS App Install Link, we will use this link to install DEC Service in the next section.

Install DEC Service

In the previous section Publish service to OOD, we have successfully published DEC Service to OOD. Now, let's install DEC Service.

AppManager installation principle

Before installing DEC Service, it is necessary for us to understand how AppManager installs DEC App.

When a user installs the specified version of the DEC App onto ood, ood does the following 4 things:

  1. According to the target of ood, find out whether there is a service/<target>.zip file in the corresponding dir. The zip is used here to reduce the size of publishing and downloading. If there is, extract the file to the {cyfs_root}/app/<app_id> folder;

  2. Find out if there is a web folder in the corresponding dir, if so, download the web folder to the {cyfs_root}/app/web/<app_id> folder, then add the folder to the cyfs stack to get a new web dir id;

  3. If the DEC App has service, the service install script will be executed (corresponding to the install configuration in the service_package.cfg file in the project root directory);

  4. If the DEC App has a service, the service start script will be executed (corresponding to the start configuration in the service_package.cfg file in the project root directory).

Install DEC Service via CYFS browser

  1. Copy the CYFS link behind the CYFS App Install Link, go to the CYFS browser to open the DEC App Store page (cyfs://static/DEC AppStore/app_store_list.html), click the Install via URL button , paste the installation link and click the Get App Information green button.
  2. In the Version List area on the page, you can see the historical versions of the DEC App. We select the latest version and click Install.
  3. Go back to the DEC App Store page (cyfs://static/DEC AppStore/app_store_list.html), click the Installed green button at the top of the page, you can see the installed DEC Service. If it says Installing , please wait patiently for a while.

Further explain the decentralized nature of AppManager based on the process (366-377)

By learning the installation principle and process of AppManager above, we can find that AppManager is not a centralized node, but a distributed system with many nodes. AppManager finds the target OOD node through target, and the OOD nodes can freely connect to each other. When we install the DEC App, any node may become a phased center, but it does not have a mandatory center control function. The influence between the OOD node and the OOD node will form a nonlinear causal relationship through the network. AppManager reflects an open, flat, and equal system structure. We call AppManager a decentralized AppManager.

Debug each function of DEC Service

At this point, our DEC Service is running on OOD. Now, let's debug the function of adding, deleting, modifying and checking messages.

How to write test programs

The main function of the test program is to start a Client to interact with the service in order to test the function of each interface. In principle, it is to start a runtime-stack and use the runtime-stack to initiate a request to the DEC Service.

Compile debug files

In the project root directory, open a terminal and execute the following command:

npx tsc

After execution, all test script files are in the deploy/src/service/test folder.

Debug release message function

  • See src/service/test/publish_message_test.ts for the complete source code

The main entry for debugging the post message module is the main function, which needs to complete the following three steps:

  1. Initialize runtime-stack
  2. Set the new message key value and content text content
  3. Make a request

Debug commands

In the project root directory, open a terminal and execute the following command:

node ./deploy/src/service/test/publish_message_test.js

If the interface is normal, the Client console will print publish message msgKey is ${msgKey}, result: ${r}, where msgKey is the key value of the newly created message object. Otherwise, print publish message failed.

It is recommended to copy the key value of the new message object printed on the Client console for debugging of the following query, modification and deletion functions.

Use cyfs shell to view new message objects on RootState

The cyfs shell is a very easy-to-use tool for quickly viewing and verifying the state of RootState data. Earlier, we posted a new message, which corresponds to a new message object under the messages_list path of the application. Let's use the cyfs shell to verify this. Open a terminal and enter the following command to start the cyfs shell:

cyfs shell

After execution, the cyfs shell command line terminal will appear. Let's follow the steps to view the newly created message object:

  1. Use the up and down keys of the keyboard to select the OOD to be viewed or the RootState of Device, because our Service is on OOD, so select OOD (the first) and press Enter;
  2. Enter ls and press Enter to view all the child nodes under the root path of RootState, you can see the dec id corresponding to message-board, copy the dec id to the terminal and press Enter;
  3. Enter cd <dec id> and press Enter to enter the application root path of the message board message-board;
  4. Enter ls and press Enter to view all the child nodes under the RootState root path of the message board, you will see .cyfs and messages_list, this messages_list is where all message objects are stored;
  5. Enter cd messages_list and press Enter to enter the messages_list path;
  6. Enter ls and press Enter, you can see all the message objects under the messages_list path, the left is the id of the message object, and the right is the key value of the message object.

After the previous cyfs shell practice, I believe you now have a more concrete understanding of RootState. Using the cyfs shell, make the data on the RootState visible! This is really cool!

View the logs on the Service side

Now, we have successfully debugged the function of posting a message with the Service on OOD. If there is an error inside the Service on OOD, how can we locate the problem? In fact, on OOD, we can easily view the logs on the service side. The following is the storage path of service logs in different operating system environments of OOD.

  • mac: ~/Library/cyfs/log/app/<dec_id>
  • windows: C:\cyfs\log\app<dec_id>
  • linux: /cyfs/log/app/<dec_id>

Debug query message function

  • See src/service/test/retrieve_message_test.ts for the complete source code

The main entry of the debugging query message module is the main function, which needs to complete the following three steps:

  1. Initialize runtime-stack
  2. Set the key value of the message object to be queried
  3. Make a request

Debug commands

Open the deploy/src/service/test/retrieve_message_test.js file and assign the msgKey string printed in the Client console in the previous step to the msgKey constant of the main function.

After the modification, in the project root directory, open the terminal and execute the following command:

node ./deploy/src/service/test/retrieve_message_test.js

If the interface is normal, the Client console will print retrieve message result: current Message key is ${msgRawObj.key}, content is ${msgRawObj.content}, which contains the key value and content of the message object. Otherwise, print retrieve message failed.

Debugging and modifying the message function

  • See src/service/test/update_message_test.ts for the complete source code

The main entry for debugging and modifying the message module is the main function, which needs to complete the following three steps:

  1. Initialize runtime-stack
  2. Set the key value of the message object to be modified and the new content value
  3. Make a request

Debug commands

Open the deploy/src/service/test/update_message_test.js file, and assign the msgKey string printed by the Client console in the debug release message function to the msgKey constant of the main function. Also, you can manually set the content constant to any string you like, in order to change the original message content.

After the modification, in the project root directory, open the terminal and execute the following command:

node ./deploy/src/service/test/update_message_test.js

If the interface is normal, the Client console will print update message result: ${r}. Otherwise, print update message failed.

Debugging delete message function

  • For the complete source code, see src/service/test/deletee_message_test.ts

The main entry for debugging and deleting the message module is the main function, which needs to complete the following three steps:

  1. Initialize runtime-stack
  2. Set the key value of the message object to be deleted
  3. Make a request

Debug commands

Open the deploy/src/service/test/delete_message_test.js file, and assign the msgKey string printed by the Client console in the debug release message function to the msgKey constant of the main function.

After the modification, in the project root directory, open the terminal and execute the following command:

node ./deploy/src/service/test/delete_message_test.js

If the interface is normal, the Client console will print delete message result: ${r}. Otherwise, print delete message failed.

View root_state using command line tools

Using CYFS-SHELL can quickly perceive the data state change on root_state.

CYFS-SHELL use

1.Enter cyfs shell [ -e runtime/ood], enter the interactive command line, select device_id and dec_id and enter the corresponding Root-State root. 2. Use the following command

  • ls: List all child nodes in this directory

  • cd: Enter the child node, if the child node is not an ObjectMap, prompt an error and stay in the current directory

  • cat: display the object content of the child node in json format

  • dump: save the object content of the child node in binary format, the save path defaults to the current path, and the save file name is .obj

  • get: save the files of this node and subsequent nodes to the local, the save path defaults to the current path + node name

  • rm: delete the node, if the node is an object map and has child nodes, the deletion fails

  • target: re-select target, the path is reset to the root directory after selection

  • clear: clear the screen

  • help: help information

  • exit: exit the shell

View root_state with browser

This is a cool feature that we are working on, so stay tuned!

Summary

At this point, our message service has been debugged. The biggest feature of this service is that it can only serve itself, that is to say, you can only see the content of your own message board. If you want to see your friend's message board, you can't help it for the time being. Don't be discouraged, though, when you've completed this series of tutorials, you can easily do this!