diff

README.md

@@ -1,48 +1,101 @@
-<p align="center">
-  <a href="https://docs.filecoin.io/" title="Filecoin Docs">
-    <img src="documentation/images/lotus_logo_h.png" alt="Project Lotus Logo" width="244" />
-  </a>
-</p>
 
-<h1 align="center">Project Lotus - 莲</h1>
+Filecoin 挖矿软件分布式Miner版本，支持PoSt-miner、Deal-miner、Seal-miner功能分离，实现多Miner的分布式部署。
 
-<p align="center">
-  <a href="https://circleci.com/gh/filecoin-project/lotus"><img src="https://circleci.com/gh/filecoin-project/lotus.svg?style=svg"></a>
-  <a href="https://codecov.io/gh/filecoin-project/lotus"><img src="https://codecov.io/gh/filecoin-project/lotus/branch/master/graph/badge.svg"></a>
-  <a href="https://goreportcard.com/report/github.com/filecoin-project/lotus"><img src="https://goreportcard.com/badge/github.com/filecoin-project/lotus" /></a>  
-  <a href=""><img src="https://img.shields.io/badge/golang-%3E%3D1.15.5-blue.svg" /></a>
-  <br>
-</p>
+## 解决了哪些问题？
+- 单Miner节点负载过高，机器不稳定容易导致时空证明失败掉算力的问题；
+- 解决Miner节点负载过高，出块时由于计算不够快，导致的出块失败问题；
+- 解决Window-PoSt和Winning-PoSt同时进行时，显卡锁定冲突的问题；
+- 解决接单的时候，由于内存占用过高导致Miner异常崩溃，掉算力、出块失败等问题；
 
-Lotus is an implementation of the Filecoin Distributed Storage Network. For more details about Filecoin, check out the [Filecoin Spec](https://spec.filecoin.io).
+## 功能说明
+- 支持同一个MinerId在多台机器上分布式部署；
+- 支持Window-post-miner、Winning-post-miner、Deal-miner、Seal-miner功能分离：
+  - PoSt-miner：负责Window-PoSt和Winning-PoSt，可分开为两台机器，也可由一台机器完成；
+    - Window-post-miner：负责扇区窗口证明时的扇区抽查及提交；
+    - Winning-post-miner：负责出块时扇区的随机抽查；
+  - Deal-miner：负责接单和检索，和订单扇区密封的任务分配；
+  - Seal-miner：负责普通扇区密封的任务分配。
+- 调度管理基于自定义的任务数量，可以精确控制机器的工作状态
 
-## Building & Documentation
+![Image text](https://github.com/moran666666/lotus-1.4.0/blob/master/documentation/images/WX20210124-204732%402x.png)
 
-For instructions on how to build, install and setup lotus, please visit [https://docs.filecoin.io/get-started/lotus](https://docs.filecoin.io/get-started/lotus/).
+## 实现思路
+**1. 解决SectorID不重复**
 
-## Reporting a Vulnerability
+利用一台Miner集中分配SectorID，由任意一台Miner的`--sctype=alloce`参数实现。
+并由启用`--sclisten=IP:PORT`参数的Miner作为分配SectorID的Server，其他Miner设置`--sctype=get`作为Client，从Server Miner上申请SectorID。
 
-Please send an email to security@filecoin.org. See our [security policy](SECURITY.md) for more details.
+```sh
+lotus-miner run --sctype=alloce --sclisten=192.168.1.50:1357
+```
 
-## Development
+**2. 解决时空证明由单独的机器完成**
 
-The main branches under development at the moment are:
-* [`master`](https://github.com/filecoin-project/lotus): current testnet.
-* [`next`](https://github.com/filecoin-project/lotus/tree/next): working branch with chain-breaking changes.
-* [`ntwk-calibration`](https://github.com/filecoin-project/lotus/tree/ntwk-calibration): devnet running one of `next` commits.
+由任意一台Miner的`--wdpost=true`和`--wnpost=true`参数实现时空证明由一台/两台独立的机器完成，其他Miner要设置`--wdpost=false`和`--wnpost=false`。
 
-### Tracker
+```sh
+lotus-miner run --wdpost=true --wnpost=true
+```
 
-All work is tracked via issues. An attempt at keeping an up-to-date view on remaining work towards Mainnet launch can be seen at the [lotus github project board](https://github.com/orgs/filecoin-project/projects/8). The issues labeled with `incentives` are there to identify the issues needed for Space Race launch.
+**3. 解决接单及检索**
 
-### Packages
+由任意一台Miner的`--p2p=true`参数实现，接单的Miner需要配置多地址。
 
-The lotus Filecoin implementation unfolds into the following packages:
+```sh
+lotus-miner run --p2p=true
+```
 
-- [This repo](https://github.com/filecoin-project/lotus)
-- [go-fil-markets](https://github.com/filecoin-project/go-fil-markets) which has its own [kanban work tracker available here](https://app.zenhub.com/workspaces/markets-shared-components-5daa144a7046a60001c6e253/board)
-- [spec-actors](https://github.com/filecoin-project/specs-actors) which has its own [kanban work tracker available here](https://app.zenhub.com/workspaces/actors-5ee6f3aa87591f0016c05685/board)
+## 参数说明
+- `--wdpost` [boolean] 是否允许window-PoSt证明，默认`true`；
+- `--wnpost` [boolean] 是否允许winning-PoSt证明，即出块，默认`true`；
+- `--p2p` [boolean] 是否允许接单，默认`true`；
+- `--sctype` [alloce | get] 扇区分配服务类型，`alloce`表示分配SectorID，`get`表示获取SectorID;
+- `--sclisten` [string] 扇区分配服务端监听的地址和端口;
 
-## License
+## 配置举例
+现有的单Miner集群要切换到分布式Miner，需要先停掉原有Miner和所有Worker，保证没有进行中的任务。然后将原有单Miner的`$LOTUS_STORAGE_PATH`目录，复制到其他Miner上，并将`$LOTUS_STORAGE_PATH/config.toml`文件中[API]部分的`ListenAddress`和`RemoteListenAddress`IP改为当前所在Miner的内网IP。
 
-Dual-licensed under [MIT](https://github.com/filecoin-project/lotus/blob/master/LICENSE-MIT) + [Apache 2.0](https://github.com/filecoin-project/lotus/blob/master/LICENSE-APACHE)
+比如有3个Miner，由PoSt-miner负责SectorID分配(也可由其他Miner分配)。
+
+**1. PoSt-miner**
+```
+lotus-miner run --wdpost=true --wnpost=true --p2p=false --sctype=alloce --sclisten=192.168.1.50:1357
+```
+其中`192.168.1.50`为PoSt-miner的内网IP，`1357`为扇区分配服务的监听端口。
+
+如果PoSt-miner为两台机器，则需要分别配置`--wdpost`和`--wnpost`参数：
+
+**Window-post-miner**（负责窗口抽查证明）配置为：
+```sh
+lotus-miner run --wdpost=true --wnpost=false --p2p=false
+```
+
+**Winning-post-miner**（负责出块的证明）配置为：
+```sh
+lotus-miner run --wdpost=false --wnpost=true --p2p=false --sctype=alloce --sclisten=192.168.1.50:1357
+```
+
+**注意：负责扇区分配的Miner，首次部署的时候，需要修改`$LOTUS_STORAGE_PATH/sectorid`文件(首次部署需要创建该文件)，将其中的数字改为大于当前所有扇区ID的一个数字。**
+
+**2. Deal-miner**
+```
+lotus-miner run --wdpost=false --wnpost=false --p2p=true --sctype=get --sclisten=192.16810.50:1357
+```
+其中`--sclisten`监听的地址和端口为做扇区分配Server Miner的IP和端口。  
+Deal-miner需要配置外网端口转发和`lotus-miner actor set-addrs`配置多地址。
+
+**注意：Deal-miner也只能配置一台，不然订单检索的时候会路由失败。**
+
+**3. Seal-miner**
+```
+lotus-miner run --wdpost=false --wnpost=false --p2p=false --sctype=get --sclisten=192.16810.50:1357
+```
+
+**需要注意的几个问题：**
+- Seal-miner和Deal-miner的配置参数相同，原理上都是密封扇区的Miner；  
+- Seal-miner可根据自己的集群规模，配置多台；
+
+## Worker配置
+使用分布式Miner以后，需要根据配置的Seal-miner和Deal-miner数量，将Seal-worker分为多个组，和Miner一一对应。
+
+假设现在有1台Deal-miner，2台Seal-miner，5台Seal-worker，因为Deal-miner的接单速度比较慢，可以给Deal-miner分配一台Seal-worker，其他两台Seal-miner各分配2台Seal-worker。

cmd/lotus-seal-worker/main.go

@@ -103,6 +103,26 @@ var runCmd = &cli.Command{
 	Name:  "run",
 	Usage: "Start lotus worker",
 	Flags: []cli.Flag{
+		&cli.StringFlag{
+			Name:  "c2address",
+			Usage: "extern c2 ip and port",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "rpc",
+			Usage: "worker rpc ip and port",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "workername",
+			Usage: "worker name will display on jobs list",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "ability",
+			Usage: "worker sealing ability",
+			Value: "AP:1,PC1:1,PC2:1,C1:1,C2:1,FIN:1,GET:1,UNS:1,RD:1",
+		},
 		&cli.StringFlag{
 			Name:  "listen",
 			Usage: "host address and port the worker api will listen on",
@@ -170,6 +190,16 @@ var runCmd = &cli.Command{
 	Action: func(cctx *cli.Context) error {
 		log.Info("Starting lotus worker")
 
+		if cctx.String("c2address") != "" {
+			os.Setenv("C2_ADDRESS", cctx.String("c2address"))
+		}
+		if cctx.String("workername") != "" {
+			os.Setenv("WORKER_NAME", cctx.String("workername"))
+		}
+		if cctx.String("ability") != "" {
+			os.Setenv("ABILITY", cctx.String("ability"))
+		}
+
 		if !cctx.Bool("enable-gpu-proving") {
 			if err := os.Setenv("BELLMAN_NO_GPU", "true"); err != nil {
 				return xerrors.Errorf("could not set no-gpu env: %+v", err)
@@ -509,6 +539,9 @@ var runCmd = &cli.Command{
 
 					select {
 					case <-readyCh:
+						if cctx.String("rpc") != "" {
+							address = cctx.String("rpc")
+						}
 						if err := nodeApi.WorkerConnect(ctx, "http://"+address+"/rpc/v0"); err != nil {
 							log.Errorf("Registering worker failed: %+v", err)
 							cancel()

cmd/lotus-storage-miner/run.go

@@ -2,11 +2,13 @@ package main
 
 import (
 	"context"
+	scServer "github.com/moran666666/sector-counter/server"
 	"net"
 	"net/http"
 	_ "net/http/pprof"
 	"os"
 	"os/signal"
+	"path/filepath"
 	"syscall"
 
 	mux "github.com/gorilla/mux"
@@ -36,6 +38,41 @@ var runCmd = &cli.Command{
 	Name:  "run",
 	Usage: "Start a lotus miner process",
 	Flags: []cli.Flag{
+		&cli.BoolFlag{
+			Name:  "wdpost",
+			Usage: "enable windowPoSt",
+			Value: true,
+		},
+		&cli.BoolFlag{
+			Name:  "wnpost",
+			Usage: "enable winningPoSt",
+			Value: true,
+		},
+		&cli.BoolFlag{
+			Name:  "p2p",
+			Usage: "enable P2P",
+			Value: true,
+		},
+		&cli.StringFlag{
+			Name:  "sctype",
+			Usage: "sector counter type(alloce,get)",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "sclisten",
+			Usage: "host address and port the sector counter will listen on",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "workername",
+			Usage: "worker name will display on jobs list",
+			Value: "",
+		},
+		&cli.StringFlag{
+			Name:  "ability",
+			Usage: "worker sealing ability",
+			Value: "AP:1,PC1:1,PC2:1,C1:1,C2:1,FIN:1,GET:1,UNS:1,RD:1",
+		},
 		&cli.StringFlag{
 			Name:  "miner-api",
 			Usage: "2345",
@@ -56,6 +93,45 @@ var runCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
+		if cctx.Bool("wdpost") {
+			os.Setenv("LOTUS_WDPOST", "true")
+		} else {
+			os.Unsetenv("LOTUS_WDPOST")
+		}
+
+		if cctx.Bool("wnpost") {
+			os.Setenv("LOTUS_WNPOST", "true")
+		} else {
+			os.Unsetenv("LOTUS_WNPOST")
+		}
+
+		scType := cctx.String("sctype")
+		if scType == "alloce" || scType == "get" {
+			os.Setenv("SC_TYPE", scType)
+
+			scListen := cctx.String("sclisten")
+			if scListen == "" {
+				log.Errorf("sclisten must be set")
+				return nil
+			}
+			os.Setenv("SC_LISTEN", scListen)
+
+			if scType == "alloce" {
+				scFilePath := filepath.Join(cctx.String(FlagMinerRepo), "sectorid")
+				go scServer.Run(scFilePath)
+			}
+		} else {
+			os.Unsetenv("SC_TYPE")
+		}
+
+		if cctx.String("workername") != "" {
+			os.Setenv("WORKER_NAME", cctx.String("workername"))
+		}
+
+		if cctx.String("ability") != "" {
+			os.Setenv("ABILITY", cctx.String("ability"))
+		}
+
 		if !cctx.Bool("enable-gpu-proving") {
 			err := os.Setenv("BELLMAN_NO_GPU", "true")
 			if err != nil {
@@ -138,14 +214,18 @@ var runCmd = &cli.Command{
 			return xerrors.Errorf("getting API endpoint: %w", err)
 		}
 
-		// Bootstrap with full node
-		remoteAddrs, err := nodeApi.NetAddrsListen(ctx)
-		if err != nil {
-			return xerrors.Errorf("getting full node libp2p address: %w", err)
-		}
+		if cctx.Bool("p2p") {
+			// Bootstrap with full node
+			remoteAddrs, err := nodeApi.NetAddrsListen(ctx)
+			if err != nil {
+				return xerrors.Errorf("getting full node libp2p address: %w", err)
+			}
 
-		if err := minerapi.NetConnect(ctx, remoteAddrs); err != nil {
-			return xerrors.Errorf("connecting to full node (libp2p): %w", err)
+			if err := minerapi.NetConnect(ctx, remoteAddrs); err != nil {
+				return xerrors.Errorf("connecting to full node (libp2p): %w", err)
+			}
+		} else {
+			log.Warn("This miner will be disable p2p.")
 		}
 
 		log.Infof("Remote version %s", v)

cmd/lotus-storage-miner/sealing.go

@@ -85,24 +85,39 @@ var sealingWorkersCmd = &cli.Command{
 
 			var barCols = uint64(64)
 			cpuBars := int(stat.CpuUse * barCols / stat.Info.Resources.CPUs)
-			cpuBar := strings.Repeat("|", cpuBars) + strings.Repeat(" ", int(barCols)-cpuBars)
+			cpuBar := ""
+			if int(barCols)-cpuBars > 0 {
+				cpuBar = strings.Repeat("|", cpuBars) + strings.Repeat(" ", int(barCols)-cpuBars)
+			} else {
+				cpuBar = strings.Repeat("|", int(barCols))
+			}
 
 			fmt.Printf("\tCPU:  [%s] %d/%d core(s) in use\n",
 				color.GreenString(cpuBar), stat.CpuUse, stat.Info.Resources.CPUs)
 
 			ramBarsRes := int(stat.Info.Resources.MemReserved * barCols / stat.Info.Resources.MemPhysical)
 			ramBarsUsed := int(stat.MemUsedMin * barCols / stat.Info.Resources.MemPhysical)
-			ramBar := color.YellowString(strings.Repeat("|", ramBarsRes)) +
-				color.GreenString(strings.Repeat("|", ramBarsUsed)) +
-				strings.Repeat(" ", int(barCols)-ramBarsUsed-ramBarsRes)
+			ramBar := ""
+			if int(barCols)-ramBarsUsed-ramBarsRes > 0 {
+				ramBar = color.YellowString(strings.Repeat("|", ramBarsRes)) +
+					color.GreenString(strings.Repeat("|", ramBarsUsed)) +
+					strings.Repeat(" ", int(barCols)-ramBarsUsed-ramBarsRes)
+			} else {
+				ramBar = strings.Repeat("|", int(barCols))
+			}
 
 			vmem := stat.Info.Resources.MemPhysical + stat.Info.Resources.MemSwap
 
 			vmemBarsRes := int(stat.Info.Resources.MemReserved * barCols / vmem)
 			vmemBarsUsed := int(stat.MemUsedMax * barCols / vmem)
-			vmemBar := color.YellowString(strings.Repeat("|", vmemBarsRes)) +
-				color.GreenString(strings.Repeat("|", vmemBarsUsed)) +
-				strings.Repeat(" ", int(barCols)-vmemBarsUsed-vmemBarsRes)
+			vmemBar := ""
+			if int(barCols)-vmemBarsUsed-vmemBarsRes > 0 {
+				vmemBar = color.YellowString(strings.Repeat("|", vmemBarsRes)) +
+					color.GreenString(strings.Repeat("|", vmemBarsUsed)) +
+					strings.Repeat(" ", int(barCols)-vmemBarsUsed-vmemBarsRes)
+			} else {
+				vmemBar = strings.Repeat("|", int(barCols))
+			}
 
 			fmt.Printf("\tRAM:  [%s] %d%% %s/%s\n", ramBar,
 				(stat.Info.Resources.MemReserved+stat.MemUsedMin)*100/stat.Info.Resources.MemPhysical,