Stuck in FinalizeSector

[long568]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.17.0+mainnet+git.2830429ad+api1.5.0
Local: lotus version 1.17.0+mainnet+git.49a964e77.dirty
Daemon:  1.17.0+mainnet+git.49a964e77.dirty+api1.5.0
Local: lotus-miner version 1.17.0+mainnet+git.49a964e77.dirty

Describe the Bug

Stuck in FinalizeSector
I use BOOSTD for markets.
I ONLY modified the RUST code.
Config:
[Subsystems] EnableMining = true EnableSealing = true EnableSectorStorage = true EnableMarkets = false SealerApiInfo = "" SectorIndexApiInfo = "" [Storage] ParallelFetchLimit = 10 AllowAddPiece = false AllowPreCommit1 = false AllowPreCommit2 = false AllowCommit = false AllowUnseal = false AllowReplicaUpdate = false AllowProveReplicaUpdate2 = false AllowRegenSectorKey = false ResourceFiltering = "hardware" DisallowRemoteFinalize = true

Logging Information

lo@m9:~$ lotus-miner sealing sched-diag
{
  "CallToWork": {},
  "EarlyRet": null,
  "ReturnedWork": null,
  "SchedInfo": {
    "OpenWindows": [
      "c881ec6b-d90a-4f19-9411-037709404c1c",
      "c881ec6b-d90a-4f19-9411-037709404c1c",
      "4cce183b-870e-4e2a-b952-b16529ed596f",
      "4cce183b-870e-4e2a-b952-b16529ed596f",
      "66c5a3dc-faa5-4a03-9b23-cda0d9f8d371",
      "87505310-86b4-4811-a71c-40cf6487e5ff",
      "66c5a3dc-faa5-4a03-9b23-cda0d9f8d371",
      "87505310-86b4-4811-a71c-40cf6487e5ff"
    ],
    "Requests": [
      {
        "Priority": 0,
        "Sector": {
          "Miner": 1192325,
          "Number": 1161
        },
        "TaskType": "seal/v0/fetch"
      },
      {
        "Priority": 0,
        "Sector": {
          "Miner": 1192325,
          "Number": 1162
        },
        "TaskType": "seal/v0/fetch"
      },
      {
        "Priority": 1024,
        "Sector": {
          "Miner": 1192325,
          "Number": 1129
        },
        "TaskType": "seal/v0/addpiece"
      }
    ]
  },
  "Waiting": null
}
lo@m9:~$ lotus-miner storage find 1161
In e3522768-cd2c-4046-814c-34107f69a92f (Sealed, Cache)
	Sealing: true; Storage: false
	Remote
	URL: http://192.168.1.5:3456/remote/sealed/s-t01192325-1161
lo@m9:~$ lotus-miner storage find 1162
In e3522768-cd2c-4046-814c-34107f69a92f (Sealed, Cache)
	Sealing: true; Storage: false
	Remote
	URL: http://192.168.1.5:3456/remote/sealed/s-t01192325-1162

lo@w005:~$ lotus-worker info
Worker version:  1.6.0
CLI version: lotus-worker version 1.17.0+mainnet+git.2830429ad.dirty

Session: 87505310-86b4-4811-a71c-40cf6487e5ff
Enabled: true
Hostname: w005
CPUs: 32; GPUs: [GeForce RTX 3070]
RAM: 10.52 GiB/996 GiB; Swap: 0 B/0 B
Task types: FIN GET FRU UNS C1 PC2 PC1 PR1 RU 

e3522768-cd2c-4046-814c-34107f69a92f:
	Weight: 10; Use: Seal 
	Local: /lotus/worker/p
3cd04477-e40b-4560-9191-c41063e5e9c4:
	Weight: 10; Use: Store
	Local: /home/lo/s_tst/

Repo Steps

lotus-miner sectors list 1161 FinalizeSector YES NO 3604124 (in 1 year 24 weeks) CC RecoveryTimeout: 2177442 (in 5 weeks 6 days) 1162 FinalizeSector YES NO 3604491 (in 1 year 24 weeks) CC RecoveryTimeout: 2177442 (in 5 weeks 6 days)

[Reiers]

Hi @long568

DisallowRemoteFinalize = true
This config will force all Finalize tasks to run on the workers - do all the workers have access and permission to write to your long term storage?

 # DisallowRemoteFinalize when set to true will force all Finalize tasks to
  # run on workers with local access to both long-term storage and the sealing
  # path containing the sector.
  # --
  # WARNING: Only set this if all workers have access to long-term storage
  # paths. If this flag is enabled, and there are workers without long-term
  # storage access, sectors will not be moved from them, and Finalize tasks
  # will appear to be stuck.

[long568]

Hi @long568

DisallowRemoteFinalize = true This config will force all Finalize tasks to run on the workers - do all the workers have access and permission to write to your long term storage?

 # DisallowRemoteFinalize when set to true will force all Finalize tasks to
  # run on workers with local access to both long-term storage and the sealing
  # path containing the sector.
  # --
  # WARNING: Only set this if all workers have access to long-term storage
  # paths. If this flag is enabled, and there are workers without long-term
  # storage access, sectors will not be moved from them, and Finalize tasks
  # will appear to be stuck.

Yes
My test environment:
1 Miner + 1 Worker(AP) + 1 Worker(P1 +P2) + 1 Worker(C2)
Miner & All the workers have access and permission to write to the long term storage(NFS).

[magik6k]

#9648