SmesherService.SmeshingStatus API Endpoint Proposal

[abergasov]

Discussed in #3266

create new endpoint SmesherService.SmeshingStatus, which will serve info about epochs

by default, endpoint should serve return a list of SmeshingStatusResponse, which contain info about current epoch, next and next after next. Other epoch can be requested from query.

type SmeshingStatusResponse struct {
  status: 'not_eligible',
  epoch: uint64;
  estimatedTime: number;
  reason: string;
  processedBytes: string // like "Processed 176 GB of 256 GB"
  processedPercentage: string // like "86% completed"
}

^{Originally posted by brusherru June 19, 2022}

Rationale

We want to make clear to Smeshers what is going on at every moment in the time:

• does his Node smeshes properly
• does he eligible for rewards in the current/next epoch
• if not — then why

Let's refresh our memory on how it works, assign some statuses to each "step" and clarify which additional data we may require.

1. Not smeshing yet
   Status: Not smeshing
2. Smesher set up smeshing: define PoS directory, data units amount, coinbase. The node starts creating PoS data.
   Status: Generating PoS data
3. When it is done, the Node waits for PoET
   Status: Waiting for PoET challenge
   Extra: estimated time / layer
4. Node generates proof-of-space
   Status: Generating Proof-of-Space
   Extra: time estimation
5. Node publishes ATX and starts waiting for the next epoch
   Status: Waiting for the next epoch
   Extra: time estimation
6. If there were no errors and timing issues — Smesher is eligible for rewards in the started epoch
   Status: Smeshing on epoch 23 (current)
7. If Smesher is eligible for rewards, then he will start receiving them within this epoch
   Status: Smeshing on epoch 23 (current)
8. Meanwhile, Node starts waiting for PoET challenge for the next epoch (goto 3)

Here is a good illustration by @noamnelke:

We already had some related discussion, so you can check it out: spacemeshos/smapp#721

Problem

Some of the steps, mentioned above, take a long time.
Some of them may finish with failure, which means that time from the 2nd step to the 6th (getting rewards) may take much more time. So we need to notify Smesher about the current status.

We already have some API endpoints, but they have a narrow scope and do not cover all the steps from the list above:

• SmesherService.IsSmeshing -> bool — true if PoS creation complete and "ready to send ATXs". But actually, if the node does not sync, it is still true. So from my point of view, it seems almost useless (the same as PostSetupStatus === complete). Fix me if I miss something.
• SmesherService.PostSetupStatus — enum (not_started | in_progress | complete | error). Related only to the PoS creating process.

So we need some API endpoints that will provide other statuses. Or, I think it will be much better, one API endpoint that will answer all these questions.

Proposal

Let's make a brand-new SmesherService.SmeshingStatus endpoint, that will return any of the statuses mentioned above with required extra data.
Such a solution will encapsulate some logic on the Node side, like requesting/listening for ATXs, rewards, PoET, etc.
Actually, I've already created one issue for it some time ago: #3222

So I think it might be a nullary endpoint, which returns something like:

SmeshingStatusResponse =
  | { status: 'not_started' }
  | {
       status: 'generating_pos_data' | 'generating_pos_proof' | ...
       estimatedTime: number;
    }
  | ...
  | {
       status: 'waiting_for_poet',
       epoch: number;
       estimatedTime/Layer: number;
    }
  | {
       status: 'smeshing', 
       epoch: number;
    }
  | {
       status: 'not_eligible',
       epoch: uint64;
       reason: string;
    };

Multiple statuses at once

Actually, Smesher can have more than one status at a time, regarding different epochs.

For example:

• Epoch 23: smeshing
• Epoch 24: waiting for PoET

Or even:

• Epoch 23: smeshing finished
• Epoch 24: Not eligible, missed timeframe
• Epoch 25: Waiting for PoET

So probably this endpoint should return a list of SmeshingStatusResponse.
But in this case, we have an open question, which epochs should be on the list. Putting all previous epochs is a bad idea.

Can we implement only streaming API endpoint, but with one requirement: can it send "current status" to the new client on connection?

What's next

Please, share your thoughts about the problem and my proposal.
If you found that such implementation will be complicated / not optimal / etc — tell us about it, please.
If I miss something — feel free to fix me. If you have other ideas — share them.

---

P.S.
We also had a discussion about API for retrieving ATXs. But from my point of view, it is not required for Smapp/Smrepl, in case all this "smeshing status logic" will be encapsulated on the Node side, as proposed above.
However, most likely we will require such an API for Explorer. And I think we should not make Smapp/Smrepl more complicated, and just put a link to explorer where it is needed (aka on Smesher screen).

So I propose to leave API for retrieving ATXs is out of scope, and open another discussion/issue in the context of Explorer.

---

cc: @oriya @lrettig @noamnelke @dshulyak @countvonzero