docs/design: Storage Node Graceful Exit #2734

ethanadams · 2019-08-07T20:51:58Z

What: When a Storage Node wants to leave the network but does not want to lose their escrow we need to have a mechanism for them to exit the network “gracefully”.

Why: Give Storage Nodes a mechanism to leave the network while receiving their escrows, and reduce repair caused by node churn on satellites.

Please describe the tests:

Test 1:
Test 2:

Please describe the performance impact:

Code Review Checklist (to be filled out by reviewer)

Does the PR describe what changes are being made?
Does the PR describe why the changes are being made?
Does the code follow our style guide?
Does the code follow our testing guide?
Is the PR appropriately sized? (If it could be broken into smaller PRs it should be)
Does the new code have enough tests? (every PR should have tests or justification otherwise. Bug-fix PRs especially)
Does the new code have enough documentation that answers "how do I use it?" and "what does it do?"? (both source documentation and higher level, diagrams?)
Does any documentation need updating?
Do the database access patterns make sense?

aligeti · 2019-08-08T18:15:26Z

docs/design/storagenode-graceful-exit.md

+
+		field node_id           blob
+		field path              blob
+		field peice_info        blob


nit pick .. piece.. also instead of dt mention 'date'

changed _dt to _at to be consistent

aligeti · 2019-08-08T18:23:38Z

docs/design/storagenode-graceful-exit.md

+		rpc Initiate(stream InitiateRequest) returns (stream InitiateResponse) {}
+	}
+
+	message InitiateRequest {


This is not a comment, but a thought... when I first read 'initiaterequest' didn't think it was referring to graceful exit , so what do you think, of naming it 'GracefulExitInitiateRequest'

Updated to GracefulExit rpc InitiateExit(InitiateExitRequest) returns (InitiateExitResponse)

aligeti · 2019-08-08T18:26:58Z

docs/design/storagenode-graceful-exit.md

+		field completed_dt      timestamp ( updateable )
+	)
+   ```
+- Add `PieceAction` field to, `cache.FindStorageNodesRequest`. Update `cache.FindStorageNodesWithPreferences` to ignore exiting nodes for uploads and repairs.


instead of piece action, can u query to find nodes that have only 'exit_initiated_dt' null?, that way no need to add a new piece action.

The new PieceAction field indeed seems unnecessary. The caller can take advantage of the existing ExcludedNodes field to list the current nodes from the pointer.

Agreed. Not sure where I was going with this one. Document updated.

aligeti · 2019-08-08T18:48:08Z

docs/design/storagenode-graceful-exit.md

+    - Pushes the pieces to the storage node identified in the order using `ecclient`
+    - Sends the signed new node response to the satellite via a new `CommitPiece` method (uses `metainfo.UpdatePieces`).
+    - Updates `bytes_deleted` with the number of bytes deleted and sets `completed_dt` to current time
+  - Execution intervals and batch sizes should be configurable

 ## Open Questions


"What happens if a piece is deleted when the piece is in the process of being transferred from the exiting node to other node in the network?"

I believe if I understood this correctly, the exiting storage node shall only honor download request and not delete requests or new upload request, Infact exiting SN should internal be made to accept only the download and reject accepting new uploads and/or delete requests

mniewrzal · 2019-08-09T09:31:14Z

docs/design/storagenode-graceful-exit.md

- When a Storage Node is exiting the network gracefully I want the satellite to have the ability to track how much egress they used for exiting so that we do not pay them for that bandwidth.
+- When a Storage Node is in the process of gracefully exiting the network, it shall continue to participate in the requested audits and uptimes checks.
+- When a Storage Node is in the process of gracefully exiting the network, it shall continue to honor to download requests.
+	- The Storage Node keep a separate and detailed metric of network bandwidth used to serve the data to clients vs bandwidth used for graceful exit. The Storage Node shall NOT get paid for bandwidth used for graceful exit but shall get paid for the bandwidth used to serve data to clients(downloads, audits, uptime checks etc...)


I don't think we are paying for audits, uptime checks etc...

For sure, we are not paying for uptime checks. We should double-check if we are paying for audits.

updated to downloads and audit egress

For sure, we are not paying for uptime checks. We should double-check if we are paying for audits.

We are paying for audits traffic.

mniewrzal · 2019-08-09T09:43:48Z

docs/design/storagenode-graceful-exit.md

+  - Initiates the exit by setting `nodes.exit_initiated` to current time
+  - ``` 
+	service GracefulExit {
+		rpc Initiate(stream InitiateRequest) returns (stream InitiateResponse) {}


Why stream? Do we plan to keep it open during the whole exit process?

Oversight. Fixed

egonelbre

There are still open questions, TBD notes and the document is not simple enough.

egonelbre · 2019-08-20T11:16:47Z

docs/design/storagenode-graceful-exit.md

+        - Updates `bytes_deleted` with the number of bytes deleted
+        - Deletes the pieces that were successfully moved
+      - On failure (ex. satellite is unavailable), successful orders limits stored in `exit_orders` should be reprocessed on the next iteration
+  - Execution intervals and batch sizes should be configurable

 ## Open Questions

 - What happens if we are doing a repair and a graceful exit on the same segment?


There are still a lot of critical open questions.

egonelbre · 2019-08-20T11:18:56Z

docs/design/storagenode-graceful-exit.md

+		field completed_exit_signature  blob
+	)
+
+	model exit_orders {


Why do we need this table?

…r validaation sequence diagram

ethanadams · 2019-08-23T17:04:54Z

Reworked the document into multiple documents. Closing in favor of a new pull request

ethanadams and others added 10 commits August 2, 2019 12:57

wip

696f518

wip

66b7327

wip

6500dfd

updates based on review with dennis. still wip

bcba62e

Update storagenode-graceful-exit.md

1a99c38

adding some changes back

9a1b221

added more protobuf detail, and updated field names

db5726e

fixed proto message names and formatting

626144e

more around the satellite and SN services

d4cd655

Merge branch 'master' into ge-doc

3b276f8

ethanadams added Request Code Review Code review requested Reviewer Can Merge If all checks have passed, non-owner can merge PR labels Aug 7, 2019

ethanadams requested a review from a team August 7, 2019 20:51

cla-bot bot added the cla-signed label Aug 7, 2019

ghost requested review from aleitner and aligeti and removed request for a team August 7, 2019 20:52

updated proto rpc calls

9f1afea

ethanadams requested a review from a team August 8, 2019 13:39

ghost requested review from kaloyan-raev and littleskunk and removed request for a team August 8, 2019 13:39

egonelbre changed the title ~~Storage Node Graceful Exit Design Document~~ design/docs: Storage Node Graceful Exit Aug 8, 2019

egonelbre added the Design Doc Adding or updating a design document. label Aug 8, 2019

egonelbre changed the title ~~design/docs: Storage Node Graceful Exit~~ docs/design: Storage Node Graceful Exit Aug 8, 2019

aligeti reviewed Aug 8, 2019

View reviewed changes

mniewrzal reviewed Aug 9, 2019

View reviewed changes

ethanadams added 2 commits August 9, 2019 08:09

removed stream from proto def

828907f

renamed exit proto rpc/messages

bcce219

egonelbre requested changes Aug 20, 2019

View reviewed changes

egonelbre added 2 commits August 20, 2019 15:48

add overview document

493b2dd

add gathering pieces

93b406f

egonelbre added WIP Work In Progress and removed Request Code Review Code review requested Reviewer Can Merge If all checks have passed, non-owner can merge PR labels Aug 20, 2019

ethanadams and others added 21 commits August 20, 2019 16:30

updates to pieces, reports, and ui

6d09ad5

Adding discussion points for next meeting

d109c7d

move reporting to the common location

8d81eda

add more information about ui

57ff93a

add overview of the process

6903a45

add satellites table discussion

9c3a828

fix links

efc77a1

nicer sentence

0f7f0ec

write background in more detail

8c6e632

add sketch and rough description

6cac604

updates to protocol, and changed completed_at to finished_at

5ab8c55

add transferring piece logic

d709fc3

add sections

5a593ca

add todo about diagram

4fbbc4e

cleanup, consistency, and some rewording.

8ded023

add satellites table to overview

c91346b

add table notes

ba42118

add note

9a2c8e2

moved db definitions to overview, changed some wording, added transfe…

8da627b

…r validaation sequence diagram

updated to use original svg

b719b14

Merge branch 'master' into ge-doc

b6d4a77

ethanadams closed this Aug 23, 2019

ethanadams mentioned this pull request Aug 23, 2019

docs/design: Graceful Exit #2866

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs/design: Storage Node Graceful Exit #2734

docs/design: Storage Node Graceful Exit #2734

ethanadams commented Aug 7, 2019

aligeti Aug 8, 2019

ethanadams Aug 12, 2019

aligeti Aug 8, 2019

ethanadams Aug 9, 2019

aligeti Aug 8, 2019

kaloyan-raev Aug 13, 2019

ethanadams Aug 13, 2019

aligeti Aug 8, 2019

mniewrzal Aug 9, 2019

kaloyan-raev Aug 12, 2019

ethanadams Aug 12, 2019

AlexeyALeonov Aug 16, 2019

mniewrzal Aug 9, 2019

ethanadams Aug 9, 2019

egonelbre left a comment

egonelbre Aug 20, 2019

egonelbre Aug 20, 2019

ethanadams commented Aug 23, 2019

docs/design: Storage Node Graceful Exit #2734

docs/design: Storage Node Graceful Exit #2734

Conversation

ethanadams commented Aug 7, 2019

Code Review Checklist (to be filled out by reviewer)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

egonelbre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ethanadams commented Aug 23, 2019