Feature/reduce container creation delay #770

carpawell · 2021-08-25T13:00:33Z

Closes #519.

Implement notary requests subscription and handling mechanisms that can be also
reused for other node -> contract ---*notification*---> alphabet -> contract
flows in the future.

Fix errors/typos in commentaries. Delete spaces in logs' keys.

codecov · 2021-08-25T13:03:11Z

Codecov Report

Merging #770 (84ecf90) into master (3ff2e31) will decrease coverage by 0.24%.
The diff coverage is 15.52%.

❗ Current head 84ecf90 differs from pull request most recent head 2458ac5. Consider uploading reports for the commit 2458ac5 to get more accurate results

@@            Coverage Diff             @@
##           master     #770      +/-   ##
==========================================
- Coverage   40.28%   40.03%   -0.25%     
==========================================
  Files         223      224       +1     
  Lines       10900    11013     +113     
==========================================
+ Hits         4391     4409      +18     
- Misses       6158     6253      +95     
  Partials      351      351

Impacted Files	Coverage Δ
cmd/neofs-node/config/cast.go	`86.66% <0.00%> (-13.34%)`	⬇️
pkg/innerring/processors/audit/processor.go	`0.00% <0.00%> (ø)`
pkg/innerring/processors/governance/processor.go	`0.00% <0.00%> (ø)`
pkg/innerring/processors/netmap/processor.go	`0.00% <0.00%> (ø)`
pkg/morph/client/notary.go	`0.00% <0.00%> (ø)`
pkg/morph/client/static.go	`0.00% <0.00%> (ø)`
pkg/morph/event/container/delete.go	`60.86% <ø> (ø)`
pkg/morph/event/container/put.go	`50.00% <0.00%> (-3.85%)`	⬇️
pkg/morph/event/container/put_notary.go	`0.00% <0.00%> (ø)`
pkg/network/address.go	`87.23% <ø> (+3.56%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ff2e31...2458ac5. Read the comment docs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

`fallbackTime` is delta b/w `ValidUntilBlock` of the main transaction and block when `fallback` transaction is sent. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Refactor comments so they start with the names of methods. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Prepare all listening structures for notary events: rename(add prefix/suffix 'notification') all notification specific handlers/parsers. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

alexvanin · 2021-09-02T14:26:00Z

pkg/morph/event/notary.go

+}
+
+// NotaryTypeFromBytes converts bytes slice to NotaryType.
+func NotaryTypeFromBytes(data []byte) NotaryType {


Unused, do we need that?

did just the same as in notification.go(not used too). for future "potential" usage I guess

delete?

pkg/morph/event/listener.go

pkg/morph/client/notary.go

pkg/morph/client/constructor.go

pkg/innerring/bindings.go

cmd/neofs-node/config/morph/config.go

alexvanin · 2021-09-02T16:00:26Z

cmd/neofs-node/container.go


 func initContainerService(c *cfg) {
-	wrap, err := wrapper.NewFromMorph(c.cfgMorph.client, c.cfgContainer.scriptHash, 0)
+	wrap, err := wrapper.NewFromMorph(c.cfgMorph.client, c.cfgContainer.scriptHash, 0, wrapper.TryNotary())


Check if it breaks container.Delete and container.SetEACL

yes, it breaks

carpawell · 2021-09-05T21:44:22Z

@alexvanin, added resigning SN's notary request by IR instead of creating a new one

key changes:

notaryPreparator is struct now, not func. checking received requests is stricter now
added resigning notary request in morph/client.notary.go
added retrieving low-level morph client via Morph() method in wrappers
Put event contains a raw notary request pointer. Put handling logic in IR processor depends on that fact

Questions:

naming for IsAlphabet() option
time of contracts processors interface merging(in this PR or after all PR about notary flow notifications are merged)

carpawell · 2021-09-05T21:47:36Z

@AnnaShaleva, can you, please, take a look at ce63109 and give us feedback about SN's notary request checking by alphabet node?

AnnaShaleva · 2021-09-06T07:43:13Z

pkg/morph/subscriber/subscriber.go


 				s.blockChan <- b
+			case response.NotaryRequestEventID:
+				notaryRequest, ok := notification.Value.(*response.NotaryRequestEvent)


You're probably not interested in removed notary requests, only in those that have been added to the pool. So you may filter out removed ones right here:

if notaryRequest.Type == mempoolevent.TransactionRemoved { continue }

I see that removed requests are being filtered later, but anyway, do we need them at all?

To be fair that's the question I'm asking myself in general (subscription API-wise), who needs them?

just wanted to make base functionality in node listener: it can listen to, parse and handle any notary requests

but yes, there are no parsers and handlers for such(removes notary requests) events in the alphabet node now

/cc @alexvanin

To be fair that's the question I'm asking myself in general (subscription API-wise), who needs them?

I think it still can be a nice indicator of the case when neither main nor fallback transaction was completed. Other cases are covered by existing subscriptions API, e.g. subscriptions for transactions execution (main or fallback).

pkg/morph/event/listener.go

pkg/morph/event/notary_preparator.go

pkg/morph/client/notary.go

alexvanin

Let's rename IsAlphabet to AsAlphabet and call it a day for this PR.
In next PR I expect to see all container related notification with notary notification support, merged interfaces of NotaryContractProcessor and ContractProcessor and small processor event handling refactor.

AnnaShaleva · 2021-09-07T07:50:48Z

pkg/morph/event/notary_preparator.go

+}
+
+func (p Preparator) validateExpiration(fbTX *transaction.Transaction) error {
+	if len(fbTX.Attributes) != 3 || fbTX.Attributes[1].Type != transaction.NotValidBeforeT {


The attributes order is undefined in general, so here you check the way neo-go RPC client constructs them. I suggest to keep attributes length check and use fbTX.GetAttributes(transation.NotValidBefore) instead of fetching attribute by index.

thanks, added more general attribute check

Add preparator for notary requests. Is parses raw notary requests, checks if it should be handled by Alphabet node. If handling is required, returns `NotaryEvent` that contains information about contract scripthash, method name and arguments of the call. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add handlers and parsers functionality for listener. Separate notification and notary events by files. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add access to low-level morph client in wrappers Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

`fallbackTime` is delta b/w `ValidUntilBlock` of the main transaction and block when `fallback` transaction is sent. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Refactor comments so they start with the names of methods. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Prepare all listening structures for notary events: rename(add prefix/suffix 'notification') all notification specific handlers/parsers. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add preparator for notary requests. Is parses raw notary requests, checks if it should be handled by Alphabet node. If handling is required, returns `NotaryEvent` that contains information about contract scripthash, method name and arguments of the call. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add handlers and parsers functionality for listener. Separate notification and notary events by files. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add access to low-level morph client in wrappers Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add `NotaryInvokeNotAlpha` to low-level client. It creates and sends notary request that must be signed by Alphabet nodes, but does not sign it by current node's private key. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Add `NotarySignAndInvokeTX` method to morph client. This function allows invoking notary request with passed main TX(not creating a new one). It signs passed main TX with client's key. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Implement `NotaryContractProcessor` by IR container processor. Add support for notary `put` container operation. Do not parse `put` non-notary notifications in notary enabled environment. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Storage Node needs to have notary deposit for successful notary request sending. Add notary deposit on startup(and wait for its acceptance). Add notary deposit timer, its config in `morph` section and env vars for its tuning. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added enhancement Improving existing functionality neofs-ir Inner Ring node application issues neofs-storage Storage node application issues labels Aug 25, 2021

carpawell self-assigned this Aug 25, 2021

carpawell marked this pull request as ready for review August 25, 2021 15:28

carpawell requested review from alexvanin and cthulhu-rider August 25, 2021 15:28

carpawell mentioned this pull request Aug 25, 2021

Storage Node notary deposit dynamic frequency #771

Closed

carpawell added 7 commits September 2, 2021 11:38

[#770] ir: Reuse read notary config

ddc5c85

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] morph/client: Fix comment for fallbackTime

ce04fc7

`fallbackTime` is delta b/w `ValidUntilBlock` of the main transaction and block when `fallback` transaction is sent. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph: Fix commentaries for Listener interface

4f9ebc6

Refactor comments so they start with the names of methods. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph/subscriber: Add subscription for notary

70fc4dd

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph/event: Fix typo in AddPeer parser

6d3236c

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph: Rename all parsers and handlers structs/interfaces

89d7340

Prepare all listening structures for notary events: rename(add prefix/suffix 'notification') all notification specific handlers/parsers. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/innerring: Fix commentary to bind functions

2c5b8e0

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

alexvanin reviewed Sep 2, 2021

View reviewed changes

carpawell requested a review from alexvanin September 5, 2021 21:39

carpawell requested a review from AnnaShaleva September 5, 2021 21:44

AnnaShaleva reviewed Sep 6, 2021

View reviewed changes

alexvanin approved these changes Sep 7, 2021

View reviewed changes

AnnaShaleva reviewed Sep 7, 2021

View reviewed changes

carpawell added 6 commits September 7, 2021 11:55

[#770] pkg/morph/event: Add notary notifications support

2d890b8

Add handlers and parsers functionality for listener. Separate notification and notary events by files. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] morph/event/container: Add notary Put parser

e1f924e

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/innerring: Add NotaryContractProcessor interface

66e0729

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph: Add Morph() method

b3496e7

Add access to low-level morph client in wrappers Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

[#770] pkg/morph/container: Add Morph() method

931e531

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell merged commit b4378f7 into nspcc-dev:master Sep 7, 2021

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] ir: Reuse read notary config

3400a07

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] morph/client: Fix comment for fallbackTime

c0e25d1

`fallbackTime` is delta b/w `ValidUntilBlock` of the main transaction and block when `fallback` transaction is sent. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph: Fix commentaries for Listener interface

e24489d

Refactor comments so they start with the names of methods. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph/subscriber: Add subscription for notary

f0cbf2e

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph/event: Fix typo in AddPeer parser

c87bc70

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/innerring: Fix commentary to bind functions

8d1f2e6

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph/event: Add notary notifications support

c042f6a

Add handlers and parsers functionality for listener. Separate notification and notary events by files. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] morph/event/container: Add notary Put parser

d40f898

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/innerring: Add NotaryContractProcessor interface

4e5618a

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph: Add Morph() method

46df288

Add access to low-level morph client in wrappers Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/morph/container: Add Morph() method

b46adf1

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] morph: Support non-alpha notary request by wrappers

79b350b

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] pkg/innerring: Make all morph wrappers Alphabet

f4a521c

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell deleted the feature/reduce-container-creation-delay branch September 7, 2021 09:55

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] node/config: Add proxy contract to config

d77b2d1

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] node/config: Add uint32 casting

2ef5e86

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] node/config: Add notary deposit config

e29bcd9

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] morph/event/container: Sync comment and method name

1fedd8f

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] node/cmd: Add TryNotary to SN container wrapper

2f343a1

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell added a commit that referenced this pull request Sep 7, 2021

[#770] logger: Delete spaces in log message keys

4a81781

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

carpawell mentioned this pull request Sep 7, 2021

Smart contract script hash auto negotiation #786

Merged

This was referenced Sep 8, 2021

Use notary notifications in netmap contract invocations #805

Closed

Use notary notifications in reputation contract invocations #806

Closed

Feature/reduce container creation delay #770

Feature/reduce container creation delay #770

Uh oh!

Conversation

carpawell commented Aug 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carpawell commented Sep 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carpawell commented Sep 5, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carpawell Sep 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AnnaShaleva Sep 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexvanin left a comment

Choose a reason for hiding this comment

Uh oh!

AnnaShaleva Sep 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

carpawell commented Aug 25, 2021 •

edited

Loading

codecov bot commented Aug 25, 2021 •

edited

Loading

carpawell commented Sep 5, 2021 •

edited

Loading

carpawell Sep 6, 2021 •

edited

Loading

AnnaShaleva Sep 6, 2021 •

edited

Loading

AnnaShaleva Sep 7, 2021 •

edited

Loading