validate commit MVP implementation #189

thedavidmeister · 2018-08-04T11:06:25Z

fixes #97
fixes #176
fixes #171

notes:

it is not possible to dispatch a ValidateCommit action from inside a reducer for a Commit action, this causes the main action loop to hang, instead we do validate then commit inside invoke_commit
this doesn't touch on "incoming data from the DHT" as that doesn't exist yet, so no validate put exists yet
this doesn't pull any data out of lifecycle functions through memory management, so we don't get messages for our fails in the WAT tests, only pass/fail/NI
this is NOT chasing the evolving spec app being developed by @lucksus

changes:

added action.rs to root
- contains ActionWrapper, Action, Signal
updated action/state docs in mdbook
updated zome api functions implementation docs in mdbook
updated lifecycle functions docs in mdbook
updated "lifecycle of an entry" docs in mdbook
started standardising reducer function signatures (more to do in followup)
added validate_commit to invoke_commit from zome
- invoke_commit will only dispatch the commit action if validate_commit returns Pass or NotImplemented
removed some checks to history length in tests that caused genesis edge cases to hang indefinitely
fix stack overflow in HolochainError trait implementation for to_string()
Action is now global rather than specific to a state slice, e.g. zome can send actions that are reduced by either zome or agent
ActionResponse is now reducer module specific, e.g. agent::state::ActionResponse
include unwrap_to! macro to help get at enum wrapped values more easily
abstractions for lifecycle vs. zome api functions
- zome api functions are implemented by HC and called by zomes
- implementing zome api functions is simpler, more centralised and less error prone
- lifecycle functions are implemented by zomes and called by HC
- genesis uses lifecycle function call
- added Defn trait that ensures some consistency across lifecycle vs. API function definitions
replaced adhoc index/mapping logic in zome API call dispatcher with a standardised interface into zome function definitions (to string, to index, to function, from string, etc.)
initialization result logic extracted from genesis logic
tweaks to stringification logic of various things
ZomeFunctionResult has a dedicated reducer now
added ErrorLifecycleResult as a zome WASM error code
moar tests

todo/clarification in this tx:

what is Signal::ValidateEntry and how does it relate to validate commit if at all? @ddd-mtl
reckon I could merge ActionWrapper and Action? @lucksus
- after pairing this morning, we decided we probably can as both have a unique id

followups:

flatten action structure can we take the 3-4 level wrapper/action/signal/function nested structure of actions and flatten it a bit? #192
trait for json round trip add a trait for anything that can round trip through JSON with to_json/from_json #193
standardise reduce function signatures standardise reduce function signatures #194
don't inspect history length directly abstraction for inspecting the state history for specific actions, don't inspect history state #195
standard ActionResponse trait standard ActionResponse trait #196
flaky genesis test flaky test test_genesis_err #197
standardise function call or action as key in nucleus state should reduce_zfr store state using function call or action as key? #198
unit test reducers better zome tests for reducers should call the reducer directly instead of using a live global instance reduce #199
lifecycle actions? should every lifecycle function have a dedicated action? #200
lifecycle function receive lifecycle function: receive #201
as_fn in Defn trait add as_fn to Defn trait #202
UUIDs in snowflakes for persister real uuids for snowflakes that hit the persister save/load round trip #203
document capabilities document capabilities #204
get failure reasons back out of lifecycle functions how to get fail strings back out of lifecycle functions? #205
zome integration tests for validate commit and commit together tests that failing validate_commit prevents commit from happening #206
support nested action calls can we do nested actions sensibly? #222
better display handling for errors better implementation for display of holochain error #223

# Conflicts: # core/src/instance.rs # core/src/nucleus/mod.rs # core/src/nucleus/ribosome/api/print.rs # core/src/nucleus/ribosome/debug.rs # core/src/nucleus/ribosome/mod.rs # core/src/nucleus/ribosome/print.rs # core/src/persister.rs # core_api/src/lib.rs

sphinxc0re · 2018-08-06T13:20:42Z

core/src/nucleus/mod.rs

+                            _ => return_initialization_result(None, &action_channel),
+                        }
+                    })
+                    .collect();


.for_each(drop) might be better. From the docs:

"if you really need to exhaust the iterator, consider `.for_each(drop)` instead"

ddd-mtl

It seems you have mistaken Lifecycle with callback.
Currently, the HC API docs calls functions in the zome that are called by the ribosome callbacks.
In the new API, Lifecycle is the name of the capability where callbacks related to an app's lifecycle resides, as described in the refactor doc: https://docs.google.com/document/d/1744ZnaILkFdiiIFNOo0VpS9OEmoAhWgAsu__VLeEfww/edit

This PR implies that there is a different validation function for an entry when it comes from a commit. Is this what we want?

Reviewing this PR is difficult because it mixes a new feature with refactoring where files are moved around. I would appreciate having seperate PRs for feature and refactor.

Also splitting ribosome between api and lifecycle seems premature. This is starting to become folder hell. Would like this to be undone and planned in the future when it would actually add value.
For now I would merge everything thats under lifeycle\ in a file named: callbacks.rs
On a side note, I fail to see the coherence about splitting stuff up as much as possible but not splitting up the testing code.

I do like and see the value in all the other refactors in this PR.

I believe all the confusion/misunderstandings comes from a lack of specs.

ddd-mtl · 2018-08-10T15:37:12Z

core/src/action.rs

+impl Eq for Action {}
+
+#[derive(Clone, PartialEq, Hash, Debug)]
+pub enum Signal {


I don't understand why Action has been renamed Signal, and this will be confusing with actual Signals as described in the event/signal ADR.
Also the new Action struct doesn't seem to serve any purpose.
I would like this to change the following way:

Delete the current Action struct

Rename Signal -> Action

@ddd-mtl it has not been renamed signal, there were three (four?) structural levels before this branch, one was just implicit so i made it explicit and gave it a name

there's a followup to do what you're suggesting #192

the purpose of the changes i made other than naming things is consolidation. i'm working towards:

a flatter structure overall (e.g. something like what you suggested) so less matching

a single snowflake ID per action dispatch

a single standard representation for key/values for action/result pairs across all state slices

a single enum (rather than struct) that defines which types go where

happy to rename Signal if its confusing, there aren't many good words in this space though

ddd-mtl · 2018-08-10T15:54:34Z

core/src/error.rs

 }

 impl fmt::Display for HolochainError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        write!(f, "{}", self)
+        write!(f, "{:?}", self)


This seems to contradicts fmt::Display's purpose:
Display is similar to Debug, but Display is for user-facing output

Why would HolochainError display work only in Debug?
Doesn't Rust strip this out in optimized code?

@ddd-mtl i don't know the subtleties

the high level is that what was there before blows a stack overflow on to_string() calls (which is also why i added that in a test below)... some implicit recursion somewhere

if you know the right fix we can drop it in, otherwise i'd be inclined to flag a followup for this

@sphinxc0re any ideas for a real fix here?

i've opened a followup #223

ddd-mtl · 2018-08-10T16:12:23Z

core/src/nucleus/ribosome/api/mod.rs

+/// Enumeration converts to str
+#[repr(usize)]
+#[derive(FromPrimitive)]
+pub enum ZomeAPIFunction {


Please don't use CAPS in names, even for acronyms.
Rename to ZomeApiFunction

ddd-mtl · 2018-08-10T16:14:58Z

core/src/nucleus/ribosome/api/mod.rs

+        }
+    }
+
+    fn str_index(s: &str) -> usize {


This function name is confusing to me.
How about get_index?

ddd-mtl · 2018-08-10T16:20:46Z

core/src/nucleus/ribosome/api/mod.rs

+
+/// Object holding data to pass around to invoked API functions
+#[derive(Clone)]
+pub struct FunctionRuntime {


Don't understand why you renamed this. What was wrong with Runtime?
I find FunctionRuntime misleading because it makes you believe that it has a lifetime of just one ZomeApi function call but that's not true, it has the lifetime of the Wasm call.
How about WasmCallRuntime ?

@ddd-mtl this was part of something i did that i ended up rolling back, i'll change it back

ddd-mtl · 2018-08-10T18:35:47Z

core/src/nucleus/ribosome/lifecycle/mod.rs

+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s {
+            "genesis" => Ok(LifecycleFunction::Genesis),
+            "validate_commit" => Ok(LifecycleFunction::Genesis),


copy-paste error?
LifecycleFunction::ValidateCommit

yar, nice catch

i dropped some tests in for this

ddd-mtl · 2018-08-10T18:42:59Z

core/src/nucleus/ribosome/lifecycle/mod.rs

+        }
+    }
+
+    fn capabilities(&self) -> ReservedCapabilityNames {


A callback function can only live in one capability, so capability would make more sense.

ddd-mtl · 2018-08-10T18:53:11Z

core/src/nucleus/ribosome/api/mod.rs

+        }
+    }
+
+    fn capabilities(&self) -> ReservedCapabilityNames {


rename to capability
Also shouldn't this not be implemented, or panic instead?

ddd-mtl · 2018-08-10T18:55:37Z

dna/src/zome/capabilities.rs

@@ -20,6 +28,7 @@ impl FromStr for ReservedCapabilityNames {
        match s {
            "hc_lifecycle" => Ok(ReservedCapabilityNames::LifeCycle),
            "hc_web_gateway" => Ok(ReservedCapabilityNames::Communication),
+            "hc_zome_api_function" => Ok(ReservedCapabilityNames::ZomeAPIFunction),


This implies a capability named hc_zome_api_function which doesn't make any sense in my understanding of the new rust version. Zome Api functions should be callable in any capability, no?

ddd-mtl · 2018-08-10T19:01:23Z

core/src/nucleus/ribosome/lifecycle/mod.rs

+// @see https://github.com/holochain/holochain-rust/issues/200
+
+#[derive(FromPrimitive)]
+pub enum LifecycleFunction {


Confusion between lifecycle and callback
Rename to callback or list only the functions that belongs the the lifecycle capability

@ddd-mtl as i'm sure you picked up, i don't know how capabilities work, i cargo culted that in... still waiting on a pair session with @zippy to go over that but maybe you can run me through it too

we've been going back and forward on names like "hook", "zome function", etc.

callback doesn't seem right to me either because they aren't being passed around at runtime, let's have a quick straw poll in our next meeting to see if we can pick something

I'm also not comfortable with callback and agree with what you said. I was just sticking with the names used in the Go version.

# Conflicts: # doc/holochain_101/src/SUMMARY.md

sphinxc0re · 2018-08-15T09:17:44Z

doc/holochain_101/src/zome/callback_functions.md

-Lifecycle functions are passed some parameters and expected to return one of
-three possible `LifecycleFunctionParams` variants:
+Callback functions are passed some parameters and expected to return one of
+three possible `CallbackParams` variants:

 - `Pass`: The lifecycle function has executed/validated successfully


lifecycle -> callback

you can just edit this in github UI... @sphinxc0re

Connoropolous · 2018-08-15T11:58:44Z

This was a heavy lift! Nice work David!

timotree3 · 2018-08-15T16:13:32Z

core/src/action.rs

+        let a1 = test_action();
+        let a2 = test_action();
+
+        // unlike actions and wrappers, signals are equal to themselves


did you mean to say 'unlike signals and wrappers, actions are equal to themselves'?

timotree3 · 2018-08-15T17:34:16Z

core/src/instance.rs

+            .history
+            .iter()
+            .find(|aw| match aw.action() {
+                Action::ReturnZomeFunctionResult(_) => true,


this should be Action::ReturnZomeInitializationResult

lucksus

I'm too late with reading this PR but I nevertheless responded to @Connoropolous' direct question.

Wow, a lot of great work! Thank you @thedavidmeister!

lucksus · 2018-08-18T22:33:18Z

doc/holochain_101/src/lifecycle_of_an_entry.md

+Every entry must pass a ValidateCommit lifecycle function check before it can be
+committed.
+
+If ValidateCommit is not implemented for the zome committing the entry then this


Yes, this is contradictory to what I said at some point.
A missing validation function should be treated as not valid.

What we saw (and you can confirm or invalidated, @Connoropolous) a lot with people starting to write Holochain apps with the Go prototype was: not really implementing the validation function and just returning true. But actually that is the most important part of every Holochain app. So: for the current state it is not a problem to have it this way, but we should change this before the next release.

lucksus · 2018-08-18T22:41:26Z

doc/holochain_101/src/zome/callback_functions.md

+Called internally by the `commit` Zome API function.
+
+- `Pass`: the entry will be committed
+- `NotImplemented`: the entry will be committed


no, should not be committed.

WIP on action module

dde5e61

thedavidmeister added the review label Aug 4, 2018

thedavidmeister added 15 commits August 5, 2018 00:05

wip action split

0b89e92

WIP action factor

dbb6a45

passing tests

dbcfc42

WIP on API vs lifecycle in zome definitions

1ecbce2

post-merge repair

310ce7e

Merge commit '16e254b72bda4d78433a05827d3b9e2bcd51b5a6' into 97

be55a6f

WIP on lifecycle functions

15176f3

fmt

1c2e5c1

lint tests

aeba7db

WIP on lifecycle function results

553adc4

fix stack overflow

12f3e6e

refactor initialization

c847a1b

naive (no params) validate_commit implementation

00c53ab

WIP on generic lifecycle fn implementation

113e64a

sphinxc0re suggested changes Aug 6, 2018

View reviewed changes

thedavidmeister added 12 commits August 7, 2018 17:28

generic lifecycle function dispatch

2858f94

WIP on calling validate_commit

aaeafa4

fix call_wrong_ribosome_function

ffacc59

temporary fix for tests

729c3c6

hook validation into commit inline

26a1bc7

passing tests with validate commit

28f9e03

fmt

11513b4

update state_actions.md

d2826dc

remove functioncall from the commit signal

aa0292e

action wrapper takes an action reference

1fe0397

tweak comments for action.rs

33caacf

agent handle_commit returns a result

3878be9

thedavidmeister requested a review from ddd-mtl August 9, 2018 23:06

ddd-mtl suggested changes Aug 10, 2018

View reviewed changes

thedavidmeister added 14 commits August 13, 2018 18:26

Merge commit '22ee9513aff43f50dbf90e6579c551439cde1cb7' into 97

d96424c

# Conflicts: # doc/holochain_101/src/SUMMARY.md

fix and tests for LifecycleFunction from_str

d87942d

from_str tests for use std::str::FromStr;

99129ee

rename capabilities to capability in Defn

fbe4b66

renaming things as per PR review

e074a58

remove action signal

e369e9b

lints

e2a36e3

Merge branch 'develop' into 97

e1b5cf0

extended action wrapper tests

af39805

Merge branch '97' of github.com:holochain/holochain-rust into 97

03606b6

test action wrapper hashing

d521cfa

push context through to all reducers

b5cd4b3

lint

72acb1e

lint

787fc7c

ddd-mtl approved these changes Aug 14, 2018

View reviewed changes

sphinxc0re approved these changes Aug 14, 2018

View reviewed changes

thedavidmeister added 3 commits August 14, 2018 23:47

fmt

0014ef9

change lifecycle to callback in zome docs

0c1caf9

fix docs to reference action wrappers

1601509

sphinxc0re suggested changes Aug 15, 2018

View reviewed changes

fix callback vs. lifecycle

2d66021

thedavidmeister merged commit 71c2841 into develop Aug 15, 2018

thedavidmeister removed the review label Aug 15, 2018

timotree3 reviewed Aug 15, 2018

View reviewed changes

lucksus reviewed Aug 18, 2018

View reviewed changes

thedavidmeister mentioned this pull request Aug 21, 2018

validate NotImplemented should fail entry commit #235

Open

sphinxc0re deleted the 97 branch September 4, 2018 12:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

validate commit MVP implementation #189

validate commit MVP implementation #189

thedavidmeister commented Aug 4, 2018 •

edited

Loading

sphinxc0re Aug 6, 2018

ddd-mtl left a comment •

edited

Loading

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

thedavidmeister Aug 13, 2018

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

thedavidmeister Aug 13, 2018

thedavidmeister Aug 14, 2018

ddd-mtl Aug 10, 2018

ddd-mtl Aug 10, 2018

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

thedavidmeister Aug 13, 2018

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

ddd-mtl Aug 10, 2018

ddd-mtl Aug 10, 2018

ddd-mtl Aug 10, 2018

thedavidmeister Aug 13, 2018

ddd-mtl Aug 14, 2018

sphinxc0re Aug 15, 2018

thedavidmeister Aug 15, 2018

Connoropolous commented Aug 15, 2018

timotree3 Aug 15, 2018

timotree3 Aug 15, 2018

lucksus left a comment

lucksus Aug 18, 2018

lucksus Aug 18, 2018

validate commit MVP implementation #189

validate commit MVP implementation #189

Conversation

thedavidmeister commented Aug 4, 2018 • edited Loading

Choose a reason for hiding this comment

ddd-mtl left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Connoropolous commented Aug 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucksus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thedavidmeister commented Aug 4, 2018 •

edited

Loading

ddd-mtl left a comment •

edited

Loading