Validation logic

[pospi]

Beyond schema validation we need to define all the custom validation rules that apply to records at the API boundary. This includes validation for 'either' fields as well as any other custom dependencies, existence checks and domain-specific constraints that need to be imposed in order for requests to be considered valid.

Starting with the observation, planning & specification records (in that order) would be optimal in terms of informing HoloREA's development as it progresses.

[bhaugen]

@jvanbockryck did you learn anything from Wim that would be useful for validations? I know he was working on documenting and implementing REA model constraints.

[jvanbockryck]

Yes @bhaugen, Wim is going to send me documentation on this topic. The approach can be as on any other data modelling language that you extend with a formal constraint language. I've done such things on UML before (see INSPIRE data specs, they are full of them), but that's not so useful to generated anything from, although some have tried with some success (see shapechange.net, transformations section).

Wim's proposition is doing this on the ontology layer, so that constraints could be generated for multiple constraint languages at a lower level (e.g. to RUST/Holochain validations). I've had good results with this approach in the past on the topic of Complex Event Processing, that deals not only with constraints on the schema - object/events instance - but also on a series of events, dependencies between events. So I "clicked" with him on that approach.

I like to work out an example for this approach for Value Flows, as it could then be used to generate constraints for any other lower level implementation (if there's a demand for that), but Holochain/HoloREA would be my first to try this out. I say Holochain/HoloREA, because I think it is useful for all Holochain apps, not only HoloREA. With that example and with the input from Wim, I would like to show this to @pospi to see if it would make sense to him.

[fosterlynn]

Observation layer validations

Note: This is a draft, comments appreciated. Meant to be implementation agnostic. Doesn't include individual required fields, which are already documented in the graphql and json-schema specs. [edit: I'm adding them for completeness and so reviewers don't have to look at those references.]

Process

• name is required
• (thought about requiring hasBeginning and hasEnd, but I can see edge cases where they wouldn't make sense, like a truly ongoing process)

EconomicEvent

• (note that occasionally in the case of services, an event could be output of one process and input to another)
• action is required
• provider is required
• receiver is required
• if the event action is not defined as input and/or output, it should not be related to a process
• resourceInventoriedAs or resourceClassifiedAs or resourceConformsTo is required
• resourceQuantity or effortQuantity is required
• hasBeginning or hasEnd or hasPointInTime is required

EconomicResource

• conformsTo is required in the graphql definition (but as I think about it, I don't think that should be required, some people might want a more minimal implementation)
• (what do you think about resourceQuantity @bhaugen ? I think not, some things are obviously 1 one)

Fulfillment

• fulfilledBy is required
• fulfills is required
• (I think we don't want to require a quantity, it can be assumed to be the same as the event if not recorded - disagreement?)

Appreciation

• appreciationOf is required (need to fix graphql)
• appreciationWith is required (need to fix graphql)

[fosterlynn]

Planning layer validations

Intent

• action is required (although this may be too rigorous? we could assume transfer)
• provider or receiver is required
• (I don't think anything else is required, in combination or singly, in deference to the simplest implementations of intents, which are basically text)

Commitment

• action is required
• provider is required
• receiver is required
• if the event action is not defined as input and/or output, it should not be related to a process
• resourceInventoriedAs or resourceClassifiedAs or resourceConformsTo is required
• resourceQuantity or effortQuantity is required
• hasBeginning or hasEnd or hasPointInTime or due is required

Satisfaction

• satisfies is required
• satisfiedBy is required
• (I think we don't want to require a quantity, it can be assumed to be the same as the commitment or event if not recorded - disagreement?)

Agreement

• (no validations imo - need to fix AgreementCreateParams)

Claim

• action is required
• provider is required
• receiver is required
• resourceInventoriedAs or resourceClassifiedAs or resourceConformsTo is required
• resourceQuantity or effortQuantity is required
• (should triggeredBy be required? I think not)

Settlement

• settles is required
• settledBy is required
• (I think we don't want to require a quantity, it can be assumed to be the same as the event if not recorded - disagreement?)

Plan

• name is required

Scenario

• name is required
• (we have definedAs (a ScenarioDefinition) as required, but thinking about it now, I don't think we want to require it)

[fosterlynn]

Specification validations

ResourceSpecification

• name is required

ProcessSpecification

• name is required

[pospi]

@jvanbockryck you're talking about a lot of things there that I have no familiarity with, but I'm game. When have materials ready for discussion just hit me up :)

Is what you're proposing basically a way of generating the constraints that Lynn has outlined above? Or is it a different type of constraints that you're talking about?

[jvanbockryck]

Well, the ones listed by @fosterlynn of mostly of one type of constraint, "is required" and the occasional "or" between attributes.
There are other types I see that apply to one object and related objects:

• about cardinality
• about format of a string for IDs of associations (this goes beyond datatype validation, but that is off course assumed being taken care of via schema validation)
• about existence of the object that is associated (is the associated object existing?)
• about the correct object that is associated (is the associated object the correct one?)

Then there are other more complex types that I'm going to demonstrate with my example, later on.

[fosterlynn]

about format of a string for IDs of associations

If I understand what you mean correctly, this might be different for different technologies, for example in Holochain it is the hash token, for linked open data it is a uri. So it might be defined more generally for the VF app specs and more specifically for a technology-related validation.

[jvanbockryck]

Yes, @fosterlynn , that's correct on the format.

But also, the referenced objects must exist.

[pospi]

@jvanbockryck on the topic of these:

• about existence of the object that is associated (is the associated object existing?)
• about the correct object that is associated (is the associated object the correct one?)

In Holochain these require some thought towards cyclic inter-network dependencies, see this forum thread (CC @pdaoust)

But all these constraint types you are talking about would be really good things to formalise :)

[pospi]

@fosterlynn it occurs to me that all the validations given so far are in the context of creating new records.

Would you be able to provide a similar list of constraints for update operations?

[pospi]

RE EconomicResource.conformsTo does it makes sense to require either that parameter OR the associated EconomicEvent.resourceClassifiedAs?

[pospi]

Also in terms of the "adjoining" records, I think I prefer to have the quantity explicitly stated in the Fulfillment / Satisfaction / Settlement. Otherwise it's just another condition that has to be programmed in to every app...

[fosterlynn]

Oops, never sent this....

about cardinality

I can help with that if needed. That is also defined in the graphql spec, but not so easy to look at it there.

about existence of the object that is associated (is the associated object existing?)

When I said "required" that should also mean that for relationships the object exists. Will that work?

But anyhow @jvanbockryck good ideas!

Would you be able to provide a similar list of constraints for update operations?

Yes, can do.

RE EconomicResource.conformsTo does it makes sense to require either that parameter OR the associated EconomicEvent.resourceClassifiedAs?

I think that either EcononomicEvent.resourceConformsTo or EconomicEvent.resourceClassifiedAs if available on the event should be also put on the EconomicResource. At least on create. Does this answer the question? I think I'm missing the thought process behind the one you posed?

Also in terms of the "adjoining" records, I think I prefer to have the quantity explicitly stated in the Fulfillment / Satisfaction / Settlement. Otherwise it's just another condition that has to be programmed in to every app...

OK. The UI can handle it then if there is no functional need to have both for some use cases, which will frequently be true. We did decide to do them as separate mutations, right?

[fosterlynn]

Would you be able to provide a similar list of constraints for update operations?

Actually, I think they are a subset of the ones defined for create. Nothing is required besides the ID. Then people can't remove anything that was required on create, like set to null or take to zero, etc.

[pospi]

I think that either EcononomicEvent.resourceConformsTo or EconomicEvent.resourceClassifiedAs if available on the event should be also put on the EconomicResource. At least on create. Does this answer the question?

Half way there! Confirming: currently EconomicResourceCreateParams.conformsTo is required. I should make this not required; and if it's not present then EconomicEvent.resourceConformsTo is used. But if that latter field is not provided either, then it's an error.

EconomicEvent.resourceClassifiedAs is already stored in EconomicResource.classifiedAs on creation, so that much is done.

[pospi]

Also @fosterlynn curious whether move is meant to mess with quantities. Previously it did not affect them (I had presumed you'd do the transfer-* action independently), but after the rewrite I've been doing it now decrements resourceInventoriedAs. Unsure if this is intentional or not.

There are also no requirements on a pairing of resourceInventoriedAs / toResourceInventoriedAs that I've seen... are there any action types which require one or both to be specified?

[pospi]

RE updates, this is the current situation. Please correct as needed:

• EconomicEvent & EconomicResource we're mostly done with in EconomicEvent and EconomicResource create and update #4, excluding the above.
• everything in Process is editable, including deleteable which I'm pretty sure should be a readonly field.
• for Commitment it is only the inputOf and outputOf links which are not editable at the moment; I'm presuming most of the accounting fields should not be editable but am not sure.
• for Intent it is pretty much the same deal- everything except the Process links are editable, and probably mostly shouldn't be.
• Satisfaction and Fulfillment are currently fully editable, but I expect they should be non-editable and that reversing records should be appended to handle edits, "accountant's logbook" style. I also haven't wired up anything to do with resource math in response to satisfactions & fulfillments, I guess that needs speccing out somewhere (please link me!)

[fosterlynn]

Also @fosterlynn curious whether move is meant to mess with quantities. Previously it did not affect them (I had presumed you'd do the transfer-* action independently), but after the rewrite I've been doing it now decrements resourceInventoriedAs. Unsure if this is intentional or not.

Depends on what you mean by "mess with quantities". A move will often result in a different resource identifier, even though visually it is the same set of material items. This is because some operations include location in the logical identifer, so there will be a different technical identifier on the from and the to resources. So, in those cases, the from is decremented and the to is incremented. Note also the to resource may already exist, or not.

A move could also just update the currentLocation property, and keep everything else, if there is not a toResourceInventoriedAs on the move. That is for cases where the location is not part of the logical identifier.

I see move as independent of the transfers, happens if there is not a change of agents.

cc @bhaugen

[fosterlynn]

There are also no requirements on a pairing of resourceInventoriedAs / toResourceInventoriedAs that I've seen... are there any action types which require one or both to be specified?

I don't think so. Agents can always have resources that are not inventoried in their computer system explicitly, is an administrative choice.

cc @bhaugen

[fosterlynn]

everything in Process is editable, including deleteable which I'm pretty sure should be a readonly field.

Right, deletable is never editable on any record, the app figures that out.

for Commitment it is only the inputOf and outputOf links which are not editable at the moment; I'm presuming most of the accounting fields should not be editable but am not sure.

for Intent it is pretty much the same deal- everything except the Process links are editable, and probably mostly shouldn't be.

I honestly don't see anything on Commitment or Intent that is above the "# inverse relationships and queries" line that can't be edited. People make data entry mistakes. This assumes that the app is keeping track of these changes functional-style - which I understand both holochain and activity pub do out of the box. Any disagreements and is my understanding correct?

Satisfaction and Fulfillment are currently fully editable, but I expect they should be non-editable and that reversing records should be appended to handle edits, "accountant's logbook" style. I also haven't wired up anything to do with resource math in response to satisfactions & fulfillments, I guess that needs speccing out somewhere (please link me!)

Thinking about it now, seems like we should add a validation that satisfaction and fulfillment should not have quantities greater than that which they satisfy or fulfill. Same with settlement.

Yeah, I think reversing entries would be appropriate for Fulfillments and Settlements, I can see where they might affect accounting entries like receivables or payables. Intents don't need so much ceremony around them, as they aren't part of any periodic accounting I can think of. So it seems to me that Satisfaction could allow anything to be editable?

Please double check me @bhaugen ?

[pospi]

satisfaction and fulfillment should not have quantities greater than that which they satisfy or fulfill. Same with settlement.

I didn't think that was the case... what about over-satisfying?

RE the move event stuff, this manifested for me as unexpected behaviour when this test assertion failed and I eventually traced the problem back to this event decrementing the resource quantity. I wasn't expecting that, and the logic must not have been clear because I interpreted that as a passing test earlier.

It seems to me that if you are going to move some resource out of an inventory, you want to move it into another inventory? And if you're not moving it to another inventory, aren't you usually throwing it away or recycling it, and hence either consume or pickup or transfer-complete would be more appropriate? I kinda feel like for move events there should be a validation on either both inventory fields being required, or neither being present.

Also no confirmation yet on- currently EconomicResourceCreateParams.conformsTo is required. I should make this not required; and if it's not present then EconomicEvent.resourceConformsTo is used. But if that latter field is not provided either, then it's an error?

[pospi]

Re Commitment, well, since you can specify quantities with it and therefore have negative quantities (right?) or at least reciprocal actions, it might make sense to treat it more like EconomicEvent and not have editable "accounting fields". That would mean that only note, agreedIn, clauseOf, independentDemandOf & inScopeOf are editable. You would also want to have finished editable so that one can indicate when both parties are happy that it has been satisfied.

The reason I suggest this is that in a distributed environment, I'm not sure what importance remote actors might place on the presence of Commitment records. We have already introduced some complexity into the EconomicEvent records by adding some (necessary) counter-signing and an (optional) dispute resolution layer. The question is: how do systems not using the dispute resolution module treat a contested EconomicEvent? @fosterlynn @bhaugen and I decided the answer was, they wouldn't see it. In the base framework observation module, we code get_economic_event such that it only retrieves records which have been signed by both provider and receiver. At an application layer, UIs which allow contesting of events would not perform the counter-signing action prior to logging a dispute claim.

A similar issue exists with third-party modules interpreting Commitments. It would generally make their jobs much simpler if they could reason to the effect of "hey, a Commitment exists here, so these two agents have formally agreed to do something together"; the same way that the above approach allows other modules to reason about EconomicEvents as records that have been deemed as truth by at least 2 parties (OK, provider and receiver can be the same but I digress). As a third-party app developer I don't want to have to think about "did this EconomicEvent actually occur or is one agent lying about it?" and "is this Commitment actually agreed upon or is one of the parties going to shirk responsibility suddenly?"

So I think if you need to alter fields in either of the above, maybe it's an "append new record to correct" thing?

But then I wonder about Intent and its role in offers / needs matching marketplaces. Is there a similar pattern of logic here for Intent, or is editing those the one thing that it makes sense to loosely evolve over time?

@tommycp3 @bhaugen @fosterlynn care to comment on the above from the perspective of real industry use-cases?

[fosterlynn]

I didn't think that was the case... what about over-satisfying?

I'm re-thinking, and I think you are right, people can put what it is, and over-satisfying is fine. In general, I want to be more careful about over-specifying validations.

I kinda feel like for move events there should be a validation on either both inventory fields being required, or neither being present.

What about the case where it is changing location but not identification - they should enter the resource id in both places? Because they'll need the resourceInventoriedAs to properly identify the resource to be moved. In that case, we could leave the resource identified as is, meaning no decrement or increment. What do you think? Could also work to require both or neither.

Confirming: currently EconomicResourceCreateParams.conformsTo is required. I should make this not required; and if it's not present then EconomicEvent.resourceConformsTo is used. But if that latter field is not provided either, then it's an error.

Sorry I have probably confused this, I keep forgetting the change we made to the definitions of conformsTo and classifiedAs here valueflows/valueflows#520.

But, talked to @bhaugen , and we are thinking that we should not be too prescriptive, and let people have all resource references as optional. For example in a timebank near us, they have basically a description, and some very non-specific classifications, at most, for the service being given in return for mutual credit. So that means, not required on the event. keep if it's not present then use it from the event. But no errors. Should we override the resource.conformsTo if another event comes along with a different event.resourceConformsTo?

Re Commitment, well, since you can specify quantities with it and therefore have negative quantities (right?) or at least reciprocal actions, it might make sense to treat it more like EconomicEvent and not have editable "accounting fields". That would mean that only note, agreedIn, clauseOf, independentDemandOf & inScopeOf are editable. You would also want to have finished editable so that one can indicate when both parties are happy that it has been satisfied.

Yes, it may not have all the unchangeables that events have because there is no need to use it to recreate resources. Talking with @bhaugen , his suggestion is that commitments could be changeable (if agreed by both agents) until there is an event. Once an event happens, the commitments have to be backed out and re-put-in, because if an event happens, then there is an (implied) claim, which would translate to a receivable or payable on an agent's books, so now we are into accounting periods that might be closed. The ones I see that would affect the books are: the quantities and inScopeOf (that's whose books it will show up on).

[pospi]

Could also work to require both or neither

I'll take that as a nod to lock in this validation for move.

RE EconomicResource.conformsTo, we previously decided that it was best to have it only editable via the EconomicResource API. But then in here I think that we decided it shouldn't be editable at all, because it has accounting-like repercussions and so it should require zeroing the resource via adjustment events and creating a new one.

Should we override the resource.conformsTo if another event comes along with a different event.resourceConformsTo?

I think the answer here is "no", for the same reason that we shouldn't be able to edit conformsTo.

So that means, not required on the event. keep if it's not present then use it from the event. But no errors.

I read that as you saying that EconomicResource.conformsTo is an optional field, which I don't think is correct. My interpretation is that for a newly created resource, conformsTo is set from:

• EconomicResourceCreateParams.conformsTo if provided, which may also override the value set in the related EconomicEvent.
• EconomicEvent.resourceConformsTo if provided
• if neither of the above fields are present, the resource / event creation operation is an error.

[Commitment] may not have all the unchangeables that events have

For clarification: note, agreedIn, clauseOf, independentDemandOf & finished are editable? Can you please confirm the exact field list?

Do we need more complex validation regarding "until there is an event"? This sounds like a separate, more complex task. Does this involve following links through a Process' inputs & outputs, or Fulfillment links, or both?

I am not sure I agree that clauseOf does not have far-reaching implications RE dealing with edits. Is that not likely to trigger (potentially irreversible) actions in a linked Agreement?

Also, there is a problem if inScopeOf is seen as an accounting field. It has been made editable for most of the other record types which possess it.

[fosterlynn]

A general thought: I'm having trouble thinking about the implications of distributed data for a lot of this. And there are a lot of very different use cases for HoloREA, many of which we have not encountered. I think we have to get some more experience. I don't know whether that means we go in with minimal validations and add them as we learn; or go in with more validations and subtract them as we learn. The other factor that I don't yet understand in practice is what the responsibilities of HoloREA are as a "framework". That may imply looser validations, if the architecture will allow apps using the framework to add their own, which I don't know? But in general, I don't want to go too crazy trying to define something we can't fully understand yet.

I read that as you saying that EconomicResource.conformsTo is an optional field, which I don't think is correct.

Yes, @bhaugen and I were re-thinking the earlier decision, sorry about the change. Thinking about some of the simpler implementations people might want to do. Is there any reason the app requires it to be present for its internal logic?

[Commitment] may not have all the unchangeables that events have

For clarification: note, agreedIn, clauseOf, independentDemandOf & finished are editable? Can you please confirm the exact field list?

I don't see any problem with adding as editable resourceClassifiedAs, due, atLocation, off the top of my head. Might be more, but we can leave them as is. @bhaugen ?

Do we need more complex validation regarding "until there is an event"? This sounds like a separate, more complex task. Does this involve following links through a Process' inputs & outputs, or Fulfillment links, or both?

An addition to what we said earlier: This only applies to commitments that are part of an Agreement (has a clauseOf). Commitments that are only related to processes shouldn't affect any accounting reports as far as we can think. In terms of how to tell, you could query for EconomicEvents with realizationOf equal to the clauseOf the commitment. Unless we aren't saving realizationOf when we create a Fulfillment, then you would have to check all the commitments of that agreement for fulfillments. (Either is fine, we just need to be consistent.)

Also, there is a problem if inScopeOf is seen as an accounting field. It has been made editable for most of the other record types which possess it.

inScopeOf will often be the scope the accounting is being done for. Like the name at the top of a balance sheet or income statement could be the name of the inScopeOf object. On the other hand, not everything goes into that kind of periodic report that is officially 'closed'. Just EconomicEvents and Commitments that are part of an Agreement is I think our working definition. So it would be not editable for those.

On the other hand, if change of scope means a record could be in the wrong dht or something like that, then we shouldn't allow editing it, people will have to delete it and re-enter. Will that ever happen?

[pospi]

Is there any reason the app requires [EconomicResource.conformsTo] to be present for its internal logic?

The only reason so far is for inferring EconomicResource.unitOfEffort when initialising resource quantities and measures. We also decided to update unitOfEffort every time an event is logged against the resource with a resourceConformsTo.unitOfEffort which differs from the resource's existing unit.

This basically means allowing EconomicResources which only have a classifiedAs and do not link to any ResourceSpecification. You're the experts! Fine by me if it's fine by you (;

On the other hand, if change of scope means a record could be in the wrong dht or something like that, then we shouldn't allow editing it, people will have to delete it and re-enter. Will that ever happen?

Probably. It's really hard to say without having come to any final conclusions on what group agents mean in Holochain. I think there might be cases where inScopeOf implies a separate DNA, and other cases where it just implies a usergroup within a wider DNA. I'm going to leave it editable for now and revisit when it becomes clearer.

On Commitment & Intent, I'm finding getting clarity from you on what is updateable really difficult. I'm going to go ahead with updating Commitment so that only these are editable:

note, resource_inventoried_as, resource_classified_as, resource_conforms_to, has_beginning, has_end, has_point_in_time, due, at_location, plan, agreed_in, clause_of, independent_demand_of, finished, in_scope_of

I feel iffy about making the time fields editable, but given there is revision history this might be OK. That said, you might want to create some constraint in the app specs that revisions must be kept for Commitments & Intents if you are going to allow these sorts of edits. I guess my main question is- if a Commitment is broadcast to a nearby work planning coordination space and subsequently needs to be "de-listed", what is the process its original author must go through in order to do that?

I still don't have any guidance on an editable field list for Intent, and we haven't talked about the role it plays in marketplace listings. So, same question here: if an Intent is broadcast to a nearby marketplace network and subsequently needs to be "de-listed", what is the process its original author must go through in order to do that?

[fosterlynn]

On Commitment & Intent, I'm finding getting clarity from you on what is updateable really difficult. I'm going to go ahead with updating Commitment

I understand, and am not immediately sure how to improve the state of knowledge on some of it. Certainly we can consolidate the messy history of discussion, and also derive from the code. In the meantime, I think best to just go for your suggestions (which you probably did already), and we'll shake it out with some users.

I'll document the Commitments with Events thing in your separate issue.

if a Commitment is broadcast to a nearby work planning coordination space and subsequently needs to be "de-listed", what is the process its original author must go through in order to do that?

For Commitments, I think the coordination is contained within the IPO structure, distributed or not. I don't see this as "broadcast". So it is just a change or a delete, same for everyone coordinating a process. Depending on the use case, notifications are very useful for this kind of coordination.

I still don't have any guidance on an editable field list for Intent, and we haven't talked about the role it plays in marketplace listings. So, same question here: if an Intent is broadcast to a nearby marketplace network and subsequently needs to be "de-listed", what is the process its original author must go through in order to do that?

To "de-list" an Intent, change the Proposal.hasEnd date to now. In this case the model is explicit about the listing of intents.

you might want to create some constraint in the app specs that revisions must be kept

good idea, will do

[pospi]

To "de-list" an Intent, change the Proposal.hasEnd date to now. In this case the model is explicit about the listing of intents.

Ok. Does that mean that marketplace-type networks always need to plug in to Proposal functionality as well as Intent, Commitment & EconomicEvent?

[pospi]

Rather than go with my prior suggestions I'm going to err on the side of being permissive for now. So that means all the accounting fields of Commitment and Intent are editable for the moment, until h-REA/hREA#93 is closed. Let's go with that and see what happens?

Ping me of any further updates to this logic so that I can address in Holo-REA as requirements solidify, but I'll go ahead and submit a PR for now.

[fosterlynn]

To "de-list" an Intent, change the Proposal.hasEnd date to now. In this case the model is explicit about the listing of intents.

Ok. Does that mean that marketplace-type networks always need to plug in to Proposal functionality as well as Intent, Commitment & EconomicEvent?

Yes, that is what it is for - for when intents need to be published out to find matches.

Rather than go with my prior suggestions I'm going to err on the side of being permissive for now. So that means all the accounting fields of Commitment and Intent are editable for the moment, until h-REA/hREA#93 is closed. Let's go with that and see what happens?

OK, let's give it a try.

[pospi]

Are there any validation rules for Proposal? Currently all fields seem to be optional.

[fosterlynn]

I don't see anything that is absolutely required. Something has to be there, but won't always be the same thing, but I also don't see getting into some either or or or thing. Let's let specific apps figure it out.

[almereyda]

We have moved the ValueFlows organization from GitHub to https://lab.allmende.io/valueflows.

This issue has been closed here, and all further discussion on this issue can be done at

https://lab.allmende.io/valueflows/vf-apps/-/issues/5.

If you have not done so, you are very welcome to register at https://lab.allmende.io and join the ValueFlows organization there.