Policy negotiation flow

[juliapampus]

As discussed last week, @ronjaquensel and I want to share our thoughts about a possible contract negotiation flow containing states - similar to our current transfer process.

.puml files: edc-negotiation.zip

We tried to keep the number of states as small as possible. States on both sides express the same behaviour and, therefore, trigger the same action.

• REQUESTED (prov, cons): A contract request or counter offer has been initiated. The connector sends an appropriate message to the recipient. The state is changed to REQUEST_ACK as soon as the request or a response to a request has been received.
• REQUEST_ACK (prov, cons): The connector has sent a contract request/offer. The connector (on e.g. consumer side) that initiated the request/offer is waiting for a response. The connector that approved the receipt of a request/offer is switching to the next state.
• VALIDATING (prov, cons): Depending on the input (contract request/offer/agreement), the connector validates the contract by comparing it with the contract of the last state. E.g. request - offer, offer - offer, agreement - offer, agreement - request, agreement - agreement. Optionally, the user or another system is being involved in the negotiation process.
• APPROVED (prov, cons): As soon, as the contract's content is approved by the connector, a user, or an external system, the connector builds an appropriate contract agreement. This is stored in a dedicated location. As soon as the initial sender of an agreement has not received a matching contract agreement, the stored one is marked as not confirmed. Next, a message containing the agreement is sent. The status changes to PENDING.
• DECLINED (prov, cons): If a contract is malformed or invalid, or a user wants to abort the negotiation process, the state changes to DECLINED. Thereupon, a contract rejection is sent (and the process gets removed). If necessary, the rejection is logged on both sides and propagated involved users or systems.
• PENDING (prov, cons): This state implies that an agreement has been sent and the connector is waiting for confirmation. This can happen both on provider and consumer side, depending on the one that first agreed.
• CONFIRMED (prov, cons): If the validation of an agreement is successful, the state changes to CONFIRMED. As a result, the stored agreement is marked as confirmed and persisted permanently. Thus, it can be used for later access and usage control.
• ENDED (prov, cons): This is the final state that marks the negotiation process as ended. No further action is required.

Below, you can see a possible negotiation process that is based on the one that has been created some time ago (see here). Please note that the validation process within the negotiation phase was shortened for readability purpose. On top of that, not everything may be necessary for the "happy flow".

Since agreements can expire or be revoked, they have different states - as indicated above.

• INITIATED: not confirmed by the assignee yet
• CONFIRMED: valid contract that can be used for access and usage control
• EXPIRED: the end date has been reached, nevertheless the contract can be extended
• REVOKED: the contract has been ended by e.g. a user with appropriate rights

[jimmarino]

Hi @juliapampus @ronjaquensel

Really thorough! I need to digest this a bit but should there also be an error state from contract negotiation?

[jimmarino]

@juliapampus @ronjaquensel

Another comment: if the contract is malformed, I would tend to model that as an error state as opposed to declined. What do you think?

[DominikPinsel]

@jimmarino @juliapampus @ronjaquensel
I'm also not sure whether a state model is the best solution in case. I think for audibility it might be interesting to have all these state changes in the contract. So maybe we should collect these events somewhere in the contract.
The state would would be read from the contract

• if endDate > now -> expired
• if revokedEvent -> revoked
• if confirmedEvent -> confirmed
• else initiated

Also in theory, it would be possible to go from initiated to expired when the other party just doesn't confirm the contract. And I'd assume a contract could be revoked and expired at the same time. This may be important if we get a use case to do something with all expired or revoked contracts.

[jimmarino]

@juliapampus @ronjaquensel I took the states you defined and started modeling a contract negotiation state machine. I came across some issues and simplifications I think we can make. I'll start with suggested additions and removals:

• Modelled REQUESTED as the state when a provider receives a request from the client. This state is only entered once.

• Removed REQUESTED_ACK since that can be handled by a transition (this will be a transition state, more on that later)

• Introduced the PROVIDER_OFFER and CLIENT_OFFER states. We need to distinguish between these two states since the subsequent transitions will be different. After entering the PROVIDER_OFFER state, the client can transition to CLIENT_APPROVED, DECLINED, or CLIENT_OFFER (or ERROR). If the CLIENT_OFFER state is active, the provider can transition to CONFIRMED. DECLINED, or ERROR.

• Changed APPROVED to CLIENT_APPROVED. Only the client needs to approve since we can assume the provider implicitly approves an offer it sends. The provider can also transition directly to CONFIRMED from the CLIENT_OFFER state.

• Removed VALIDATING. I think we can handle this with existing states.

• Made DECLINED an end state.

Based on these, here is the state diagram I came up with. Transitions happen after events initiated by the client (C), provider (P), or both (C/P). Either a client or provider can transition to the ERROR state from any other state. This is not shown to simplify the diagram.

Let me know what you think and if these changes sound reasonable.

Implementation Notes

1. Contract offers must have a UUID.

2. Contract offers should be stored with the contract negotiation in an ordered list. Each offer should be annotated with the side it originated from.

3. Since contract negotiation is a bipartite system, a consensus algorithm cannot be used to reconcile state differences between the client and provider. To account for this, each participant should include a hash of all contract offer ids that are part of the negotiation in messages it sends to the other participant. If the receiving participant's view of the most recent hash does not align with the one contained in the message, the receiving participant can reject the message, decline the negotiation, or move it to the error state.

4. All messages passed between participants must be done in a reliable manner. This means each message must be idempotent and participants must perform deduplication. In order to accommodate this, the implementation of the state machine will include transition states. These transition states do not appear in the state machine model shared by the participants. The implementation will have *_OFFERING and *_OFFER states. After a request is received on the participant and a counteroffer is ready, the state machine will be transitioned to PROVIDER_OFFERING. The state machine timer thread will then transition to PROVIDER_OFFER after an offer message is sent and acknowledged by the client. Transition states are conceptually similar to a hierarchical state machine, but they simplify the implementation.

Contract Negotiation Implementation

This part outlines an implementation for the contract negotiation process based on the specification created by @juliapampus @ronjaquensel.

Architectural Considerations

The contract negotiation subsystem will be modelled as a state machine, which aligns with the existing design of the transfer process subsystem. This state-machine design is necessary to support the key requirements of asynchronous and reliable operation.

A contract negotiation is conducted between two PariticipantAgents. There will be correlated, persistent state-machines representing a contract negotiation on the client and provider connectors. The ContractNegotition will be a serializable (via JSON) representation of the state machine.

Asynchronous, Reliable Operation

As outlined by the state charts, a ContractNegotiation will transition to various states in the ParticipantAgent runtimes in response to messages. Transitions may be prompted by the participant runtime that contains the negotiation, or by a message received from the counter-party.

The ContractNegotiation will be managed by a ContractNegotiationManager. This service will have subtypes for managing the negotiation state machine on the client and provider. The base ContractNegotiation Manager will define operations common to both sides (these operations are a rough sketch and not complete):

public interface ContractNegotiationManager {

    /**
     * A negotiation was declined by the counter-party represented by the claim token.
     */
    NegotiationResponse declined(ClaimToken token, String negotiationId);

}

The client and provider subtypes will define the following operations respectively (these operations are a rough sketch and not complete):

public interface ClientContractNegotiationManager extends ContractNegotiationManager {

    /**
     * Initiates a contract negotiation for the given provider offer. The offer will have be obtained from a previous contract offer request sent to the provider.
     */
    NegotiationResponse initiate(ContractOffer offer);

    /**
     * An offer was received from the provider.
     */
    NegotiationResponse offerReceived(ClaimToken token, String negotiationId, ContractOffer contractOffer, String hash);

    /**
     * The negotiation has been confirmed by the provider and the final contract received.
     */
    NegotiationResponse confirmed(ClaimToken token, String negotiationId, ContractAgreement contract, String hash);
}

and (these operations are a rough sketch and not complete):

public interface ProviderContractNegotiationManager extends ContractNegotiationManager {

    /**
     * A contract negotiation has been requested by the client represented with the given claims.
     */
    NegotiationResponse requested(ClaimToken token, String correlationId, ContractOffer offer);

    /**
     * A new offer was made by the client represented by the claim token.
     */
    NegotiationResponse offerReceived(ClaimToken token, String negotiationId, ContractOffer offer, String hash);


    /**
     * Confirms a contract negotiation after it has been approved by both counter-parties. A final contract will be sent to the client.
     */
    NegotiationResponse clientApproved(String negotiationId, String hash);

}

All operations will be idempotent and return immediately. However, side effects of those operations will be run on a separate timer thread. For example, a provider may confirm a negotiation. This operation will transactionally update a backing persistent store with the state transition. However, sending a notification and finalized Contract to the client will be handled on another thread. Each manager implementation will contain a scheduled thread that periodically ticks over to handle these transitions.

This design is the same as the TransferProcessManager and enables reliability, failure recovery, and the ability to support asynchronous workflows such as human approval.

Auditability

All metadata associated with a negotiation process will be persisted as part of the ContractNegotiation POJO. This class will contain a list (ordered) of ContractOffers that were made during the negotiation processes. The last ContractOffer in the list is the current. When a ContractNegotiation is confirmed, a Contract will be added that represents the in-force usage contract. Note that only references to Assets will be held by ContractNegotiation.

The ContractNegotiation type will provide a mechanism for adding auditing in the future. It will also have a GUID that can be used for further correlation.

Non-Repudiation

Non-repudiation (the ability for each party to prove the contents of a ContractNegotiation) will not be implemented in this iteration.

PR Descriptions

@jimmarino will provide the following PRs (if someone is interested in working with on this together, please DM).

Part 1: Introduce types and service interfaces

This PR will introduce the basic type and service definitions for ContractNegotiation and ContractNegotiationManager. These will be mostly skeleton implementations and their definitions will likely evolve over time. In addition, ContractOfferService will be updated to add a validation and sanitization method for contract offers received from a client
during contract negotiation. This in turn will require an update to ContractDefinitionServiceto return a sanitized ContractOffer based on a previous copy. Finally, a method will be added to AssetIndexto return a set of assets for a given collection of Criteria. This method is the same as needed for Part 3 of the policy PR.

Part 2: ContractNegotiationManager implementation

This PR will implement the skeletons introduced in Part 1.