New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sequence to Events emitted by Runtime Bundle #2142

Closed
salaboy opened this Issue Nov 9, 2018 · 12 comments

Comments

2 participants
@salaboy
Copy link
Member

salaboy commented Nov 9, 2018

Each event sent by the Runtime Bundle should include a sequence number which helps to identify the order of the event inside the transaction. Each event is providing a timestamp, but this is not enough to understand which event happened first from the consumer side (ie audit, query).

@salaboy salaboy added this to the Beta3 milestone Nov 9, 2018

@salaboy salaboy added this to Open in Activiti 7.x via automation Nov 9, 2018

@salaboy salaboy modified the milestones: Beta3, Beta4 Nov 9, 2018

@salaboy salaboy added the api-events label Nov 12, 2018

@salaboy salaboy modified the milestones: Beta4, Beta5 Nov 25, 2018

@miguelruizdev miguelruizdev self-assigned this Dec 10, 2018

@salaboy salaboy modified the milestones: Beta5, RC1 Dec 14, 2018

@salaboy salaboy added the risk label Jan 7, 2019

@salaboy salaboy modified the milestones: RC1, RC2 Jan 8, 2019

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 17, 2019

We need to clearly define the usage and presence of this new field.

@salaboy is the scope of the field sequenceNumber attached to the entity it contains?
The same way that timestamp scope is the calendar itself, we need a definition to base the sequenceNumber upon.
If we get a list of events based on entityId like this:

GET /audit/v1/events?sort=timestamp,desc&sort=id,desc&search=entityId:2b3e0dbb-19db-11e9-b5dd-0a580a18004b

We get this:

"events": [
            {
                "id": "6f76b594-15fc-4edb-82fd-16444433b3dc",
                "timestamp": 1547676480762,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "new-task-name",
                    "status": "CANCELLED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "dueDate": "2019-01-17T22:07:51.941+0000",
                    "priority": 3,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "new-task-form-key"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_CANCELLED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "426d588e-463c-4ea5-8319-e3d225960c09",
                "timestamp": 1547676473546,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "new-task-name",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_UPDATED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "0a6f5174-d364-407c-858b-808c42ff6e4a",
                "timestamp": 1547676472553,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "my-task",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_ASSIGNED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "8484b58f-01df-46c1-a43d-402748a6631f",
                "timestamp": 1547676472553,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "my-task",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_CREATED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            }
        ]
    }

The intention behind this issue is to have a field in each of the events that establishes a sequence number based on chronological order, but without relying in the timestamp, right?

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 17, 2019

@salaboy A different approach would be populating that field once is persisted in the database on the consumer end, without the need for establishing in the runtime bundle.

@salaboy

This comment has been minimized.

Copy link
Member Author

salaboy commented Jan 17, 2019

@miguelruizdev we need to generate from the producer side.. in the same way that we do with the timestamp.
So sequence should be there.. I am starting to think that we need a unique (transaction/)message identifier as well. Remember that we batch all the events of the transaction together.

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 18, 2019

@salaboy we set the timestamp at the moment of instantiation in every event:
https://github.com/Activiti/activiti-api/blob/develop/activiti-api-model-shared-impl/src/main/java/org/activiti/api/runtime/event/impl/RuntimeEventImpl.java#L37
Along the same lines, what should be best the place for the SequenceNumber to be set?

The value of sequenceNumber will depend on the existence of other events, so that place needs to have that information.

Also, the lifespan of this sequence should be bound to the execution of a process, right?

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 18, 2019

Regarding the transaction unique identifier, do we have model for transactions?
Also, what is the criteria to start and end one of those batches of events?

@salaboy

This comment has been minimized.

Copy link
Member Author

salaboy commented Jan 19, 2019

@miguelruizdev after discussing this with @erdemedeiros and @ryandawsonuk I think that we agreed that we should:

  1. pick up the message from the Stream message containing the Event. There are some examples on how to inject the @Header of the message received from a @StreamListener. we can use that as the unique message identifier.
  2. We can create and store the sequence of the events based on the array of events that we receive in Audit. We just need to store that into the Event itself.

I would start with that basic approach first.. then we can see if we need to change Query as well.

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 20, 2019

Then, I guess that the place where to start in RB is the MessageProducerCommandContextCloseListener.

Regarding the changes in the api layer, I think that the addition of a method in CloudRuntimeEvent and its implementation will suffice.

@salaboy

This comment has been minimized.

Copy link
Member Author

salaboy commented Jan 20, 2019

@miguelruizdev you need to start in query.. both of my points were in query service. No need to touch RB

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 21, 2019

@salaboy @erdemedeiros @ryandawsonuk regarding the value of the Header the message carries, will it be enough to have a UUID or do we need a specific value already present? Will it be a value we extract from the execution itself?

@miguelruizdev

This comment has been minimized.

Copy link
Contributor

miguelruizdev commented Jan 21, 2019

Regarding the id of the batch of events processed by audit, the logical place for those changes to take place seems to be AuditConsumerChannelHandlerImpl.

But then, one question arises: Audit is a dumb consumer since we don't process the data we get from the outside world, we just store it.
With changes in AuditConsumerChannelHandlerImpl to set the trasanctionId, we're kinda breaking that definition cause we'd be actually modifying the objects we receive.

Is this right from a design point of view?

@salaboy

This comment has been minimized.

Copy link
Member Author

salaboy commented Jan 21, 2019

@miguelruizdev it is totally right as soon as we decorate the data that we are getting. So it is ok to do it there.

@salaboy

This comment has been minimized.

Copy link
Member Author

salaboy commented Jan 30, 2019

merged

@salaboy salaboy closed this Jan 30, 2019

Activiti 7.x automation moved this from Open to Closed Jan 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment