Feature: Users / Team / Organization Activities

[evereq]

We need to build a feature that will show the history of activity in Gauzy:

• what actions currently logged-in user did, when, etc.
• what actions members of the team (users who correspond to employees from the given team who login into Gauzy) did, when, etc.
• what actions the organization users did, when, etc.

Note: do not mix it with "Activity" related to Time Tracking! In this Epic we are talking about user actions inside Gauzy Platform, like someone added expense today, someone fire employee yesterday, some employee joined an organization today and so on.

So basically, we need to have activity tabs for:

• User (where we can see what actions user did in Gauzy)
• Employee (where we can see what properties was changed on the employee, when, who did such changes, etc)
• Candidate (we can record different steps in Candidate flow, like "date A applied", "date B interviewed", "date C rating changed", "date D hired" etc)
• Organization (where we can see changes in Organization properties when some employee join company, when some employee quit the company, when salary paid in the Organization, and so on)

The best form for such "Tabs" is to display timeline (top represent recent events, bottom old events).

Example of Events in Activities:

• Ruslan created a new Team called "Designers" on 9:18 10/03/2020 (with links to Team page, to Ruslan User page, etc)
• Alex created a new Project called "WOW" on ...
• Alex salary was paid on ...
• Alex bonus was paid on ...
• Dima Koko left Ever company on ...
• Email notification was sent to User Ruslan K to change his password
• etc

[rmagon]

We can implement it generically like an 'activity logging' service which logs changes in the database along with the user id who sent the request.

[evereq]

@rmagon not only user id, but also it can be changed with user Id = null because some "changes" events could be raised by the Gauzy system (e.g. on schedule) or by integration, etc.
So it's best if table will contain userId, organizationId, TenantId, TeamId, employeeId (for events connected to employees. For such records both userId and employeeId will store same value), projectId, etc. Events itself of course best to store as serialized Json data (because we need them to be very flexible). We can (should) use CQRS for that, i.e. send commands / events / handlers you know what I mean :)

[rmagon]

@evereq I'm planning to implement a basic system initially & ultimately we will have to do some database partitioning (perhaps something like horizontal partitioning so the table structure remains the same but we have multiple tables based on tenants) else the activity log would grow to be huge & if the table grows then the query would become really slow & pulling the data would be slow.
Since we are using the data in the UI realtime, archiving the data wouldn't work for us.
If you have something else in mind which I'm overlooking please let me know 😀

[evereq]

@rmagon think let's work on this task at later stage

[rmagon]

@evereq Ya makes sense

[evereq]

@rmagon my thinking is this way:

• we can later make sharding (partitioning) for a single activity table by few columns, e.g. by "ActivityType" and by "TenantId". So we can easily found all activities of specific "Type" (e.g. Invoice related or Employee related or Organization related, etc) in specific Tenant.
  Plus we can have another table which will store everything sharded only by TenantId and such table can be used for quick query of whole Tenant Activities.
  Plus we can have another table sharded by TenantId and OrganizationId so we can quickly query only specific Organization activities
  Plus we can have another table partitioned by TenantId, OrganizationId and ActivityType for ... you get idea :D

I.e. it's pretty common approach to denormalize such structures and store same activity related data multiple times in different tables for fast queries

Also, we can just have additionally tables for each type of activity separately, e.g. so we have EmployeeActivity Table, OrganizationActivity Table, TenantActivity Table, InvoiceActivity Table etc

I.e. there are LOTs of possible solutions for that and that's why it's best to do it at later stage :)

[evereq]

@prysiazhna did some work on history for candidates, we probably can base the solution on it making it more generic.

[ryusoft]

maybe we could consider another type of data storage as configurable options rather lock in ourself to only postgres, as logging type of data is not so well fit into RDBMS, e.g we could use firestore from Google cloud. or any NoSQL kind.

any concern if I planning ahead on this direction and make it dynamic configurable default to postgres.

[evereq]

@ryusoft we do not lock ourselves to PostgreSQL, because of TypeORM supports lots of different storage options.... Firestore - no go for sure (as it's not open-source... maybe only via some extension), some NoSQL which supported by TypeORM we can consider in the future, e.g. MongoDB. But I wound not focus on that at all, let's keep it simple and in the currently used stack.
I.e. we should have some generic table called "Activities" (or "History"), and just store fields we need like Date, UserId, OrganizationId, TeamId, etc, but also store "activity" data as JSON serialized value for flexibility (so we can push to same table different activities). I.e. we can have TS classes for each type of activity (all inherit from some base class for all activities), and serialize objects of such classes into JSON and store it in the table. And so on.

[ryusoft]

@evereq what i try to suggest is multiple database type. like normal we go for RDBMS and logging type then we can use mongodb.

as logging data and end up very huge then might cause the query very slow.

there are some idea like make a generic and some decorator and make table that we wish to have activity logs and that trigger on typeorm event and transform those changes to JSON, plan around and see doable or not first.

[evereq]

@ryusoft well, at some point we may do it, but not now. For now, I would prefer to keep it simple and focus on functionality first. (otherwise, we end up with Redis, RabbitMQ and many other things very quickly). PostgreSQL can hold easy millions of historical records in one table, we will quickly index by userId / teamId etc and till some time it will be enough, we don't need to introduce another storage (yet). At the same time, regarding idea with a decorator... could be doable, but I would prefer to not make some kind of "framework" on top of TypeORM, unless you can find someone who already did it. In addition, I think we should level up that activity logging to say "service" layer, not do it in DB layer. Because we are interested in Application commands/events logging, NOT really how DB data inserts / changed. So Application Business layer (or services layer in our case) has more knowledge about what should be logged etc. I will give you another example, say someone created record A, but in DB it triggers 10 create operations. We are not interested in writing 10 records to our Activity log, we interested in just one record with lots of specific details, etc.

My line of thinking in such tasks is that: do it like you don't know about some interceptors etc, but take some Junior developer approach (you can see it's halfway done already for activity logging of candidates) and just make it more generic with what we already using. In this case, it's CQRS + some base classes, etc.

[ryusoft]

okay try it out

[ryusoft]

Idea & Proposal:

1. One general Entity with:

   • id (pk)
   • userId, tenantId, (whatever Id of the auditor (user who trigger that we might use to filter)
   • referenceEntity (store the entity reference or tableName ? e.g. "User" or "user"
   • referenceKey (store the primary key e.g uuid if multiple columns one then comma separated)
   • type: string
   • data (serialized, or JSON string)

2. Injectable Service like Repository (Not sure how yet, theoretically should work)
   
   ** and if the CRUD need it, pass it into **

3. With standard Input
   

4. Ideas:
   

5. When any one need to develop new activity "type" it should design the input and output (stand by to be translatable, as now I yet to see backend reading "Accept-Language" header to response with the requested language, a huge works !)

6. I not sure how to do it yet, but the idea is make is general, input just to store the variables required by the "type", then the "type" class, maybe another generic or interface it can receive the data and transform to "output" as string, then when call /candidate/{uuid}/activities, the API should return paginated items: [
   {type: "CHANGE_CANDIDATE_DETAIL", text: "someone change status", timestamp: xxxx, "user": obj },
   {type: "CHANGE_CANDIDATE_DETAIL",text: "another change date", timestamp: xxxx, , "user": obj }
   ]

basically UI only interested on the "text", and user (maybe need avatar) and render it

[evereq]

@ryusoft few comments:

1. I would prefer to use https://docs.nestjs.com/recipes/cqrs#events instead of directly call such activity service from other services. For example, let's say something changes (e.g. Candidate get hired as Employee). In such case, we will probably handle somewhere command like "CandidateHiringCommaon" which itself will produce event like "CandidateHiredEvent" and inside one of handlers for such event, we can make a call to activity logging service (e.g. inside CandidateHiredActivityHandler handler). That way all our services will not have direct knowledge of activity logging service etc.
2. It's not a text really when we display in UI. More like for different types of activities, we will have Angular Templates which can be rendered to represent such activity log record. So we will read JSON from DB, deserailize it to object, pass that object to some Angular component (which will be decided by the name of activity type) and such Angular component renders with data from that object. That is very flexible way where we can easy change how different types of activities rendered in UI, while still save it all in one table in DB etc.
3. Don't see any issues with translations, it will be done in Angular templates for each activity type, like we are doing in many other places.