[Workstreams] create topics for works

[maxceem]

This is part of #315

Currently, for works we create only one topic with tag like "phase#1231". Instead for works, we should create two topics, with tags like "work#1234-details" and "work#1234-requirements".

[vikasrohit]

@maxceem I am thinking of doing this by consuming the bus event of project creation. And as per new v5 standards, we are creating separate processors for any work that is done by consuming events from bus e.g. https://github.com/topcoder-platform/project-processor-es. Do you think we can write yet another processor for any event based operations which we need to perform within project service. I mean we already have processor for ES indexing of project data and another for legacy operations like creating direct project, so we can create another processor for our future event based work which we are doing right now within project service or project bus consumers. May be we can call this new processor as project-processor-plan which processes events related to the project plan and in this case, it would create a topic on work creation event. We can use this same processor for creating the default work streams for a new project.

[maxceem]

Do you think we can write yet another processor for any event based operations which we need to perform within project service.

1. I think we can do it. The only concern I have is, do you know how fast Bus events are processed in our system? I wonder if it's possible that when we create a new project and go to the project Dashboard, the event is not yet processed, and workstreams with topics are not yet created. This would force the user to reload the page several times and wait until the data is updated.
   We have such a situation with the challenges in community app. As far as I know it's connected with some legacy systems though it's super frustrating. So I just want to make sure that we don't have such situations for Connect App for the new code we are creating.

2. Another thought I have regarding this thing. Is how we can enforce data consistency if something went wrong during some processing. Though for workstreams/posts creating that doesn't sound supercritical. But is there any good practice we are using for such cases? Or we just don't put the critical actions in processors?

3. Meanwhile, for the sake of the fast release, as I'm hoping to release it with comments. Do you think it would be ok, to make this task first using the old approach with Rabbit events?
   And later migrate to the processor. Also, note, that I've already done workstreams creating using just synced code inside project creating endpoint, same like for phases. So we can after migrate several features to the processor.

[vikasrohit]

1. Yes, I can understand that situation. And personally, I don't like to depend on BUS events for situations where the user is expecting some results to see on the screen after operation e.g. when a phase is activated, user wants to see the timeline updated immediately and don't want to refresh the page multiple times to see the updated timeline. And theoretically, timeline and project services are different microservices, so to communicate between them we have to use bus events. That is why, we are handling project and timeline calls directly from front end making sure that user always see the updated results.
   For topic creation of a phase, we are already consuming the rabbitmq event of phase addition. So, I guess, it just a matter of using the kafka bus event instead of rabbitmq for this specific use case.
   Now lets say if the event processing is taking some time after workstream creation, we can show some default message to the user saying that we are creating a discussion thread for the workstream. or how about handling creation of topics for workstreams directly from frontend just like we are managing the phase activation and timeline activation afterwards.

2. The approach we are taking for such situations is that, a event processor should send the error to an error topic common.error.reporting. You can check its usage in https://github.com/topcoder-platform/topcoder-error-logger

3. Yep, I am completely okay to use existing systems as is to speed up things. I started this discussion to check if we can follow the new standards without affecting our timeline.
   And just for making us on the same page, I think creation of workstreams for a new project should always be a sync method just like you implemented. We can not move that to a separate processor (my bad if I said otherwise somewhere else).

[maxceem]

And theoretically, timeline and project services are different microservices, so to communicate between them we have to use bus events.

Could you please share details of your view why they should be separate microservices? So far I only see the usage of timelines inside Project Service. It seems that splitting them would only make it harder for development as we would have to deploy two separate services instead of one. And I don't see any kind of work which can be done in parallel in Project Service and Timeline Service to benefit from such splitting.

or how about handling creation of topics for workstreams directly from frontend just like we are managing the phase activation and timeline activation afterwards.

Actually, I'm kind of like this approach. This would decouple Message Service and Project Service. As now, to show comments, we have to create a topic for the entity on server-side first. If we create it client-side so we can easily and faster add new places with comments by making client-side changes only.

[vikasrohit]

The real world example of having timeline as a separate service is to use it for challenge timeline, if we want to use it. However, that day is very very far in future. But still, I would like to keep things decoupled between direct project related operations and timeline operations.
IMO, the only case where we are coupled, with project related code, right now in timeline related code, is to get the project or its dependent entities when raising or consuming timeline events. That can be skipped as well by getting project related data via apis just like we do for members data right now. However, I don't want to break things right now so will keep that logic as is while keeping any new feature as decoupled as possible.

Actually, I'm kind of like this approach. This would decouple Message Service and Project Service. As now, to show comments, we have to create a topic for the entity on server-side first. If we create it client-side so we can easily and faster add new places with comments by making client-side changes only.

👍

[maxceem]

@vikasrohit regarding creating topics client-side I came into one downside:

1. When we are creating a topic client-side it would be created using the current user. So we login as pshah_manager and create a work and topic for it client-side than topic would be created by user pshah_manager.
2. We should keep the consistency between the topic and work. So every time we rename Work we should rename the title of the topic accordingly, as we do for phases. Also, when we are removing the Work, we should also remove all related topics.
3. Now as the topic has been created by pshah_manager another user couldn't rename topic or delete it only admin. So if some user with permissions to delete work would try to do so, the work would be deleted, but during deleting related topics, there would be an error.

From first glance, I don't see a good solution for this situation. And looks like we have to come back to the way of creating/updating/deleting topics for works server-side.

Maybe you have any ideas to overcome the limitation I've described above using client-side only?

[vikasrohit]

I think as of now we have only option to do it server side. We can update message service to use the same expression evaluator logic for determining the owners of the topic instead of alway relying on the userId field of the topic. I think problem in doing that is right now we are creating phase/work topics with reference being project which is same as of normal discussion topics of the project and hence we can not have two separate access conditions for them which in turn prevents us from having separate owner condition for normal discussion topics and phase/work topics.

[maxceem]

Now, when creating topics for works is implemented, we can think about what should be the first initial post on the Details and Requirements tabs. For now, I defined the same initial post as for phases:

Details tab

Requirements tab

We can easily config this text in Project Service https://github.com/topcoder-platform/tc-project-service/blob/dev/src/events/projectPhases/index.js#L28-L41

[vikasrohit]

Sounds good for now. Do you think is it possible for us to just create a topic without any post?

[maxceem]

No. Message Service always creates a post when we creating a topic https://github.com/appirio-tech/tc-message-service/blob/hotfix/creator-permissions/app/routes/topics/create.js#L68.

[vikasrohit]

Yep, recalled now. I think we might need to hide the default post some time in future, but that depends on the user feedback. If that happens, we can hide the first post from UI itself, so I think we are good.