[Feature][BE] After users remove a board from a blueprint or connection, the data relates to the board will be deleted

[Startrekzky]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

As a scrum manager, I re-organize Jira boards at times. When I first started using DevLake, I collected data from Jira board-1, then I created a new board in Jira called board-2, added board-2, and removed board one from the corresponding DevLake blueprint.

I was expecting that DevLake would delete the board-1 data, but it did not happen.

Description

I want DevLake to support the data deletion of a removed data scope after editing a blueprint, a connection, or any operation that might delete the data scope.

For example, if I collected b1, and then edited the blueprint to ONLY collect b2.

• b1: i1, i2, i3, i4
• b2: i2, i3, i5, i6

b1 in table.boards, i1,i4 in table.issues, and all issue_changelogs, issue_labels, issue_worklogs related to i1 and i4 will be deleted.

i2 and i3 in b1 won't be affected as they also belong to other boards. See the pic below, the shadow part should be kept.

To Do

• Detect if there're boards removed after users save a 'data scope edit'
• If so, prompt the users if they want to delete the data. The default operation is to delete or not (needs discussion)?
  • If users choose to delete the data of the removed data scope, only the data of the existing data scope will be kept.
  • If not, delete the data (belonging to existing boards) automatically when the bp runs the next time

Related issues

Obsolete PR:
#4906

Child PRs:
#5153
#5155

Child Issue:
#5277

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[keon94]

This sounds conceptually similar to deleting a blueprint. We essentially need to know what data belongs to a given scope which is just like the problem with blueprints: what data belongs to a given blueprint? I think both of these problems have to be solved in a similar way

[Startrekzky]

Based on today's discussion, it seems @klesh wants the approach to be simple. If a user is facing the problem described above, deleting board-1 will delete all records associated with board-1, which might include issue-2 and 3, depending on issue-2 and 3 are "tagged" with which data scope, board-1 or board-2.

If issue-2 and 3 are "tagged" with board-1, the deletion will affect the data integrity of board-2, the user has to rerun bp to collect all board -2 issues.

Correct me if I'm wrong. Cc. @keon94 @hezyin @klesh @yumengwang03

[klesh]

Based on today's discussion, it seems @klesh wants the approach to be simple. If a user is facing the problem described above, deleting board-1 will delete all records associated with board-1, which might include issue-2 and 3, depending on issue-2 and 3 are "tagged" with which data scope, board-1 or board-2.

If issue-2 and 3 are "tagged" with board-1, the deletion will affect the data integrity of board-2, the user has to rerun bp to collect all board -2 issues.

Correct me if I'm wrong. Cc. @keon94 @hezyin @klesh @yumengwang03

Yes, that is correct.

[keon94]

But wouldn't this lead to a full data collection for those blueprints? Maybe it'd be better to not allow deletion if a scope has multiple blueprints using it

[klesh]

But wouldn't this lead to a full data collection for those blueprints? Maybe it'd be better to not allow deletion if a scope has multiple blueprints using it

Do you mean that scope that gets deleted is still being included in some bp?

1. it is problematic since the scope record is gone, thus the bp wouldn't function properly, we should do sth about it.
2. as for the collected data, maybe it is a good thing because Incremental Collection can not delete records in the database that were removed from the data source. In fact, we don't have a full collection option at the moment, come to think about it, we should offer a "Clear" button for the scope for deleting data only.

[yumengwang03]

Adding the prototype here: