What and why to refactor
As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the _devlake_subtasks table occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.
Describe the solution you'd like
Solutions from @klesh
- Reduce the update rate to _devlake_subtasks table when collect data.
- Store progress info in memory firstly and write it to db in a fixed rate.
Related issues
No
Additional context
- The issue was introduced in v1.0.
- We are using postgres.
What and why to refactor
As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the
_devlake_subtaskstable occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.Describe the solution you'd like
Solutions from @klesh
Related issues
No
Additional context