Skip to content

[Refactor][core]Data inflation when using postgres #8142

@narrowizard

Description

@narrowizard

What and why to refactor

As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the _devlake_subtasks table occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.

Describe the solution you'd like

Solutions from @klesh

  1. Reduce the update rate to _devlake_subtasks table when collect data.
  2. Store progress info in memory firstly and write it to db in a fixed rate.

Related issues

No

Additional context

  • The issue was introduced in v1.0.
  • We are using postgres.

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions