Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix on migrating MetaDB tables regarding on dataprep #1072

Closed
joohokim1 opened this issue Dec 11, 2018 · 3 comments
Closed

Bugfix on migrating MetaDB tables regarding on dataprep #1072

joohokim1 opened this issue Dec 11, 2018 · 3 comments
Assignees
Labels
bug Something isn't working @dataprep Component Name : Data preparation
Milestone

Comments

@joohokim1
Copy link
Contributor

Is your feature request related to a problem? Please describe.
I'm going to implement a monitoring thread for the following tasks:

  • Entity migration (at startup)
  • Garbage collection (incomplete snapshots, unreferenced revision sets)
  • Back-ground row counting

다음의 작업들을 위해 monitoring thread가 필요합니다.

  • Entity migration (부팅시)
  • Garbage collection (불완전한 snapshots, 더이상 참조되지 않는 revision set들)
  • Back-ground로 테이블 행 카운팅

Describe the solution you'd like
I want to implement a well-organized monitoring service thread.
Dataprep entity migration will be just one of the tasks of it.

체계화된 monitoring service를 구현하고자 합니다.
Dataprep entity migration은 그 중 한 가지 일로 하면 될 것 같습니다.

Additional context
Never block another thread.
Especially, in JUnit tests using Rest Assure, check if the JUnit testcases get control after the server startup.

절대 다른 thread를 block시키면 안됩니다.
특히 Rest Assure를 이용한 JUnit 테스트에서 서버 startup 완료 후, JUnit thread로 제어권이 돌아오는지 잘 체크해야합니다.

@joohokim1 joohokim1 added enhancement Request Change and Feature Enhancement @dataprep Component Name : Data preparation labels Dec 11, 2018
@joohokim1 joohokim1 added this to the 3.2.0 milestone Dec 11, 2018
@joohokim1 joohokim1 self-assigned this Dec 11, 2018
@joohokim1
Copy link
Contributor Author

@YooYoungmo @kyungtaak
This issue will be addressed in the direction of using Quartz which is widely used in metatron.
By Quartz, synchronization issues between the multiple instances are already well considered, and above all, it would be better to apply a unified way to do similar works across the solution.

본 이슈는 metatron에서 널리 쓰이고 있는 Quartz를 사용하는 방향으로 진행하겠습니다.
Multi instance간의 동기화 문제들도 잘 고려되어있고, 무엇보다도, 솔루션 전체에서 동종의 일을 하는데 일원화된 방법을 적용하는 것이 좋다고 판단되어서 입니다.

@joohokim1 joohokim1 changed the title Prep Monitoring Service Thread Prep Monitoring Service Thread (Using Quartz) Dec 14, 2018
@joohokim1
Copy link
Contributor Author

@YooYoungmo @kyungtaak
Deciding not to do entity migration periodically, we need to separate out other works described above, such as garbage collection and back-ground row counting. In addition, such operations are currently low in implementation priority, so it is unreasonable to tie them to this issue.

So I'm going to solve the entity migration problem so that JUnit tests work well, and I'll do the rest with another issue.

Entity migration을 더이상 주기적으로 하지 않기로 결정하면서, 위에 적은 다른 작업들, 즉 Garbage collection과 Back-ground 행 카운팅 등, 주기적으로 수행할 일들과 구분이 필요해졌습니다.
또한 그러한 작업들은 현재 구현 우선순위가 낮아서, 본 이슈에 묶어서 처리하는 것이 불합리하다는 판단이 듭니다.

그래서 entity migration을 한 번만 수행하도록 바꿔서 JUnit 테스트들이 잘 돌아가도록 해놓고, 나머지 일들은 다른 이슈로 진행하려고 합니다.

@joohokim1 joohokim1 changed the title Prep Monitoring Service Thread (Using Quartz) Bugfix on migrating MetaDB tables regarding on dataprep Dec 17, 2018
@joohokim1 joohokim1 added bug Something isn't working and removed enhancement Request Change and Feature Enhancement labels Dec 17, 2018
joohokim1 added a commit that referenced this issue Dec 18, 2018
* #1072 do not loop dataprep meta-db migration

* #1072 make not running meta db migration to be default
@joohokim1
Copy link
Contributor Author

Works done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working @dataprep Component Name : Data preparation
Projects
None yet
Development

No branches or pull requests

1 participant