Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support granularity automation on ingestion stage #1058

Closed
kyungtaak opened this issue Dec 10, 2018 · 1 comment
Closed

Support granularity automation on ingestion stage #1058

kyungtaak opened this issue Dec 10, 2018 · 1 comment
Assignees
@eltriny @brandon-wonjune
Labels
@datasource Component Name : Datasource/Connection Management enhancement Request Change and Feature Enhancement
Milestone
3.2.0

Comments

@kyungtaak
Copy link
Contributor

Is your feature request related to a problem? Please describe.
현재 데이터 적재 단위인 granularity 지정시 고정된 설정으로 지정하게 됩니다. granularity 에 생소한 사용자들은 기본 설정으로 지정하여 druid 내 공간/성능에 이점을 누리지 못할 가능성이 있습니다. 이에 대해 일반적인 granularity 규칙을 넣어, 사용자가 조금이나마, granularity 를 편리하게 지정할수 있도록 자동화 할 필요성이 있습니다. (MT-38)

Describe the solution you'd like

  • 아래 처럼 timestamp field 의 포맷을 인식하여 보편적인 Granularity 자동화 규칙을 적용합니다.
    image
  • 실시간 데이터소스 질의시 필요하여 Query granularity에 Second 단위를 추가합니다.

Describe alternatives you've considered
(N/A)

Additional context
(N/A)
@kyungtaak kyungtaak added enhancement Request Change and Feature Enhancement @datasource Component Name : Datasource/Connection Management labels Dec 10, 2018
@kyungtaak kyungtaak added this to the 3.2.0 milestone Dec 10, 2018
@brandon-wonjune brandon-wonjune self-assigned this Dec 10, 2018
@brandon-wonjune
Copy link
Contributor

f#1058-granularity_automation_in_ingestion 브랜치 생성하였습니다

brandon-wonjune added a commit that referenced this issue Dec 10, 2018
@brandon-wonjune
#1058 add granularity automation logic
94f34f6
brandon-wonjune added a commit that referenced this issue Dec 12, 2018
@brandon-wonjune
#1058 fix change schema logic
19f0cb3
brandon-wonjune added a commit that referenced this issue Dec 12, 2018
@brandon-wonjune
#1058 add changed timestamp field flag in schemaData
36aafb6
brandon-wonjune added a commit that referenced this issue Dec 12, 2018
@brandon-wonjune
#1058 if UNIX timestamp, check change timestamp field
17696fd
@brandon-wonjune brandon-wonjune mentioned this issue Dec 12, 2018
Merged
9 tasks
brandon-wonjune added a commit that referenced this issue Dec 13, 2018
@brandon-wonjune
#1058 if UNIX timestamp, fix query granularity error
3dbb4bd
brandon-wonjune added a commit that referenced this issue Dec 14, 2018
@brandon-wonjune
#1058 add remove changed timestamp logic, fix segment granularity list
17636f4
brandon-wonjune added a commit that referenced this issue Dec 18, 2018
@brandon-wonjune
#1058 if not time format, call recursive method in automation granula…
9cebad3 
…rity setting
brandon-wonjune added a commit that referenced this issue Dec 18, 2018
@brandon-wonjune
#1058 change automation granularity param
11ba64d
brandon-wonjune added a commit that referenced this issue Dec 18, 2018
@brandon-wonjune
#1058 change automation granularity slice format logic
0c38dc1
brandon-wonjune added a commit that referenced this issue Dec 18, 2018
@brandon-wonjune
#1058 set default granularity in automation logic
b1c4902
alchan-lee pushed a commit that referenced this issue Dec 19, 2018
@brandon-wonjune @alchan-lee
#1058 Granularity automation in ingestion (#1084)
53f291e 
* #1058 add granularity automation logic

* #1058 fix change schema logic

* #1058 add changed timestamp field flag in schemaData

* #1058 if UNIX timestamp, check change timestamp field

* #1058 if UNIX timestamp, fix query granularity error

* #1058 add remove changed timestamp logic, fix segment granularity list

* #1058 if not time format, call recursive method in automation granularity setting

* #1058 change automation granularity param

* #1058 change automation granularity slice format logic

* #1058 set default granularity in automation logic
brandon-wonjune added a commit that referenced this issue Dec 19, 2018
@brandon-wonjune
#1058 change default granularity used current_date timestamp, remove …
9541b69 
…SECOND, MINUTE in segment granularity
@brandon-wonjune brandon-wonjune mentioned this issue Dec 19, 2018
Merged
9 tasks
alchan-lee pushed a commit that referenced this issue Dec 19, 2018
@brandon-wonjune @alchan-lee
#1058 remove SECOND, MINUTE in segment granularity (#1125)
d073730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eltriny eltriny

@brandon-wonjune brandon-wonjune

Labels
@datasource Component Name : Datasource/Connection Management enhancement Request Change and Feature Enhancement
Projects
None yet
Milestone
3.2.0
Development

No branches or pull requests
4 participants
@kyungtaak @eltriny @brandon-wonjune @alchan-lee