Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify preview code in dataprep (including auto type detection) #1224

Closed
joohokim1 opened this issue Jan 9, 2019 · 4 comments
Closed

Unify preview code in dataprep (including auto type detection) #1224

joohokim1 opened this issue Jan 9, 2019 · 4 comments
Assignees
Labels
@dataprep Component Name : Data preparation enhancement Request Change and Feature Enhancement
Milestone

Comments

@joohokim1
Copy link
Contributor

Is your feature request related to a problem? Please describe.
There are several codes showing the preview of datasets or to-upload files.
Because the codes had been developed independently, even if you fixed a bug on one of the codes, you cannot assure that it will not happen in another code.

현재 dataset이나 업로드 하려는 파일의 preview를 보여주는 코드가 여러군데에 난개발되어있습니다.
그래서 여긴 되고, 저긴 안되고, 또는 여기 저기서 모은 것을 짜깁기 하다가 영 엉뚱한 결과가 나오기도 합니다.

2019-01-09 11 10 20
2019-01-09 11 10 35
2019-01-09 11 10 43
2019-01-09 11 10 52

Describe the solution you'd like
We will make preview code as universally as possible so that all preview pages of dataprep will use one code. First I will apply the code to all dataprep previews, then the code might be applied to other parts that needs the preview of files. (ex. datasource creation). Functions such as auto type detection can be useful in other parts, and when considering the cost of redundant development and the difficulties of maintenance, code reusing is very preferable.

최대한 범용적으로 preview 코드를 만들어서 dataprep의 모든 preview가 한 코드를 사용하도록 할 예정입니다.
일단 dataprep의 모든 preview를 통일해서 사용해보고, file에 대한 preview가 필요한 다른 부분도 (예를 들면 datasource 생성) 이 코드를 사용하는 것을 검토할 예정입니다. Auto type detection과 같은 기능은 다른 파트에서도 유용하게 쓰일 수 있고, 중복 개발 비용 및 유지 보수의 어려움 등을 생각할 때에 최대한 많은 부분이 같은 코드를 기반하는 것이 좋을 것으로 보고있습니다.

Basically, I'm going to make all previews to show the same result as those shown in dataprep's main transform page. It could be adjusted or over-riden by the requirement of the usage, of course.

기본적으로는 모든 preview는 dataprep의 main transform 화면에 나오는 결과와 같도록 하고자 합니다. 당연한 얘기지만, 적용되는 조건에 따라 조정되거나, 재구현될 수 있습니다.

Additional context
I have discussed with @AnnieHwang, @kyungtaak, and @alchan-lee about spreading the use to other parts.

다른 파트에서 확대 사용하는 것에 대해서는 @AnnieHwang 님과 @kyungtaak 님, @alchan-lee 님과 미리 상의하였습니다.

@joohokim1
Copy link
Contributor Author

World Cup Matches.txt

joohokim1 added a commit that referenced this issue Jan 24, 2019
…ong, Double when read previews from serialized files
@joohokim1
Copy link
Contributor Author

@sang-hee-lee @paigechoi @AnnieHwang @kyungtaak @alchan-lee

For data preparation, the server-side code related to the all kinds of previews has been unified.
However, a bit of wrapping code has been applied on upload preview for compatibility with UI code.
It's time to modify both the UI code and server-side code related to the upload preview according to the change.

Data preparation의 preview 관련한 서버쪽 코드는 모두 통일되었습니다.
단, upload preview에 대해서는 UI code와의 compatability를 위한 wrapping을 해두었습니다.
이제 upload preview를 처리하는 UI code와 관련해서 wrapping되어있는 서버 코드를 함께 고치면 될 것 같습니다.

@joohokim1
Copy link
Contributor Author

@sang-hee-lee @paigechoi

It seems to be confused with what I described previously, so I have changed the server's response in the desired direction. (Ie, the preview does not work without the UI code's change). For Excel, use sheetNames [] and gridResponses []. For CVS and JSON, use gridResponses [0].

전화로 설명드린 내용에 혼동이 있는 것 같아서, 일단 제가 바람직한 방향으로 서버의 response를 바꿔둔 상태입니다. (즉, 현재 UI의 코드 수정 없이는 upload preview가 동작하지 않는 상태입니다) Excel의 경우 sheetNames[], gridResponses[]를 써주시고, CVS, JSON은 gridResponses[0]을 쓰시면 됩니다.

sang-hee-lee added a commit that referenced this issue Jan 30, 2019
sang-hee-lee added a commit that referenced this issue Jan 30, 2019
joohokim1 added a commit that referenced this issue Jan 30, 2019
…ong, Double when read previews from serialized files
joohokim1 added a commit that referenced this issue Jan 30, 2019
joohokim1 pushed a commit that referenced this issue Jan 30, 2019
joohokim1 added a commit that referenced this issue Jan 30, 2019
* #1224 put preview after auto type conversion

* #1224 refactoring file import code for auto typing

* #1224 applied auto detection and conversion on CSV upload preview

* #1224 applied auto detection and conversion on JSON upload preview

* #1224 applied auto typing for I.DS. convert all Integer, Float into Long, Double when read previews from serialized files

* #1224 null processing upon upload preview

* #1224 applied auto detection and conversion on EXCEL upload preview. changed into gridResponses

* #1224 changed CSV, JSON's upload previews into gridResponses (instead of old grids)

* #1224 modify uri preview

* #1224 resolve conflict while merging from master
@joohokim1
Copy link
Contributor Author

1st integration test checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@dataprep Component Name : Data preparation enhancement Request Change and Feature Enhancement
Projects
None yet
Development

No branches or pull requests

2 participants