New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use an external library for reading files #2824

Open
karlnapf opened this Issue Apr 18, 2015 · 10 comments

Comments

Projects
None yet
4 participants
@karlnapf
Member

karlnapf commented Apr 18, 2015

It is ridiculous that we are doing this ourselves. Also it is slow and buggy. I suggest to just remove the file reader code we have. CSV doesnt even work for reasonably sized files.

This is a list of external libs that we could use instead:

@lisitsyn

This comment has been minimized.

Show comment
Hide comment
@lisitsyn

lisitsyn May 16, 2015

Member

Header only library is mostly the same thing as supporting our own solution as we'd have to put the header(s) into our repo. We have to rely on something that is already distributed in major systems we want to support

Member

lisitsyn commented May 16, 2015

Header only library is mostly the same thing as supporting our own solution as we'd have to put the header(s) into our repo. We have to rely on something that is already distributed in major systems we want to support

@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Jun 12, 2015

Member

BUT we can bundle it like eigen3

Dont know. We just should get rid of all this hand-crafted code for that. It is

  • slow
  • unreliable
  • buggy
  • hard to maintain
  • impossible to read
  • blow up compile time
Member

karlnapf commented Jun 12, 2015

BUT we can bundle it like eigen3

Dont know. We just should get rid of all this hand-crafted code for that. It is

  • slow
  • unreliable
  • buggy
  • hard to maintain
  • impossible to read
  • blow up compile time
@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Dec 14, 2015

Member

I did some more research and there does not seem to be any csv lib that is nice and bundled in many systems.....
I still tend to just use this one

Member

karlnapf commented Dec 14, 2015

I did some more research and there does not seem to be any csv lib that is nice and bundled in many systems.....
I still tend to just use this one

@besser82

This comment has been minimized.

Show comment
Hide comment
@besser82

besser82 Dec 14, 2015

Member

How about libcsv? It's light-weight, straight forward to integrate, available on most distributions and LGPL'ed code.

Member

besser82 commented Dec 14, 2015

How about libcsv? It's light-weight, straight forward to integrate, available on most distributions and LGPL'ed code.

@vigsterkr

This comment has been minimized.

Show comment
Hide comment
@vigsterkr

vigsterkr Dec 14, 2015

Member

first things first... let's start with adding support of libarchive and then clean up the mess with not being able to parse more than 15k lines of csv :hurtrealbad:

Member

vigsterkr commented Dec 14, 2015

first things first... let's start with adding support of libarchive and then clean up the mess with not being able to parse more than 15k lines of csv :hurtrealbad:

@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Dec 14, 2015

Member

@vigsterkr I am happy to use any library. But I just want to

  • get rid of our own file parsing code (for the above reasons)
  • be able to read files bigger than 15k

What do you suggest to do as next steps here?

Member

karlnapf commented Dec 14, 2015

@vigsterkr I am happy to use any library. But I just want to

  • get rid of our own file parsing code (for the above reasons)
  • be able to read files bigger than 15k

What do you suggest to do as next steps here?

@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Dec 14, 2015

Member

@besser82 otherwise libcsv also sounds good.

Member

karlnapf commented Dec 14, 2015

@besser82 otherwise libcsv also sounds good.

@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Dec 14, 2015

Member

@vigsterkr I just checked libvarchive. I don't get why you want to do that first? Why care about archives when we cannot deal with single files? Can you elaborate a bit?

Member

karlnapf commented Dec 14, 2015

@vigsterkr I just checked libvarchive. I don't get why you want to do that first? Why care about archives when we cannot deal with single files? Can you elaborate a bit?

@besser82

This comment has been minimized.

Show comment
Hide comment
@besser82

besser82 Dec 15, 2015

Member

@karlnapf I think libarchive affects or improves writing / reading all IO-formats in Shogun. For that reason @vigsterkr wants to get that done first…

Member

besser82 commented Dec 15, 2015

@karlnapf I think libarchive affects or improves writing / reading all IO-formats in Shogun. For that reason @vigsterkr wants to get that done first…

@karlnapf

This comment has been minimized.

Show comment
Hide comment
@karlnapf

karlnapf Feb 8, 2017

Member

This is still open I guess?

Member

karlnapf commented Feb 8, 2017

This is still open I guess?

@vigsterkr vigsterkr modified the milestones: Shogun 5.1, Shogun 6.0 Mar 15, 2017

@vigsterkr vigsterkr modified the milestones: Shogun 6.1.0, Shogun 6.2.0 Dec 14, 2017

@vigsterkr vigsterkr modified the milestones: Shogun 6.2.0, Shogun 7.0.0 May 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment