scanf equivalent

[c42f]

It would be nice to have a [sf]scanf equivalent for quickly parsing large tabulated text files with simple but ad-hoc formatting. I wasn't sure where to post an issue about this, let me know if there's a better place.

A current example on my mind is ftp://ftp.ga.gov.au/geodesy-outgoing/gravity/ausgeoid/AUSGeoid09_V1.01/AUSGeoid09_GDA94_V1.01_DOV.zip

AUSGeoid09_GDA94_V1.01                          www.ga.gov.au
GEO   14.595 S 8  0  0.000 E108  0  0.000    -29.88    -26.83
GEO   14.833 S 8  0  0.000 E108  1  0.000    -36.93    -20.99
GEO   14.968 S 8  0  0.000 E108  2  0.000    -36.59    -15.21
... actual data isn't important, but there's 7 million rows of it.

I made a little prototype, including a string macro, with the aim of making the following work:

values = read(io, scanf"GEO %f %c %d %d %f %c%d %d %f %f %f")

Gist here: https://gist.github.com/c42f/9999dc6f9b63a9bd4ea4237a95876475

The prototype just ccalls sscanf directly, but I hit a few nasty things and thought I'd record the problems here in case anyone has feedback.

• It would be nice if this worked with IO objects for streaming. However in julia streams are either unbuffered or internally buffered, with no way to get at the buffer object (if it even exists) or put back a character. This makes efficient text parsing from a stream without overreading impossible. Would need something like https://github.com/BioJulia/BufferedStreams.jl instead of a plain IO (CC @bicycle1885 ) For the moment, I worked around the problem for this particular input file using readline() but that's far from ideal.
• Related to the above, IO objects cannot be used with fscanf since they have no internal FILE data structure at the C level (unsurprising).
• sscanf("%s", ...) seems to be completely memory unsafe, with no way to work around it without disallowing it, implementing this conversion specifier on the julia side or relying on the posix "%ms" extension.
• The format width specifiers ("%lf" for Float64? ugh.) are C-isms which are a little nasty to expose at the julia level, but make sense for compatibility.

[c42f]

@r-chris @awbsmith here's an issue regarding scanf. As you can see, there's several reasons why implementing this is a bit of a mess! It's quite doable though, I think.

[awbsmith]

Looks cool

On 8 August 2016 at 13:12, Chris Foster notifications@github.com wrote:

@r-chris https://github.com/r-chris @awbsmith
https://github.com/awbsmith here's an issue regarding scanf. As you can
see, there's several reasons why implementing this is a bit of a mess! It's
quite doable though, I think.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQAxNEkAJGKIXVGmOJrIme9Zw5z7w2aYks5qdp6YgaJpZM4Jeq6t
.

[kim366]

What's the progress on this? These functions would be really useful

[c42f]

I didn't need this enough to work further on it so the idea is languishing ;-)

[Ellipse0934]

I had a similar problem/issue. What I did was use scanf via ccall. Not optimal but works.

Here's a small example:

julia> x = Ref{Float32}(0)
Base.RefValue{Float32}(0.0f0)

julia> y = Ref{Float32}(0)
Base.RefValue{Float32}(0.0f0)

julia> ccall(:scanf, Int, (Ptr{UInt8}, Ref{Cfloat}, Ref{Cfloat}), "%f %f", x, y);
2.001 3.141

julia> x
Base.RefValue{Float32}(2.001f0)

julia> y
Base.RefValue{Float32}(3.141f0)

[mgiugliano]

I had a similar problem/issue. What I did was use scanf via ccall. Not optimal but works.

Here's a small example:

julia> x = Ref{Float32}(0)
Base.RefValue{Float32}(0.0f0)

julia> y = Ref{Float32}(0)
Base.RefValue{Float32}(0.0f0)

julia> ccall(:scanf, Int, (Ptr{UInt8}, Ref{Cfloat}, Ref{Cfloat}), "%f %f", x, y);
2.001 3.141

julia> x
Base.RefValue{Float32}(2.001f0)

julia> y
Base.RefValue{Float32}(3.141f0)

How do you pass the input string, in your example?
(thank you for your clarification)

[Ellipse0934]

By the input string do you mean 2.001 3.141 what I typed ? That I typed in the repl directly after calling ccall as that's going to be consumed from the stdin.

[mgiugliano]

I thought you used a (string) variable on which you applied the C call... This is something I am unfamiliar with and your more extensive example would have help me learning this.

I understand that you instead used the REPL...

My intention was actually to call sscanf, hence the source of my misunderstanding.

[KlausC]

Scanf is now a pure julia registered package

[c42f]

Looks great!