Feature request: unbuffered I/O #2146

Closed
abudden opened this Issue Feb 16, 2015 · 15 comments

Projects

None yet

4 participants

@abudden
Contributor
abudden commented Feb 16, 2015

Python offers a command-line argument -u that offers an unbuffered version of stdio. This is extremely useful in some circumstances. Would it be possible to add this feature to Nim such that the compiled executables use unbuffered stdio?

@dom96 dom96 added the Feature label Feb 16, 2015
@Araq Araq added the Stdlib label Feb 18, 2015
@abudden
Contributor
abudden commented Jun 7, 2016

Having looked into this a bit further, the implementation is relatively simple (lifted wholesale from python's C source code):

#ifdef HAVE_SETVBUF
        setvbuf(stdin,  (char *)NULL, _IONBF, BUFSIZ);
        setvbuf(stdout, (char *)NULL, _IONBF, BUFSIZ);
        setvbuf(stderr, (char *)NULL, _IONBF, BUFSIZ);
#else /* !HAVE_SETVBUF */
        setbuf(stdin,  (char *)NULL);
        setbuf(stdout, (char *)NULL);
        setbuf(stderr, (char *)NULL);
#endif /* !HAVE_SETVBUF */

Thinking about this further, it probably wouldn't be the end of the world if this were a function that could be called from the source code of the program rather than as a predefined compiler option. This would at least have the advantage that I wouldn't have to work out where to put it if I did a pull request! The application code could then be written as something like:

when defined(unbuffered_io):
    setIOUnbuffered()

This could then be a compiler option -d:unbuffered_io (with maybe a recommended standard name for the define).

Which is preferred out of the following?

  1. A standard lib function with a name like setIOUnbuffered that runs the above C code in C or does nothing in javascript (in the system module or somewhere else?)
  2. A compiler option that inserts the above C code somewhere into the initialisation process of the generated application (and therefore is closer in terms of user experience to python's implementation).

If 1, I'm happy to do a PR as long as someone can confirm whether the name is okay and in which module it should reside. If 2, I'm happy to do a PR as long as someone can help me find the right place to add the code and can suggest a suitable argument name (as -u is taken), maybe --unbuffered?

Oh, I might also need some help figuring out what to do about HAVE_SETVBUF from the C code above.

@abudden
Contributor
abudden commented Jun 7, 2016

Oh, I probably should have included a reference for the python code: https://hg.python.org/cpython/file/f477ce0d2345/Modules/main.c#l602

@Araq
Member
Araq commented Jun 7, 2016

Nim is not Python. Command line arguments that affect every Nim program secretly (because it's processed in the stdlib) are not acceptable.

@abudden
Contributor
abudden commented Jun 7, 2016

Nim is not Python. Command line arguments that affect every Nim program secretly (because it's processed in the stdlib) are not acceptable.

Okay, but I don't think that precludes option 1 or am I missing something?

@Araq
Member
Araq commented Jun 7, 2016
@abudden
Contributor
abudden commented Jun 7, 2016 edited

If that works, that would be great; I had assumed that stdout was already open (see C example above). What would I pass in to open to make it re-open stdout and make echo unbuffered? It looks like it is expecting a filename which doesn't apply in this case.

@abudden
Contributor
abudden commented Jun 7, 2016

What would I pass in to open to make it re-open stdout and make echo unbuffered?

Actually, that was a bad example given that echo automatically does flushFile(stdout). A better example would have been using write(stdout, "Some text") or similar.

@Araq
Member
Araq commented Jun 7, 2016

system.reopen exists for that. But it needs to grow a bufSize parameter.

@abudden
Contributor
abudden commented Jun 8, 2016 edited

system.reopen exists for that. But it needs to grow a bufSize parameter.

That looks like it should be fairly easy to implement (the bufSize parameter in open just calls setvbuf in the same way as the python implementation). However, I've been playing around with a modified version of reopen and I can't see how I can use it as it seems to require a filename (which isn't applicable if you just want to change stdout to be unbuffered).

@abudden
Contributor
abudden commented Jun 8, 2016

Sorry - somehow hit "Comment" too early. I'll edit my post.

@abudden
Contributor
abudden commented Jun 8, 2016 edited

For what it's worth, I've just tested a simple setStdIoUnbuffered function as follows and it works beautifully:

proc setStdIoUnbuffered*() =
  discard setvbuf(stdout, nil, IONBF, 0)
  discard setvbuf(stderr, nil, IONBF, 0)
  discard setvbuf(stdin, nil, IONBF, 0)

I could, of course, include this in the project code, but then rather than typing:

when defined(unbuffered_io):
  setStdIoUnbuffered()

I'd have to type:

when defined(unbuffered_io):
  when NoFakeVars:
    when defined(windows):
      const
        IOFBF = cint(0)
        IONBF = cint(4)
    else:
      # On all systems I could find, including Linux, Mac OS X, and the BSDs
      const
        IOFBF = cint(0)
        IONBF = cint(2)
  else:
    var
      IOFBF {.importc: "_IOFBF", nodecl.}: cint
      IONBF {.importc: "_IONBF", nodecl.}: cint

  proc setvbuf(stream: File, buf: pointer, typ, size: cint): cint {.
    importc, header: "<stdio.h>", tags: [].}

  proc setStdIoUnbuffered*() =
    discard setvbuf(stdout, nil, IONBF, 0)
    discard setvbuf(stderr, nil, IONBF, 0)
    discard setvbuf(stdin, nil, IONBF, 0)

  setStdIoUnbuffered()

So obviously I'd much rather have this functionality in system.nim (where most of the above already exists). I guess it wouldn't be so horrible (just a little horrible) to have a setBufSize(stream: File, bufSize: int = -1) function so that I have to type:

when defined(unbuffered_io):
  setBufSize(stdout, 0)
  setBufSize(stderr, 0)
  setBufSize(stdin, 0)

but a single function would be much cleaner and could be easily documented as being the solution to those wanting to replicate python's -u option.

@Varriount
Contributor

Do people actually use Python's -u though? Most functions in Python's stdlib have a buffered parameter.

@abudden
Contributor
abudden commented Jun 8, 2016

Do people actually use Python's -u though? Most functions in Python's stdlib have a buffered parameter.

I certainly do, and I had no involvement with the implementation of the feature, so I can only assume that others do too.

@Araq
Member
Araq commented Jun 9, 2016

Feel free to put your setStdIoUnbuffered in lib/system/sysio.nim and create a PR.

@abudden
Contributor
abudden commented Jun 9, 2016

Okay, will do. Just to confirm: should the news entry go in 2016_06_08_version_0_14_2_released.rst?

@Araq Araq closed this in 52b4e8f Jun 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment