OS I/O: Raw Devices

[vyzo]

Cherry picked from #271.

Adds raw devices, currently implemented for POSIX only, so that special devices can be used to perform userland read/write operations while integrating with the Gambit scheduler.

[vyzo]

@feeley any progress with the generic port hierarchy?
I can do them if you don't have the time!

[vyzo]

I discovered a bug in the ___raw_device_select_virt -- it fails to handle for_writing correctly; fixing.

[feeley]

Halfway done...

[feeley]

Waitable ports are now implemented by commit 05cec0e.

[vyzo]

Excellent! I will merge and update the raw-device-ports to be descendants of waitable-ports.

By the way, how do you generate the type-ids for the structs? Is it a random uuid or something structured?
I used a hash to generate it for raw-device-ports.

[vyzo]

btw, ___PROCESS_DEVICE_KIND disagrees with (macro-process-kind).
I assume the two should be the same?

os_io.c:
#define ___PROCESS_DEVICE_KIND    (___PIPE_DEVICE_KIND+131072)

_io#.scm:
(##define-macro (macro-process-kind)     (+ 31 64))

[vyzo]

unless the process is just a pipe in userland, in which case it's ok:

os_io.c:
#define ___PIPE_DEVICE_KIND       (___DEVICE_KIND+64)

[vyzo]

Ok, merged! I'll do some casual testing and update #273 as well.

[vyzo]

seems like my implementation of wait is not correct - ##wait-device expects a device-port, so i'll have to call ##wait-for-io! directly.

[vyzo]

ok, verified that all port operations work as intended, this should be ready for merge!

[vyzo]

Updates for review comments in #273.

The only sticking point left is the utility of ##open-raw-device and its name.
For now it takes a device (not an fd :), opens a raw device and calls ##make-raw-device-port.

We can drop the function altogether if you think the caller should be doing the actual opening of the device (ie if you think it's too POSIX specific)

[vyzo]

Updated per review comments.

[vyzo]

small brainfart in usage of id field, fixing.

[feeley]

Please make a copy of ___device_file_select_raw_virt including the WIN32 part, because then it will probably work as-is on Windows.

[feeley]

Indent properly... open paren for parameters should be 2 spaces indented from the ___device_....

[vyzo]

do you have an emacs mode for your indentation style?

[feeley]

Nope.

[feeley]

Same indent style should be applied here.

[feeley]

I keep thinking that it is wrong to expose this function in Scheme since is is POSIX specific. It is OK however to make it available to C code, so that a device can be constructed from a fd using the FFI. The "index" parameter should be "fd" because that's what it is. The function should be called ___os_device_raw_open_from_fd.

[feeley]

See comment below. This functionality should not be exposed to the Scheme level.

[feeley]

Seems like ___device_raw_select_virt should be called ___device_raw_select_raw_virt to be consistent with the other device kinds.

[vyzo]

ok, will rename.

[vyzo]

ok, updated! Sorry for the indentation mismatch, i tend to just follow emacs in GNU style.

[vyzo]

actually there is an accident from the copy paste and it's a wrong cast in ___device_raw_select_raw_virt -- fixing.

[vyzo]

ok, now we should be good -- waiting for the build to finish to run some tests, but it looks good to me.

[vyzo]

testing successful.

[adam-7]

@vyzo , what's a raw device, what are their usecases? Can you provide a short example script that illustrates their utility here?

[vyzo]

Things like sockets, epoll, inotify and kquque file descriptors.
With raw devices we can open these OS-specific devices and integrate them with the Gambit scheduler.

[adam-7]

Ah!

By "raw device", do you just mean that you have created a facility whereby the user can allocate his own OS file descriptor of any type that the OS can produce,

(I have not studied epoll/inotify/kqueue so it's not in my awareness that they produce their own FD:s, but I understand that's what you're saying, so they fit the picture too;)

and, what you have done here is to provide a facility where the user can create a Gambit port object that functions only to represent such a file descriptor, so that the user may do |##device-port-wait-for-input!| and |##device-port-wait-for-output!| on that port object?

I guess such a raw port object is actually totally passive, as in, Gambit will not do any operations on it such as OS write, read, close operations, but they function only to make Gambit include that FD in Gambit's select/poll() call, during the ##device-port-wait-for-input/output! call?

Are any other operations than ##device-port-wait-for-input/output! supported on those raw ports?

Are the raw ports documented in the manual now or do you keep these out of the manual due to being too low level?

Probably the raw ports will be used for pretty low level stuff, right?

Except for this and making Gambit's epoll() support unlimited number of ports (whereas previously there was an ~8000 FD:s limit and exceeding that led to SIGSEGV), have you done any other major addition now?

Looks great! :) Thanks

[feeley]

This is just a heads up that the API for ##make-raw-device-port will be changing to accommodate other types of raw-devices. In particular what used to be called the "id" is now called the "type", and the "name" of the port is an explicit parameter (so it is easy to choose a custom name). The last field is a "specific" field that can be used to store anything.

Vyzo, here is how the old API can be emulated:

(define (##vyzos-make-raw-device-port raw-device device id direction)
  (##make-raw-device-port direction raw-device id (##list id device) device))

To access the "device" from the port, use (macro-raw-device-port-specific port).

[adam-7]

Like what types?

[vyzo]

thanks marc!

[feeley]

I'm implementing UDP ports and it is convenient to build them on top of raw-device ports. By the way, the API has changed again to: (##make-raw-device-port direction device kind id specific). The third parameter is a numeric "kind", for example (macro-raw-device-kind). This simplifies type testing.

[vyzo]

Can the kind be an arbitrary object (eg a symbol) and not just a numeric kind in the new API?
That would be much better for identifying user port types.