Skip to content

Schemes Notes

Jump to bottom
rgchris edited this page Jan 7, 2017 · 3 revisions

Schemes > Notes

Ports access external series such as files, networks, consoles, events, databases, data encoders, and data decoders.

In Rebol 3.0, the port system has be completely redesigned to make it simpler, faster, and easier to use. Substantial effort has been made to remove the majority of the complex internal mechanisms and replace it with a smart streamlined system.

Table of Contents

Architecture

The diagram below shows the basic relationship between schemes, ports, requests, and devices:

Definitions:

Scheme
specifies a type of port, such as file, TCP, event, sound, etc. The scheme creates a port, which is:
Port
a specific instance of a scheme. For example, when you read a file, the scheme is used to create a port that accesses the file and its data. Each open file has its own port.
Request
is a lower level (C) structure used to communicate between a port and a device. In the case of a file, it passes fields like the file name, date, size, and data to be transferred.
Device
is native C code that implements a standard set of access methods for a port. Example methods include open, close, read, write, create, and delete. This level is where file or network I/O actually happens.
Only ports that require native support need requests and ports. Higher level ports, such as HTTP, do not directly create requests. Typically, they use lower level ports, such a TCP, and those do use requests.

The primary benefit of this design is that it formalizes a method for accessing a wide range of devices without expanding the Rebol-to-environment API for each device. In a way, it implements a mini operating system to keep Rebol as system-independent as possible.

Schemes

A scheme specifies a type of port.

The name scheme comes from the description of a URI (URL): 

http://host/path

Here HTTP is the scheme name in the URL.

Built-in schemes

A variety of schemes are built-into Rebol 3 and can be used immediately on boot.

Here is a partial list of built-in schemes:

Scheme Description
system System port (primary event dispatcher)
stdio Standard input and output
stderr Standard error
console Rebol interactive console
file Local file access
dir Local file directory access
event GUI event handling
DNS Domain name lookup
TCP Transfer control protocol
UDP User datagram protocol
HTTP Hypertext transfer protocol
shell External shell commands (call)
serial Serial port access
USB USB serial port access
mutex Mutual exclusion locks

More (such as FTP, SMTP, POP) will be included as time and user-base contributions permit.

To list the current schemes: 

print words-of system/schemes

or: 

foreach val get system/schemes [print [val/name val/title]]

Scheme object structure

The scheme object is defined in system/standard as: 

scheme: context [
    name:       ; word of http, ftp, sound, etc.
    title:      ; user-friendly title for the scheme
    spec:       ; custom spec for scheme (if needed)
    info:       ; prototype info object returned from query
    actor:      ; standard action handler for scheme port functions
    awake:      ; standard awake handler for this scheme's ports
        none
]

Where:

name
is the type of scheme. This is the word used in the first part of the URL, such as TCP, HTTP, FTP, etc.. It must be a valid Rebol word (e.g. cannot begin with a number).
title
is the title of the scheme for user identification and documentation purposes.
spec
an object used to specify additional features of the scheme.
info
an prototype object that is instantiated for a query on the scheme.
actor
the handler function that processes standard actions a port of this scheme type. For example, open, close, read, write, etc.
awake
a callback function that is invoked when new events arrive for a port of this scheme type.

Adding a new scheme

The make-scheme function creates a new scheme from a block object specification.

(show example)

Ports

A port is an active instance of a scheme.

For example, when you open a file, the system creates a port from the file scheme. The port keeps track of the I/O for that specific file, as well as specific information about the file, such as date, size, and current index position.

The Rebol 3 design for ports is more streamlined than the prior version - providing faster access and with less memory.

Port object structure

The standard port object has been substantially reduced in size and complexity from Rebol 2. The new object is defined by system/standard as: 

port: context [
    spec:       ; published specification of the port
    scheme:     ; scheme object used for this port
    actor:      ; port action handler (script driven)
    awake:      ; port awake function (event driven)
    state:      ; internal state values (private)
    data:       ; data buffer (usually binary or block)
        none
]

Where:

spec
is the object used to define the port. It holds the scheme name and other fields, such as the host for network ports. The format of the spec object depends on the scheme of the port.
scheme
points back to the scheme object for this port. It is normally used by lower layers that may need to check scheme related values.
actor
a function that is called for handling port actions. Normally, it is the same as that of the scheme, but it can be unique to a specific port. This function can be a native or mezzanine.
awake
a callback function called when new events arrive for the port. This is normally the same as that of the scheme, but but it can be unique to a specific port.
state
private state data. This is an object or a C-structure in the case of native port schemes. Scripts should not access the state! The format and definition of the state is allowed to change between versions.
data
a buffer that is used for the most common types of ports (those that transfer data).
This object is converted to a PORT datatype during the port's initialization sequence.

Specific types of ports (schemes) are allowed to extend this structure for their own purposes, but developers should avoid unnecessary fields (bloat).

Port specification object

Port actor functions

Port awake functions

Make note about return value. A TRUE will satisfy a WAIT, but other values do not.

Event processing

notes about system port as primary dispatcher

Devices and Requests

covered in separate document

...Other Notes...

FAQ

What is a scheme?

A scheme is a set of definitions and functions (an object!) that handle the operations for a specific kind of port. Ports are made from scheme objects.

Schemes may be written as an interface to just about anything, but are most often used for stream-like interfaces, such as network protocols.

Since they are objects, they can store local data to maintain state.

Actions are used to access ports created from schemes. For example, when you call open or read on a url!, the url is passed to make-port, which looks at the scheme name (the part before ://), creates a port! using the matching scheme definition, and calls the actor func defined in that scheme that matches the original action (open, read, etc.).

How do I create a scheme?

A scheme is defined by calling the make-scheme function. There are a number of basic values you must set for a scheme to function correctly, but you can also add your own words to the scheme objects you create, just as you do with any Rebol object.

The spec you pass to make-scheme also contains actor functions that are mapped to port action calls.

How do I create a new scheme?

How do I modify an existing scheme?

?? What do you use as the prototype?

What are the important elements of a scheme?

A scheme doesn't have many fields, but they're all important. All schemes MUST have a name, they SHOULD have a title, most will use the spec and info fields. If you need to respond to events, as in the case of networking schemes that use TCP sub-ports, you will need an awake function; and without actor functions, your scheme won't be able to do much.

There are a large number of actor functions that you MAY include, but your scheme may not need to support them tall. For example, you may choose not to support all the series actions, like insert, append, etc. At the very least, you should include the following actors:

  • close
  • open
  • open?
  • read
  • write
And possibly also these:
  • copy
  • create
  • delete
  • length?
  • query
  • rename
  • update

What is the general architecture of a scheme?

What are the standard scheme-related values in Rebol? 

make-scheme

system/standard/port

system/standard/scheme

system/intrinsic/make-port

How does Rebol create ports from schemes?

The creation process is something like this. Given this line: 

p: open http://www.rebol.com

This is what happens:

open calls make-port with a url! arg

make-port does the following

  • Decodes the url and creates a port object from the scheme
  • Sets up port with scheme info
  • Calls scheme/init
  • Calls scheme/actor/open port
  • Returns the port
You can also specify a scheme block directly. e.g.: 
p: open [
    scheme: 'http 
    host: "www.rebol.com" 
    path: none 
    ref: http://www.rebol.com
]

Getting started

Before you write your first scheme, you should be familiar with the various data structures and functions the system uses when operating on ports. Remember that a scheme defines how a port works. What makes this a little tricky is that your scheme can change the structures of the port, adding words for state, configuration, and data storage; the system calls into your scheme definition when creating the port, and then also passes the port to actor functions.

It's important to keep the separation of scheme and port clear in your mind. A scheme is shared by all ports that use that scheme, so you shouldn't store data that is specific to a port, like the port's state in your scheme's info word for example. A scheme's actor object is shared, but the port spec defined in the scheme is a prototype used to create the spec value for the port, so that can be modified for each port. The awake function for a port will refer to the awake function in the port spec if that value exists, or to the awake function defined in the scheme if not.

See system/intrinsic/make-port to see exactly how this all works.

state / data or your own. You can add your own fields to the port with bind/new in the scheme's init function; it's a bad idea to add them to system/standard/port, since that would likely conflict with other port schemes.

In general, port/state should be the internal state vars for the scheme that users are never supposed to touch directly.

Reference

Objects / Data structures

system/standard/port 

make object! [
    spec: none
    scheme: none
    actor: none
    awake: none
    state: none
    data: none
    locals: none
]

system/standard/port-spec-head 

make object! [
    title: none
    scheme: none
    ref: none
]

system/standard/port-spec-net 

make object! [
    title: none
    scheme: none
    ref: none
    host: none
    port-id: 80
]

system/standard/scheme 

make object! [
    name:  none     ; [word!] The URL scheme name. e.g. 'http for an http:// scheme
    title: none     ; [string!] Human-friendly string; identifies the scheme in lists, errors, etc.
    spec:  none     ; [block! object!] Specification of arguments and options
    info:  none     ; [object!] The object returned from the QUERY action
    actor: none     ; [block! object!] Port action handlers - action trampoline target functions
    awake: none     ; [function!] Wake-up event function handler (optional)
        ; awake: func [event] [...]
        ; Return true to exit from wait
        ; Return false to continue waiting
]
Name
Is used as the leading part of the scheme. That is, it's the part
that comes before :// in a url. For example, if the name is 'http, you would access the scheme like *http://www.rebol.com*.
Title
Is the human-friendly string that identifies the scheme in lists,
errors, etc. It is also put by default in port/spec/title
Spec
The spec used to make the actual port spec in system/intrinsics/make-port.
You can omit the spec, in which case either the spec used will be the spec for the scheme name (system/schemes/:name/spec) or system/standard/port-spec-head. Spec can be a prototype object, or a spec block; e.g.: 
spec: make system/standard/port-spec-net [
    path: %/ 
    timeout: 15
]
Info
If a user calls QUERY on the port, this is what they'll get back.
If your scheme is a file-based scheme, e.g. HTTP, it may use system/standard/file-info, but you can return any value you want. 
info: system/standard/file-info 

info: context [ ; the object returned from the QUERY action
    data:
    len: none
]
Actor
...
Awake
...
Init
Is not in the standard scheme prototype, but can be specified.
Init is outside of actor and is called by make-port (see system/intrinsic/make-port). If specified, it must be a func that takes a single argument, the port being initialized. 
init: func [port] [ ; called for MAKE on PORT (also for direct actions)
    print ["Initializing port:" port/spec/ref "with" port/spec/title]
]
Title
not sure it is used at this point. i guess for reflectivity (ie. show port/scheme/title and port/spec/title to user)

Actors

When Rebol evaluates an action, and the next value is a port, that action is dispatched to an actor func for the port. For example 

read http://www.rebol.com

would call the READ actor func for the port opened using the HTTP scheme. If you wrote the HTTP scheme, it would be the READ func defined in the actor block of your HTTP scheme that is called. That means your actor funcs must be compatible with the associated actions that will call them.

series actions: insert, etc.

create, delete, open, close, read, write, open?, query, update, rename

Atomic actors

Quite often, ports are used to make easy calls like: 

read http://www.rebol.com

In this case, the user doesn't have to explicitly open and close the port, which is nice for them; but what does it mean for scheme writers?

Rebol will have created the port before calling the actor, so you can't check to see if the port arg is none. One common approach is to set the port state in the open actor, check that the port state is set in the open? actor, and then simply use open? in the read and write actors. For example: 

open: func [port] [
    port/state: context [
        ; more 'state values you need go here
    ]
    port
]

open?: func [port] [
    not none? port/state
]

close: func [port] [
    port/state: none
    port
]

write: func [port data] [
    if not open? port [port: open port]
    ; append port/state/data data ; or something similar
]

read: func [port /part len] [
    if not open? port [port: open port]
    if none? len [len: length? port/state/data]
    ; copy/part port/state/data len ; or something similar
]

Note that read and write do not close the port object, even if they opened it. The reason is that calls like 

read http://www.rebol.com

don't hold a reference to the port, so it will be garbage collected, which will automatically close it. Of course, if you prefer, you can add the extra code to track whether you opened the port and close it before exiting.

Port Actions

Close

Should return the port.

Create

Delete

Open

Should return the port.

Open?

Should return a logic! value.

Query

Returns the schemes info.

Read

Should return the data read from the port.

Rename

Update

Write

Common return values are the length of the data written or a logic value indicating success or failure.

Series Actions

append

at

back

change

clear

copy

empty?

find

head

head?

index?

insert

length?

modify

next

past?

pick

poke

remove

rm

select

skip

tail

tail?

Functions

  • make-port
  • make-scheme
  • set-scheme
>> ?? make-scheme
make-scheme: make function! [[
    {Make a scheme from a specification and add it to the system.}
    def [block!] "Scheme specification"
    /with 'scheme "Scheme name to use as base"
    /local actor
][
    ; 'with becomes our prototype scheme object
    with: either with [get in system/schemes scheme] [system/standard/scheme]
    ; Without a prototype, we can't go any further
    if not with [cause-error 'access 'no-scheme scheme]
    ; 'def becomes our scheme object, made from the prototype and the block passed in.
    def: make with def
    ; Every scheme MUST have a name
    if not def/name [cause-error 'access 'no-scheme-name def]

    ;!! Carl to confirm !!
    ;?? Register the scheme with the system, takes the place of R2 'net-utils/net-install call
    set-scheme def

    ; If 'actor is a block, make an object from it.    
    if block? :def/actor [
        ; The actor block must contain valid func specs.
        ; Allocate enough space in the object to avoid reallocation; the number
        ; of words should match the number of actor funcs. 
        ; It expects "name: func [args] [body]" format for func defs. Any other
        ; format will not work.
        actor: make object! (length? def/actor) / 4
        foreach [name func* args body] def/actor [
            name: to-word name
            repend actor [name func args body]
        ]
        ; Set the actor in our scheme object to reference the actor object
        ; we just created.
        def/actor: actor
    ]
    
    ; Add our scheme object to the list of schemes the system knows about.
    ;?? How does this relate to set-scheme?
    append system/schemes reduce [def/name def]
]]
>> probe get in system/intrinsic 'make-port
make function! [[
    spec [file! url! block! object! word! port!] "port specification"
    /local name scheme port
][
    case [
        file? spec [
            name: pick [dir file] dir?/any spec
            spec: join [ref:] spec
        ]
        url? spec [
            ; make url! into spec block: 
            ;   http://www.rebol.com
            ; becomes:
            ;   [scheme: 'http host: "www.rebol.com" path: none ref: http://www.rebol.com]
            spec: repend decode-url spec [to-set-word 'ref spec]
            name: select spec to-set-word 'scheme
        ]
        block? spec [
            name: select spec to-set-word 'scheme
        ]
        object? spec [
            name: get in spec 'scheme
        ]
        word? spec [
            name: spec
            spec: []
        ]
        port? spec [
            name: port/scheme/name
            spec: port/spec
        ]
        true [
            return none
        ]
    ]
    if not all [
        word? name
        scheme: get in system/schemes name
    ] [cause-error 'access 'no-scheme name]
    port: make system/standard/port []
    port/spec: make any [scheme/spec system/standard/port-spec-head] spec
    port/spec/scheme: name
    port/scheme: scheme
    port/actor: get in scheme 'actor
    port/awake: any [get in port/spec 'awake :scheme/awake]
    if not port/spec/ref [port/spec/ref: spec]
    if not port/spec/title [port/spec/title: scheme/title]
    port: to port! port
    if in scheme 'init [scheme/init port]
    port
]]
>> probe p: system/intrinsic/make-port 'tcp
make port! [
    spec: make object! [
        title: "TCP Networking"
        scheme: 'tcp
        ref: []
        host: none
        port-id: 80
    ]
    scheme: make object! [
        name: 'tcp
        title: "TCP Networking"
        spec: make object! [
            title: none
            scheme: none
            ref: none
            host: none
            port-id: 80
        ]
        info: make object! [
            local-ip: none
            local-port: none
            remote-ip: none
            remote-port: none
        ]
        actor: make native! [[]]
        awake: make function! [[event][print ['TCP-event event/type] true]]
    ]
    actor: make native! [[]]
    awake: make function! [[event][print ['TCP-event event/type] true]]
    state: none
    data: none
    locals: none
]

Schemes

  • TCP
  • UDP
  • HTTP
  • HTTPS/SSL
  • FTP
  • TFTP
  • SFTP
  • POP3
  • IMAP
  • SMTP
  • ESMTP
  • LDAP
  • WEBDAV
  • CAL
  • DNS
  • Echo
  • Finger
  • Whois, nickname
  • Gopher
  • Daytime
  • NTP
  • NNTP
  • nameserv, WINS
  • SGMP
  • SNMP
  • IRC
  • Telnet
  • SSH
  • APP
  • PPTP
  • MySQL
  • ICQ
  • Jabber
  • AIM
  • Yahoo! Messenger
  • SVN

File schemes

  • zip, rar, etc.
  • tar
  • png
  • mjpg
  • wmv
  • mpg
Clone this wiki locally