R-data.texi

\input texinfo
@c %**start of header
@setfilename R-data.info
@settitle R Data Import/Export
@setchapternewpage on
@c %**end of header

@syncodeindex fn vr

@dircategory Programming
@direntry
* R Data: (R-data).      R Data Import/Export.
@end direntry

@finalout

@include R-defs.texi
@include version.texi

@ifinfo
This is a guide to importing and exporting data to and from R.

@Rcopyright{2000}

@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).

@end ignore

@permission{}
@c ---------- ^- read that
@end ifinfo

@titlepage
@title R Data Import/Export
@subtitle Version @value{VERSION}
@author R Development Core Team
@page
@vskip 0pt plus 1filll
@permission{}

@Rcopyright{2000}
@end titlepage

@ifnothtml
@contents
@end ifnothtml

@ifnottex
@node Top, Acknowledgements, (dir), (dir)
@top R Data Import/Export

This is a guide to importing and exporting data to and from R.

The current version of this document is @value{VERSION}.
@end ifnottex

@menu
* Acknowledgements::            
* Introduction::                
* Spreadsheet-like data::       
* Importing from other statistical systems::  
* Relational databases::        
* Binary files::                
* Connections::                 
* Network interfaces::          
* References::                  
* Function and variable index::  
* Concept index::               
@end menu

@node Acknowledgements, Introduction, Top, Top
@unnumbered Acknowledgements

The relational databases part of this manual is based in part on an
earlier manual by Douglas Bates and Saikat DebRoy.  The principal author
of this manual was Brian Ripley.

Many volunteers have contributed to the packages used here.  The
principal authors of the packages mentioned are

@quotation
@multitable {stataread  xxxx} {A longggggggggggggggggggggggggggggggggg description}
@item @pkg{CORBA} @tab Duncan Temple Lang
@item @pkg{e1071} @tab Friedrich Leisch
@item @pkg{foreign} @tab Thomas Lumley, Saikat DebRoy, Douglas Bates and Duncan Murdoch
@c @item @pkg{hdf5} @tab Marcus Daniels
@item @pkg{Java} @tab John Chambers and Duncan Temple Lang
@item @pkg{netCDF} @tab Thomas Lumley
@item @pkg{RmSQL} @tab  Torsten Hothorn
@item @pkg{RMySQL} @tab David James and Saikat DebRoy
@item @pkg{RODBC} @tab Michael Lapsley
@item @pkg{RSPerl} @tab Duncan Temple Lang
@item @pkg{RPgSQL} @tab Timothy Keitt
@item @pkg{RSPython} @tab Duncan Temple Lang
@item @pkg{Rstreams} @tab Brian Ripley and Duncan Murdoch
@end multitable
@end quotation

Brian Ripley is the author of the support for connections.


@node Introduction, Spreadsheet-like data, Acknowledgements, Top
@chapter Introduction

Reading data into a statistical system for analysis and exporting the
results to some other system for report writing can be frustrating tasks
that can take far more time than the statistical analysis itself, even
though most readers will find the latter far more appealing.

This manual describes the import and export facilities available either
in @R{} itself or via packages which are available from @acronym{CRAN}.
Some of the packages described are still under development but they
already provide useful functionality.

Unless otherwise stated, everything described in this manual is
available for both Unix/Linux and Windows versions of @R{}.  (Much is not
yet available for the classic Macintosh port.)

In general, statistical systems like @R{} are not particularly well
suited to manipulations of large-scale data.  Some other systems are
better than @R{} at this, and part of the thrust of this manual is to
suggest that rather than duplicating functionality in @R{} we can make
the other system do the work!  (For example Therneau & Grambsch (2000)
comment that they prefer to do data manipulation in SAS and then use
@pkg{survival5} in @Sl{} for the analysis.)  Several recent packages
allow functionality developed in languages such as @code{Java},
@code{perl} and @code{python} to be directly integrated with @R{} code,
making the use of facilities in these languages even more
appropriate.  (See the @pkg{Java}, @pkg{RSPerl} and @pkg{RSPython}
packages.)



@cindex Unix tools
@cindex AWK
@cindex perl
It is also worth remembering that @R{} like @Sl{} comes from the Unix
tradition of small re-usable tools, and it can be rewarding to use tools
such as @code{awk} and @code{perl} to manipulate data before import or
after export.  The case study in Becker, Chambers & Wilks (1988, Chapter
9) is an example of this, where Unix tools were used to check and
manipulate the data before input to @Sl{}.  @R{} itself takes that
approach, using @code{perl} to manipulate its databases of help files
rather than @R{} itself, and the function @code{read.fwf} used a call to
a @code{perl} script until it was decided not to require @code{perl} at
run-time.  The traditional Unix tools are now much more widely
available, including on Windows (but not on classic Macintosh).

@menu
* Imports::                     
* Export to text files::        
* XML::                         
@end menu

@node Imports, Export to text files, Introduction, Introduction
@section Imports
@findex scan

The easiest form of data to import into @R{} is a simple text file, and
this will often be acceptable for problems of small or medium scale.
The primary function to import from a text file is @code{scan}, and this
underlies most of the more convenient functions discussed in
@ref{Spreadsheet-like data}.

However, all statistical consultants are familiar with being presented
by a client with a floppy disc or CD-R of data in some proprietary
binary format, for example `an Excel spreadsheet' or `an SPSS file'.
Often the simplest thing to do is to use the originating application to
export the data as a text file (and statistical consultants will have
copies of the commonest applications on their computers for that
purpose).  However, this is not always possible, and @ref{Importing from
other statistical systems} discusses what facilities are available to
access such files directly from @R{}.

In a few cases, data have been stored in a binary form for compactness
and speed of access.  One application of this that we have seen several
times is imaging data, which is normally stored as a stream of bytes as
represented in memory, possibly preceded by a header.  Such data formats
are discussed in @ref{Binary files} and @ref{Binary connections}.

For much larger databases it is common to handle the data using a
database management system (DBMS).  There is once again the option of
using the DBMS to extract a plain file, but for many such DBMSs the
extraction operation can be done directly from an @R{} package:
@xref{Relational databases}.  Importing data via network connections is
discussed in @ref{Network interfaces}.

@node Export to text files, XML, Imports, Introduction
@section Export to text files
@cindex Exporting to a text file

Exporting results from @R{} is usually a less contentious task, but
there are still a number of pitfalls.  There will be a target
application in mind, and normally a text file will be the most
convenient interchange vehicle.  (If a binary file is required, see
@ref{Binary files}.)

@findex cat
Function @code{cat} underlies the functions for exporting data.  It
takes a @code{file} argument, and the @code{append} argument allows a
text file to be written via successive calls to @code{cat}.

@findex write
@findex write.table
The commonest task is to write a matrix or data frame to file as a
rectangular grid of numbers, possibly with row and column labels.  This
can be done by the functions @code{write.table} and @code{write}.
Function @code{write} just writes out a matrix or vector in a specified
number of columns (and transposes a matrix).  Function
@code{write.table} is more convenient, and writes out a data frame (or
an object that can be coerced to a data frame) with row and column
labels.

There are a number of issues that need to be considered in writing out a
data frame to a text file.

@enumerate
@findex format
@item @strong{Precision}

These functions are based on @code{cat} not @code{print}, and the
precision to which numbers are printed is governed by the current
setting of @code{options(digits)}.  It may be necessary to increase this
to avoid losing precision.  For more control, use @code{format} on a
data frame, possibly column-by-column.

@item @strong{Header line}

@R{} prefers the header line to have no entry for the row names, so the
file looks like

@smallexample
                dist    climb   time
Greenmantle     2.5     650     16.083
   ...
@end smallexample

@noindent
Some other systems require a (possibly empty) entry for the row names, which
is what @code{write.table} will provide if argument @code{col.names = NA}
is specified.  Excel is one such system.

@item @strong{Separator}
@cindex CSV files
@cindex comma separated values

A common field separator to use in the file is a comma, as that is
unlikely to appear in any of the fields, in English-speaking countries.
Such files are known as CSV (comma separated values) files.
In some locales the comma is used as the decimal point
(set this in @code{write.table} by @code{dec = ","}) and there CSV files
use the semicolon as the field separator.

Using a semicolon or tab (@code{sep = "\t"}) are probably the safest
options.

@item @strong{Missing values}
@cindex Missing values

By default missing values are output as @code{NA}, but this may be
changed by argument @code{na}.  Note that @code{NaN}s are treated as
@code{NA} by @code{write.table}, but not by @code{cat} nor @code{write}.

@item @strong{Quoting strings}
@cindex Quoting strings

By default strings are quoted (including the row and column names).
Argument @code{quote} controls quoting of character and factor variables.

Some care is needed if the strings contain embedded quotes.  Three
useful forms are

@smallexample
> df <- data.frame(a = I("a \" quote"))
> write.table(df)
"a"
"1" "a \" quote"
> write.table(df, qmethod = "double")
"a"
"1" "a "" quote"
> write.table(df, quote = FALSE, sep = ",")
a
1,a " quote
@end smallexample

@noindent
The second is the form of escape commonly used by spreadsheets.
@end enumerate

@findex sink
It is possible to use @code{sink} to divert the standard @R{} output to
a file, and thereby capture the output of (possibly implicit)
@code{print} statements.  This is not usually the most efficient route,
and the @code{options(width)} setting may need to be increased.

@node XML,  , Export to text files, Introduction
@section XML
@cindex XML

When reading data from text files, it is the responsibility of the user
to know and to specify the conventions used to create that file,
e.g. the comment character, whether a header line is present, the value
separator, the representation for missing values (and so on) described
in @ref{Export to text files}.  A markup language which can be used to
describe not only content but also the structure of the content can
make a file self-describing, so that one need not provide these details
to the software reading the data.

The eXtensible Markup Language -- more commonly know simply as
@acronym{XML} -- can be used to provide such structure, not only for
standard datasets but also more complex data structures.
@acronym{XML} is becoming extremely popular and is emerging as a
standard for general data markup and exchange.  It is being used by
different communities to describe geographical data such as maps,
graphical displays, mathematics and so on.

The @pkg{XML} package provides general facilities for reading and
writing @acronym{XML} documents within both @R{} and S-PLUS in the hope
that we can easily make use of this technology as it evolves.  Several
people are exploring how we can use @acronym{XML} for, amongst other
things, representing datasets to be shared across different
applications; storing @R{} and S-PLUS objects so they can be shared by
both systems; representing plots via @acronym{SVG} (Scalable Vector
Graphics, a dialect of @acronym{XML}); representing function
documentation; generating ``live'' analyses/reports that contain text,
data and code.

A description of the facilities of the @pkg{XML} package is outside the
scope of this document: see the package's Web page at
@uref{http://www.omegahat.org/RSXML} for details and examples.


@node Spreadsheet-like data, Importing from other statistical systems, Introduction, Top
@chapter Spreadsheet-like data
@cindex Spreadsheet-like data

@menu
* Variations on read.table::    
* Fixed-width-format files::    
* Using scan directly::         
* Re-shaping data::             
* Flat contingency tables::     
@end menu

In @ref{Export to text files} we saw a number of variations on the
format of a spreadsheet-like text file, in which the data are presented
in a rectangular grid, possibly with row and column labels.  In this
section we consider importing such files into @R{}.

@node Variations on read.table, Fixed-width-format files, Spreadsheet-like data, Spreadsheet-like data
@section  Variations on @code{read.table}
@findex read.table

The function @code{read.table} is the most convenient way to read in a
rectangular grid of data.  Because of the many possibilities, there are
several other functions that call @code{read.table} but change a group
of default arguments.

Some of the issues to consider are:

@enumerate
@item @strong{Header line}

We recommend that you specify the @code{header} argument explicitly,
Conventionally the header line has entries only for the columns and not
for the row labels, so is one field shorter than the remaining lines.
(If @R{} sees this, it sets @code{header = TRUE}.)  If presented with a
file that has a (possibly empty) header field for the row labels, read
it in by something like

@smallexample
read.table("file.dat", header = TRUE, row.names = 1)
@end smallexample

Column names can be given explicitly via the @code{col.names}; explicit
names override the header line (if present).

@item @strong{Separator}

Normally looking at the file will determine the field separator to be
used, but with white-space separated files there may be a choice between
the default @code{sep = ""} which uses any white space (spaces, tabs or
newlines) as a separator, @code{sep = " "} and @code{sep = "\t"}.  Note
that the choice of separator affects the input of quoted strings.  

If you have a tab-delimited file containing empty fields be sure to use
@code{sep = "\t"}.


@item @strong{Quoting}
@cindex Quoting strings

By default character strings can be quoted by either @samp{"} or
@samp{'}, and in each case all the characters up to a matching quote are
taken as part of the character string.  The set of valid quoting
characters (which might be none) is controlled by the @code{quote}
argument.  For @code{sep = "\n"} the default is changed to @code{quote =
""}.

If no separator character is specified, quotes can be escaped within
quoted strings by immediately preceding them by @samp{\}, C-style.

If a separator character is specified, quotes can be escaped within
quoted strings by doubling them as is conventional in spreadsheets.  For
example

@smallexample
'One string isn''t two',"one more"
@end smallexample

@noindent
can be read by

@smallexample
read.table("testfile", sep = ",")
@end smallexample

@noindent
This does not work with the default separator.

@item @strong{Missing values}
@cindex Missing values

By default the file is assumed to contain the character string @code{NA}
to represent missing values, but this can be changed by the argument
@code{na.strings}, which is a vector of one or more character
representations of missing values.

Empty fields in numeric columns are also regarded as missing values.

@item @strong{Unfilled lines}

It is quite common for a file exported from a spreadsheet to have all
trailing empty fields (and their separators) omitted.  To read such
files set @code{fill = TRUE}.

@item @strong{White space in character fields}

If a separator is specified, leading and trailing white space in
character fields is regarded as part of the field.  To strip the space,
use argument @code{strip.white = TRUE}.

@item @strong{Blank lines}

By default, @code{read.table} ignores empty lines.  This can be changed
by setting @code{blank.lines.skip = FALSE}, which will only be useful in
conjunction with @code{fill = TRUE}, perhaps to indicate missing data in
a regular layout.

@end enumerate

@findex read.csv
@findex read.csv2
@findex read.delim
@findex read.delim2
@cindex CSV files
Convenience functions @code{read.csv} and @code{read.delim} provide
arguments to @code{read.table} appropriate for CSV and tab-delimited
files exported from spreadsheets in English-speaking locales.  The
variations @code{read.csv2} and @code{read.delim2} are appropriate for
use in countries where the comma is used for the decimal point.

If the options to @code{read.table} are specified incorrectly, the error
message will usually be of the form

@smallexample
Error in scan(file = file, what = what, sep = sep, : 
        line 1 did not have 5 elements
@end smallexample
@noindent
or
@smallexample
Error in read.table("files.dat", header = TRUE) : 
        more columns than column names
@end smallexample

@findex count.fields

@noindent
This may give enough information to find the problem, but the auxiliary
function @code{count.fields} can be useful to investigate further.

@node Fixed-width-format files, Using scan directly, Variations on read.table, Spreadsheet-like data
@section Fixed-width-format files
@cindex Fixed-width-format files

Sometimes data files have no field delimiters but have fields in
pre-specified columns.  This was very common in the days of punched
cards, and is still sometimes used to save file space.

@findex read.fwf
Function @code{read.fwf} provides a simple way to read such files,
specifying a vector of field widths.  The function reads the file into
memory as whole lines, splits the resulting character strings, writes
out a temporary tab-separated file and then calls @code{read.table}.
This is adequate for small files, but for anything more complicated we
recommend using the facilities of a language like @code{perl} to
pre-process the file.
@cindex perl

@node Using scan directly, Re-shaping data, Fixed-width-format files, Spreadsheet-like data
@section  Using @code{scan} directly
@findex scan

Both @code{read.table} and @code{read.fwf} use @code{scan} to read the
file, and then process the results of @code{scan}.  They are very
convenient, but sometimes it is better to use @code{scan} directly.

Function @code{scan} has many arguments, most of which we have already
covered under @code{read.table}.  The most crucial argument is
@code{what}, which specifies a list of modes of variables to be read
from the file.  If the list is named, the names are used for the
components of the returned list.  Modes can be numeric, character or
complex, and are usually specified by an example, e.g. @code{0},
@code{""} or @code{0i}.  For example

@smallexample
cat("2 3 5 7", "11 13 17 19", file="ex.dat", sep="\n")
scan(file="ex.dat", what=list(x=0, y="", z=0), flush=TRUE)
@end smallexample

@noindent
returns a list with three components and discards the fourth column in
the file.

@findex readLines
There is a function @code{readLines} which will be more convenient if
all you want is to read whole lines into @R{} for further processing.


@node Re-shaping data, Flat contingency tables, Using scan directly, Spreadsheet-like data
@section Re-shaping data
@cindex Re-shaping data

Sometimes spreadsheet data is in a compact format that gives the
covariates for each subject followed by all the observations on that
subject.  @R{}'s modelling functions need observations in a single
column.  Consider the following sample of data from repeated MRI brain
measurements

@example
 Status   Age    V1     V2     V3    V4
      P 23646 45190  50333  55166 56271
     CC 26174 35535  38227  37911 41184
     CC 27723 25691  25712  26144 26398
     CC 27193 30949  29693  29754 30772
     CC 24370 50542  51966  54341 54273
     CC 28359 58591  58803  59435 61292
     CC 25136 45801  45389  47197 47126
@end example

@noindent
There are two covariates and up to four measurements on each subject.
The data were exported from Excel as a file @file{mr.csv}.

@findex stack
We can use @code{stack} to help manipulate these data to give a single
response.

@smallexample
zz <- read.csv("mr.csv", strip.white = TRUE)
zzz <- cbind(zz[gl(nrow(zz), 1, 4*nrow(zz)), 1:2], stack(zz[, 3:6]))
@end smallexample

@noindent
with result

@example
      Status   Age values ind
X1         P 23646  45190  V1
X2        CC 26174  35535  V1
X3        CC 27723  25691  V1
X4        CC 27193  30949  V1
X5        CC 24370  50542  V1
X6        CC 28359  58591  V1
X7        CC 25136  45801  V1
X11        P 23646  50333  V2
...
@end example

@findex unstack.
Function @code{unstack} goes in the opposite direction, and may be
useful for exporting data.

@findex reshapeLong
Another way to do this is to use the (experimental) function
@code{reshapeLong}, by
@smallexample
> reshapeLong(zz, V1:V4)
   Status   Age reshape.i reshape.j reshape.v
1       P 23646         1        V1     45190
2       P 23646         1        V2     50333
3       P 23646         1        V3     55166
4       P 23646         1        V4     56271
5      CC 26174         2        V1     35535
6      CC 26174         2        V2     38227
7      CC 26174         2        V3     37911
8      CC 26174         2        V4     41184
...
@end smallexample


@node Flat contingency tables,  , Re-shaping data, Spreadsheet-like data
@section Flat contingency tables
@cindex Flat contingency tables

Displaying higher-dimensional contingency tables in array form typically
is rather inconvenient.  In categorical data analysis, such information
is often represented in the form of bordered two-dimensional arrays with
leading rows and columns specifying the combination of factor levels
corresponding to the cell counts.  These rows and columns are typically
``ragged'' in the sense that labels are only displayed when they change,
with the obvious convention that rows are read from top to bottom and
columns are read from left to right.  In @R{}, such ``flat'' contingency
tables can be created using @code{ftable},
@findex ftable
which creates objects of class @code{"ftable"} with an appropriate print
method.

As a simple example, consider the @R{} standard data set
@code{UCBAdmissions} which is a 3-dimensional contingency table
resulting from classifying applicants to graduate school at UC Berkeley
for the six largest departments in 1973 classified by admission and sex.

@smallexample
> data(UCBAdmissions)
> ftable(UCBAdmissions)
                Dept   A   B   C   D   E   F
Admit    Gender                             
Admitted Male        512 353 120 138  53  22
         Female       89  17 202 131  94  24
Rejected Male        313 207 205 279 138 351
         Female       19   8 391 244 299 317
@end smallexample

@noindent
The printed representation is clearly more useful than displaying the
data as a 3-dimensional array.

There is also a function @code{read.ftable} for reading in flat-like
contingency tables from files.
@findex read.ftable
This has additional arguments for dealing with variants on how exactly
the information on row and column variables names and levels is
represented.  The help page for @code{read.ftable} has some useful
examples.  The flat tables can be converted to standard contingency
tables in array form using @code{as.table}.

Note that flat tables are characterized by their ``ragged'' display of
row (and maybe also column) labels.  If the full grid of levels of the
row variables is given, one should instead use @code{read.table} to read
in the data, and create the contingency table from this using
@code{xtabs}.


@node Importing from other statistical systems, Relational databases, Spreadsheet-like data, Top
@chapter Importing from other statistical systems
@cindex Importing from other statistical systems

In this chapter we consider the problem of reading a binary data file
written by another statistical system.  This is often best avoided, but
may be unavoidable if the originating system is not available.

@menu
* Minitab SAS S-PLUS SPSS Stata::  
* Octave::                      
@end menu

@node Minitab SAS S-PLUS SPSS Stata, Octave, Importing from other statistical systems, Importing from other statistical systems
@section Minitab, S-PLUS, SAS, SPSS, Stata

The recommended package @pkg{foreign} provides import facilities for
files produced by these statistical systems, and for export to Stata.

@cindex Stata
@findex read.dta
@findex write.dta
Stata @file{.dta} files are a binary file format. Files from
versions 5.0, 6.0 and 7.0 of Stata can be read and written by functions
@code{read.dta} and @code{write.dta}.

@cindex Minitab
@findex read.mtp
Function @code{read.mtp} imports a `Minitab Portable Worksheet'.  This
returns the components of the worksheet as an @R{} list.

@cindex SAS
@findex read.xport
Function @code{read.xport} reads a file in SAS Transport (XPORT) format and
return a list of data frames.

@cindex SPSS
@findex read.spss
Function @code{read.spss} can read files created by the `save' and
`export' commands in @acronym{SPSS}.  It returns a list with one
component for each variable in the saved data set.


@cindex S-PLUS
@findex read.S
@findex data.restore

Function @code{read.S} which can read binary objects produced by S-PLUS
3.x, 4.x or 2000 on (32-bit) Unix or Windows (and can read them on a
different OS).  This is able to read many but not all @Sl{} objects: in
particular it can read vectors, matrices and data frames and lists
containing those.

Function @code{data.restore} to read S-PLUS data dumps (created by
@code{data.dump}) with the same restrictions (except that dumps from the
Alpha platform can also be read).  It should be possible to read data
dumps from S-PLUS 5.x and 6.x written with @code{data.dump(oldStyle=T)}.

@node Octave,  , Minitab SAS S-PLUS SPSS Stata, Importing from other statistical systems
@section Octave
@cindex Octave
@findex read.octave

Octave is a numerical linear algebra system, and function
@code{read.octave} in package @pkg{e1071} can read the first vector or
matrix from an Octave ASCII data file created using the Octave command
@command{save -ascii}.

@node Relational databases, Binary files, Importing from other statistical systems, Top
@chapter Relational databases

@cindex Relational databases
@cindex DBMS

@menu
* Why use a database?::         
* Overview of RDBMSs::          
* R interface packages::        
@end menu

@node Why use a database?, Overview of RDBMSs, Relational databases, Relational databases
@section Why use a database?

There are limitations on the types of data that @R{} handles well.
Since all data being manipulated by @R{} are resident in memory, and
several copies of the data can be created during execution of a
function, @R{} is not well suited to extremely large data sets.  Data
objects that are more than a few (tens of) megabytes in size can cause
@R{} to run out of memory.

@R{} does not easily support concurrent access to data.  That is, if
more than one user is accessing, and perhaps updating, the same data,
the changes made by one user will not be visible to the others.

@R{} does support persistence of data, in that you can save a data
object or an entire worksheet from one session and restore it at the
subsequent session, but the format of the stored data is specific to
@R{} and not easily manipulated by other systems.

Database management systems (DBMSs) and, in particular, relational
DBMSs (RDBMSs) @emph{are} designed to do all of these things well.
Their strengths are

@enumerate
@item
To provide fast access to selected parts of large databases.

@item
Powerful ways to summarize and cross-tabulate columns in databases.

@item
Store data in more organized ways than the rectangular grid model of
spreadsheets and @R{} data frames.

@item
Concurrent access from multiple clients running on multiple hosts while
enforcing security constraints on access to the data.

@item
Ability to act as a server to a wide range of clients.
@end enumerate

The sort of statistical applications for which DBMS might be used are to
extract a 10% sample of the data, to cross-tabulate data to produce a
multi-dimensional contingency table, and to extract data group by group
from a database for separate analysis.

@node Overview of RDBMSs, R interface packages, Why use a database?, Relational databases
@section Overview of RDBMSs

Traditionally there have been large (and expensive) commercial RDBMSs
(@uref{http://www.informix.com, Informix}; @uref{http://www.oracle.com,
Oracle}; @uref{http://www.sybase.com, Sybase}; IBM's DB/2; Microsoft
@acronym{SQL} Server on Windows) and academic and small-system databases
(such as MySQL, PostgreSQL, Microsoft Access, @dots{}), the former
marked out by much greater emphasis on data security features.  The line
is blurring, with the Open Source PostgreSQL having more and more
high-end features, and `free' versions of Informix, Oracle and Sysbase
being made available on Linux.

@cindex ODBC
@cindex Open Database Connectivity
There are other commonly used data sources, including spreadsheets,
non-relational databases and even text files (possibly compressed).
Open Database Connectivity (@acronym{ODBC}) is a standard to use all of
these data sources.  It originated on Windows (see
@uref{http://www.microsoft.com/data/odbc/}) but is also implemented on
Linux.

All of the packages described later in this chapter provide clients to
client/server databases.  The database can reside on the same machine or
(more often) remotely.  There is an @acronym{ISO} standard (in fact
several: @acronym{SQL}92 is @acronym{ISO}/IEC 9075, also known as
@acronym{ANSI} X3.135-1992, and @acronym{SQL}99 is coming into use) for
an interface language called @acronym{SQL} (Structured Query Language,
sometimes pronounced `sequel': see Bowman @emph{et al.@:} 1996 and Kline
and Kline 2001) which these DBMSs support to varying degrees.


@menu
* SQL queries::                 
* Data types::                  
@end menu

@node SQL queries, Data types, Overview of RDBMSs, Overview of RDBMSs
@subsection @acronym{SQL} queries
@cindex SQL queries

The more comprehensive @R{} interfaces generate @acronym{SQL} behind the
scenes for common operations, but direct use of @acronym{SQL} is needed
for complex operations in all.  Conventionally @acronym{SQL} is written
in upper case, but many users will find it more convenient to use lower
case in the @R{} interface functions.

A relational DBMS stores data as a database of @emph{tables} (or
@emph{relations}) which are rather similar to @R{} data frames, in that
they are made up of @emph{columns} or @emph{fields} of one type
(numeric, character, date, currency, @dots{}) and @emph{rows} or
@emph{records} containing the observations for one entity.

@acronym{SQL} `queries' are quite general operations on a relational
database.  The classical query is a SELECT statement of the type

@smallexample
SELECT State, Murder FROM USArrests WHERE rape > 30 ORDER BY Murder

SELECT t.sch, c.meanses, t.sex, t.achieve
  FROM student as t, school as c WHERE t.sch = c.id

SELECT sex, COUNT(*) FROM student GROUP BY sex

SELECT sch, AVG(sestat) FROM student GROUP BY sch LIMIT 10
@end smallexample

@noindent
The first of these selects two columns from the @R{} data frame
@code{USArrests} that has been copied across to a database table,
subsets on a third column and asks the results be sorted.  The second
performs a database @emph{join} on two tables @code{student} and
@code{school} and returns four columns.  The third and fourth queries do
some cross-tabulation and return counts or averages.  (The five
aggregation functions are COUNT(*) and SUM, MAX, MIN and AVG, each
applied to a single column.)

SELECT queries use FROM to select the table, WHERE to specify a
condition for inclusion (or more than one condition separated by AND or
OR), and ORDER BY to sort the result.  Unlike data frames, rows in RDBMS
tables are best thought of as unordered, and without an ORDER BY
statement the ordering is indeterminate.  You can sort (in
lexicographical order) on more than one column by separating them by
commas.  Placing DESC after an ORDER BY puts the sort in descending
order.

SELECT DISTINCT queries will only return one copy of each distinct row
in the selected table.

The GROUP BY clause selects subgroups of the rows according to the
criterion.  If more than one column is specified (separated by commas)
then multi-way cross-classifications can be summarized by one of the
five aggregation functions.  A HAVING clause allows the select to
include or exclude groups depending on the aggregated value.

If the SELECT statement contains an ORDER BY statement that produces a
unique ordering, a LIMIT clause can be added to select (by number) a
contiguous block of output rows.  This can be useful to retrieve rows a
block at a time.  (It may not be reliable unless the ordering is unique,
as the LIMIT clause can be used to optimize the query.)

There are queries to create a table (CREATE TABLE, but usually one
copies a data frame to the database in these interfaces), INSERT or
DELETE or UPDATE data.  A table is destroyed by a DROP TABLE `query'.

Kline and Kline (2001) discuss the details of the implementation of SQL
in SQL Server 2000, Oracle, MySQL and PostgreSQL.

@node Data types,  , SQL queries, Overview of RDBMSs
@subsection Data types

Data can be stored in a database in various data types.  The range of
data types is DBMS-specific, but the @acronym{SQL} standard defines many
types, including the following that are widely implemented (often not by
the @acronym{SQL} name).

@c  @quotation
@c  @multitable {Data type name varying} {A longgggggggggggggggggggg description}

@c  @item name @tab  description

@c  @item @code{float(p)}   @tab Real number, with optional precision
@c  @item @code{integer}   @tab 32-bit integer
@c  @item @code{smallint}  @tab 16-bit integer
@c  @item @code{character(n)}   @tab fixed-length character string
@c  @item @code{character varying(n)} @tab variable-length character string
@c  @item @code{boolean}  @tab true or false.
@c  @item @code{date}  @tab calendar date
@c  @item @code{time}  @tab time of day
@c  @item @code{timestamp}  @tab date and time

@c  @end multitable
@c  @end quotation

@table @code
@item float(@var{p})
Real number, with optional precision.  Often called @code{real} or
@code{double} or @code{double precision}.
@item integer
32-bit integer.  Often called @code{int}.
@item smallint
16-bit integer
@item character(@var{n})
fixed-length character string.  Often called @code{char}.
@item character varying(@var{n})
variable-length character string.  Often called @code{varchar}.
@item boolean
true or false.  Sometimes called @code{bool}.
@item date
calendar date
@item time
time of day
@item timestamp
date and time
@end table

@noindent
There are variants on @code{time} and @code{timestamp}, @code{with
timezone}.

The more comprehensive of the @R{} interface packages hide the type
conversion issues from the user.

@node R interface packages,  , Overview of RDBMSs, Relational databases
@section R interface packages

There are four packages available on @acronym{CRAN} to help @R{}
communicate with DBMSs.  They provide different levels of abstraction.
Some provide means to copy whole data frames to and from databases.  All
have functions to select data within the database via @acronym{SQL}
queries, and (except @pkg{RmSQL}) to retrieve the result as a whole as a
data frame or in pieces (usually as groups of rows, but @pkg{RPgSQL} can
retrieve columns).  All except @pkg{RODBC} are (currently) tied to one
DBMS.


@menu
* RPgSQL::                      
* RODBC::                       
* RMySQL::                      
* RmSQL::                       
@end menu

@node RPgSQL, RODBC, R interface packages, R interface packages
@subsection Package RPgSQL
@cindex PostgreSQL database system

Package @pkg{RPgSQL} at @uref{http://rpgsql.sourceforge.net/} and on
@acronym{CRAN} provides an interface to @uref{http://www.postgresql.org,
PostgreSQL}.

PostgreSQL is described by its developers as `the most advanced open
source database server' (Momjian, 2000).  It would appear to be buildable
for most Unix-alike OSes and Windows (under Cygwin or U/Win).
PostgreSQL has most of the features of the commercial RDBMSs.

@pkg{RPgSQL} is the most mature and comprehensive of these RDBMS
interfaces.

@findex db.connect
@findex db.read.table
@findex db.write.table
To make use of @pkg{RPgSQL}, first open a connection to a database using
@code{db.connect}.  (Currently only one connection can be open at a
time.)  Once a connection is open an @R{} data frame can be copied to a
PostgreSQL table by @code{db.write.table}, whereas @code{db.read.table}
copies a PostgreSQL table to an @R{} data frame.

@findex bind.db.proxy
@cindex proxy data frame
@pkg{RPgSQL} has the interesting concept of a @emph{proxy data frame}.
A data frame proxy is an @R{} object that inherits from the
@code{"data.frame"} class, but contains no data.  All accesses to the
proxy data frame generate the appropriate @acronym{SQL} query and
retrieve the resulting data from the database.  A proxy data frame is
set up by a call to @code{bind.db.proxy}.  To remove the proxy, just
remove the object which @code{bind.db.proxy} created.

@findex db.execute
@findex db.result.columns
@findex db.result.rows
@findex db.read.column
@findex db.fetch.result
@findex db.clear.result
@findex db.result.get.value
A finer level of control is available via sending @acronym{SQL} queries
to the PostgreSQL server via @code{db.execute}.  This leaves a result in
PostgreSQL's result cache, unless flushed by @code{clear = TRUE} (the
default).  Once a result is in the cache, @code{db.fetch.result} can be
used to fetch the whole result as a data frame.  Functions such as
@code{db.result.columns} and @code{db.result.rows} will report the
number of columns and rows in the cached table, and
@code{db.read.column} will fetch a single column (as a vector).  An
individual cell in the result can be read by @code{db.result.get.value}.
@code{db.clear.result} will clear the result cache.

@findex sql.insert
@findex sql.select
One disadvantage is that PostgreSQL maps all table and column names to
lower case, so for maximal flexibility, only use lower case in @R{}
names.  Functions @code{sql.insert} and @code{sql.select} provide
convenience wrappers for the INSERT and SELECT queries.

We can explore these functions in a simple example.  The database
@file{testdb} had already been set up, and as PostgreSQL was running on
a standalone machine no further authentication was required to connect.

@smallexample
> library(RPgSQL)
> db.connect(dbname="testdb")   # add authentication as needed
Connected to database "testdb" on ""
> data(USArrests)
> usarrests <- USArrests
> names(usarrests) <- tolower(names(USArrests))
> db.write.table(USArrests, write.row.names = TRUE)
> db.write.table(usarrests, write.row.names = TRUE)
> rm(USArrests, usarrests)
## db.ls lists tables in the database.
> db.ls()
[1] "USArrests"  "usarrests"
> db.read.table("USArrests")
               Murder Assault UrbanPop Rape
Alabama          13.2     236       58 21.2
Alaska           10.0     263       48 44.5
   ...
## set up a proxy data frame.  Remember USArrests has been removed
> bind.db.proxy("USArrests")
## USArrests is now a proxy, so all accesses are to the database
> USArrests[, "Rape"]
   Rape
1  21.2
2  44.5
   ...
> rm(USArrests) # remove proxy
> db.execute("SELECT rpgsql_row_names, murder FROM usarrests",
             "WHERE rape > 30 ORDER BY murder", clear=FALSE)
> db.fetch.result()
           murder
Colorado      7.9
Arizona       8.1
California    9.0
Alaska       10.0
New Mexico   11.4
Michigan     12.1
Nevada       12.2
Florida      15.4
> db.rm("USArrests", "usarrests") # use ask=FALSE to skip confirmation
Destroy table USArrests? y
Destroy table usarrests? y
> db.ls()
character(0)
> db.disconnect()
@end smallexample

@noindent
Notice how the row names are mapped if @code{write.row.names = TRUE} to
a field @code{rpgsql_row_names} in the database table and transparently
restored provided we preserve that field in the query.

@pkg{RPgSQL} provides means to extend its mapping between @R{} classes
within a data frame and PostgreSQL types.


@node RODBC, RMySQL, RPgSQL, R interface packages
@subsection Package RODBC
@cindex ODBC
@cindex Open Database Connectivity

Package @pkg{RODBC} on @acronym{CRAN} provides an interface to database
sources supporting an @acronym{ODBC} interface.  This is very widely
available, and allows the same @R{} code to access different database
systems.  @pkg{RODBC} runs on both Linux and Windows, and many database
systems provide support @acronym{ODBC}, including most of those on
Windows (such as Microsoft Access), and MySQL, Oracle and PostgreSQL on
Unix/Linux.

You will need an @acronym{ODBC} Driver Manager such as unixODBC
(@uref{http://www.unixODBC.org}) or iOBDC (@uref{http://www.iODBC.org})
on Unix/Linux, and an installed driver for your database system.  The
FreeODBC project (@uref{http://www.jepstone.net/FreeODBC/}) is a
repository of information related to @acronym{ODBC}.

Two groups of interface functions are provided.  The @code{odbc*} group
provide a low-level interface to the basic @acronym{ODBC} functions: see
the help page (@code{?RODBC}) for details.  The @code{sql*} group
provide an interface between @R{} data frames and @acronym{SQL} tables.

@findex odbcConnect
@findex odbcClose
@findex sqlTables
Up to 16 simultaneous connections are possible.  A connection is opened
by a call to @code{odbcConnect} which returns a handle used for
subsequent access to the database.  A connection is closed by
@code{odbcClose}.  Details of the tables on a connection can be found
using @code{sqlTables}.

@findex sqlFetch
@findex sqlSave
Function @code{sqlSave} copies an @R{} data frame to a table in the
database, and @code{sqlFetch} copies a table in the database to an @R{}
data frame.

@findex sqlQuery
@findex sqlCopy
@findex odbcQuery
@findex sqlGetResults
@findex odbcFetchRows
An @acronym{SQL} query can be sent to the database by a call to
@code{sqlQuery}.  This returns the result in an @R{} data frame.
(@code{sqlCopy} sends a query to the database and saves the result as a
table in the database.)  A finer level of control is attained by first
calling @code{odbcQuery} and then @code{sqlGetResults} to fetch the
results.  The latter can be used within a loop to retrieve a limited
number of rows at a time.  @code{sqlGetResults} by default returns a
data frame, but the raw results can also be obtained as a character
matrix.

@cindex PostgreSQL database system
Here is an example using PostgreSQL, for which the @acronym{ODBC} driver
maps column and data frame names to lower case.  We use a database
@code{testdb} we created earlier, and had the DSN (data source name) set
up in @file{~/.odbc.ini} under @code{unixODBC}.  Exactly the same code
worked using MyODBC to access a MySQL database under Linux or Windows NT
(where MySQL also maps names to lowercase).  Under Windows,
@acronym{DSN}s are set up in the @acronym{ODBC} applet in the Control
Panel.
@cindex MySQL database system

@smallexample
> library(RODBC)
## tell it to map names to l/case
> channel <- odbcConnect("testdb", uid="ripley", case="tolower")
## load a data frame into the database
> data(USArrests)
> sqlSave(channel, USArrests, rownames="state")
> rm(USArrests)
## list the tables in the database
> sqlTables(channel)
  TABLE_QUALIFIER TABLE_OWNER TABLE_NAME TABLE_TYPE REMARKS
1                              usarrests      TABLE        
## list it
> sqlFetch(channel, "USArrests", rownames = TRUE)
               murder assault urbanpop rape
Alabama          13.2     236       58 21.2
Alaska           10.0     263       48 44.5
    ...
## an SQL query, originally on one line
> sqlQuery(channel, "select state, murder from USArrests
           where rape > 30 order by murder")
       state murder
1 Colorado      7.9
2 Arizona       8.1
3 California    9.0
4 Alaska       10.0
5 New Mexico   11.4
6 Michigan     12.1
7 Nevada       12.2
8 Florida      15.4
## remove the table
> sqlDrop(channel, "USArrests")
## close the connection
> odbcClose(channel)
@end smallexample

As a simple example of using @acronym{ODBC} with a Excel spreadsheet,
suppose that an @acronym{DSN} for spreadsheet @file{bdr.xls}
has been set up in the Control Panel.  Then we can read from the
spreadsheet by
@smallexample
> library(RODBC)
> channel <- odbcConnect("bdr.xls")
## list the spreadsheets
> sqlTables(channel)
  TABLE_CAT TABLE_SCHEM        TABLE_NAME   TABLE_TYPE REMARKS
1 C:\\bdr            NA           Sheet1$ SYSTEM TABLE      NA
2 C:\\bdr            NA           Sheet2$ SYSTEM TABLE      NA
3 C:\\bdr            NA           Sheet3$ SYSTEM TABLE      NA
4 C:\\bdr            NA Sheet1$Print_Area        TABLE      NA
## retrieve the contents of sheet 1
> sh1 <- sqlQuery(channel, "select * from [Sheet1$]")
@end smallexample
@noindent
Notice that the specification of the table is different from the name
returned by @code{sqlTables}: this precludes the use of @code{sqlFetch}.

@node RMySQL, RmSQL, RODBC, R interface packages
@subsection Package RMySQL
@cindex MySQL database system

Package @pkg{RMySQL} on @acronym{CRAN} provides an
interface to the MySQL database system (see @uref{http://www.mysql.com}
and Dubois, 2000.).  This is part of a project to provide a common
@acronym{API} for access from @R{} to relational DBMSs using
@acronym{SQL}.  Currently this provides lower-level facilities than
@pkg{RPgSQL} or @pkg{RODBC}.

MySQL exists on Unix/Linux and Windows: as from version 3.23 (Jan 2001)
it is released under GPL.  MySQL is a `light and lean' database.  (It
preserves the case of names where the operating file system is
case-sensitive, so not on Windows.)  Package @pkg{RMySQL} has been used
on both Linux and Windows.

@findex MySQL
@findex dbConnect
@findex close
A call to the function @code{MySQL} returns a database connection
manager object, and then a call to @code{dbConnect} opens a database
connection which can subsequently be closed by a call to the generic
function @code{close}.

@findex quickSQL
@findex dbExecStatement
@acronym{SQL} queries can be sent by either @code{quickSQL} or
@code{dbExecStatement}.  @code{quickSQL} sends the query and retrieves the
results as a data frame.   @code{dbExecStatement} sends the query and
returns an object of class @code{"MySQLResultSet"} which can be used to
retrieve the results, and subsequently used to @code{close} the result.

@findex fetch
Function @code{fetch} is used to retrieve some or all of the rows in the
query result, as a list.  The function @code{hasCompleted} indicates if
all the rows have been fetched, and @code{getRowCount} returns the
number of rows in the result.

@findex assignTable
@findex getTable
As from @code{RMySQL} version 0.4 there are convenience functions
@code{assignTable} to load a data frame into the database and
@code{getTable} to retrieve a table as a data frame.  The row names
get copied into the table as column @code{row_names} by
@code{assignTable}, but they are not copied back by @code{getTable}.

@smallexample
> library(RMySQL)
## open a connection to a MySQL database
> con <- dbConnect(MySQL(), dbname = "test")
## list the tables in the database
> getTables(con)
## load a data frame into the database, deleting any existing copy
> data(USArrests)
> assignTable(con, "arrests", USArrests, overwrite = TRUE)
## get the whole table
> getTable(con, "arrests")
   Murder Assault UrbanPop Rape      row_names
1    13.2     236       58 21.2        Alabama
2    10.0     263       48 44.5         Alaska
3     8.1     294       80 31.0        Arizona
...
## Select from the loaded table: all on one line
> quickSQL(con, "select row_names, Murder from arrests
           where Rape > 30 order by Murder")
   row_names Murder
1   Colorado    7.9
2    Arizona    8.1
3 California    9.0
4     Alaska   10.0
5 New Mexico   11.4
6   Michigan   12.1
7     Nevada   12.2
8    Florida   15.4
> close(con)
@end smallexample

@node RmSQL,  , RMySQL, R interface packages
@subsection Package RmSQL
@cindex Mini SQL database system

Package @pkg{RmSQL} on @acronym{CRAN} provides an interface to the Mini
@acronym{SQL} database system (also known as mSQL,
@url{http://www.hughes.com.au}, Yarger @emph{et al.@:}, 1999).  The
package documentation describes mSQL as
@quotation
Note that mSQL is NOT GPL licenced but free of charge for universities
and noncommercial organisations.
@end quotation
@noindent
@pkg{RmSQL} provides the most basic interface of those in this chapter,
a wrapper to the C-@acronym{API} of mSQL with no additional
functionality.

@findex msqlConnect
@findex msqlSelect
@findex msqlClose
@findex msqlQuery
@findex msqlStoreResult
@findex msqlFreeResult
A database connection is opened by first selecting a host with
@code{msqlConnect} and then a database by @code{msqlSelect}.  The
connection is closed by a call to @code{msqlClose}.  Then an
@acronym{SQL} query is sent by a call to @code{msqlQuery}, and the
results stored by a call to @code{msqlStoreResult}.  When a query is
finished with, the result can be freed by @code{msqlFreeResult}.

@findex msqlFetchRow
@findex msqlDataSeek
@findex msqlNumRows
Once the result of a query has been stored, the values can be retrieved
row by row using @code{msqlFetchRow}.  This fetches the rows in order
unless the position is reset by a call to @code{msqlDataSeek}.  A call to
@code{msqlNumRows} gives the total number of rows in the result.

No example is given here as the basic interface makes any example
lengthy, but there is one in the @file{Example} directory of the package.


@node Binary files, Connections, Relational databases, Top
@chapter Binary files
@cindex Binary files

@menu
* Package Rstreams::            
* Binary data formats::         
@end menu

Binary connections (@ref{Connections}) are now the preferred way to
handle binary files, and package @pkg{Rstreams} will be withdrawn in
late 2001.


@node Package Rstreams, Binary data formats, Binary files, Binary files
@section Package Rstreams

Package @pkg{Rstreams} on @acronym{CRAN} provides a low-level interface
to read from and write to binary files.  It has been used to read in
data from MRI experiments and Windows sound files, for example.

@findex openstream
@findex closestream
@pkg{Rstreams}' view of the file is as a stream of bytes.  A file can be
opened by the function @code{openstream} for either reading or writing.
The return value is a number (in fact the file descriptor) that is used
to reference the open file until it is closed by @code{closestream}.

@findex readint
@findex readfloat
@findex readcomplex
@findex readchar
Once a stream is open for reading, bytes can be transferred from it to
an @R{} object by one of the functions

@example
readint(stream, n, size = 4, signed = TRUE, swapbytes = FALSE)
readfloat(stream, n, size = 8, swapbytes = FALSE)
readcomplex(stream, n, size = 8, swapbytes = FALSE)
readchar(stream, n = 1, len = NA, bufsize = 256)
@end example

@noindent
These return an @R{} object of an appropriate mode and storage mode.
(Integers too large to be represented in storage mode @code{"integer"}
will be read into a vector of storage mode @code{"double"}.)  Here
@code{n} is the number of items to be read, but fewer will be read if
there is insufficient data on the file.

The size of the data components on file need not be the same as that in
the machine running @R{}, and data written on a little-endian machine
can be read on a big-endian machine or @emph{vice versa} by setting
@code{swapbytes = TRUE}.

Character data can be read by @code{readchar} either as fixed-length
blocks of bytes (by specifying @code{len}) or as
@acronym{ASCII}-zero-delimited strings.  It is also possible to read
lines of text separated by one of the standard end-of-line marks
(LF, CRLF or CR) by
@findex readlines
@example
readlines(stream, n = 1, bufsize = 256, eol)
@end example
@noindent
where by default the @code{eol} mark is detected automatically.


@findex writeint
@findex writefloat
@findex writecomplex
@findex writechar
A file open for writing can be written to by any of

@example
writeint(stream, data, size = 4, swapbytes = FALSE)
writefloat(stream, data, size = 8, swapbytes = FALSE)
writecomplex(stream, data, size = 8, swapbytes = FALSE)
writechar(stream, data, asciiz = FALSE)
@end example

@noindent
where @code{data} is an @R{} vector of an appropriate mode.  There is no
separator between character strings on the file unless @code{asciiz =
TRUE} is specified.

@findex copystream
Function @code{copystream} can copy a specified number of bytes from one
open stream to another.

@findex truncate
Function @code{truncate} truncates a stream at its current position, so
all data after that point is lost.


@node Binary data formats,  , Package Rstreams, Binary files
@section Binary data formats
@c @findex hdf5
@c @cindex Hierarchical Data Format

@findex netCDF
@cindex network Common Data Form

@c  Packages @pkg{hdf5} (in the Devel area, and not currently updated for
@c  current @R{} versions) and @pkg{netCDF} on @acronym{CRAN} provide
@c  experimental interfaces to @acronym{NASA}'s HDF5 (Hierarchical Data
@c  Format, see @uref{http://hdf.ncsa.uiuc.edu/HDF5/}) and to UCAR's netCDF
@c  data files (network Common Data Form, see
@c  @uref{http://www.unidata.ucar.edu/packages/netcdf/}), respectively.

@c  Both of these are systems to store scientific data in array-oriented
@c  way, including descriptions, labels, formats, units, @dots{}.  HDF5 also
@c  allows @emph{groups} of arrays, and the @R{} interface writes maps lists
@c  to HDF5 groups, and can write numeric and character vectors and
@c  matrices.

@c  The @R{} interface can only read netCDF, not write it (yet).

Package @pkg{netCDF} on @acronym{CRAN} provides an experimental
interface to UCAR's netCDF data files (network Common Data Form, see
@uref{http://www.unidata.ucar.edu/packages/netcdf/}).  This is a system
to store scientific data in array-oriented way, including descriptions,
labels, formats, units, @dots{}.  The @R{} interface can only read
netCDF, not write it (yet).

@node Connections, Network interfaces, Binary files, Top
@chapter Connections

@cindex Connections
@emph{Connections} are used in @R{} in the sense of Chambers (1998), a
set of functions to replace the use of file names by a flexible
interface to file-like objects.  Connections were introduced to @R{} in
version 1.2.0.


@menu
* Types of connections::        
* Output to connections::       
* Input from connections::      
* Listing and manipulating connections::  
* Binary connections::          
@end menu


@node Types of connections, Output to connections, Connections, Connections
@section Types of connections
@cindex Connections

@findex file
@cindex File connections
The most familiar type of connection will be a file, and file
connections are created by function @code{file}.  File connections can
(if the OS will allow it for the particular file) be opened for reading
or writing or appending, in text or binary mode.  In fact, files can be
opened for both reading and writing, and @R{} keeps a separate file
position for reading and writing.

@findex open
@findex close
Note that by default a connection is not opened when it is created.  The
rule is that a function using a connection should open a connection
(needed) if the connection is not already open, and close a connection
after use if it opened it.  In brief, leave the connection in the state
you found it in.   There are generic functions @code{open} and
@code{close} with methods to explicitly open and close connections.

@findex gzfile
@cindex Compressed files
Files compressed via the algorithm used by @code{gzip} can be used as
connections created by the function @code{gzfile}.

@cindex Terminal connections
@findex stdin
@findex stdout
@findex stderr
Unix programmers are used to dealing with special files @code{stdin},
@code{stdout} and @code{stderr}.   These exist as @emph{terminal
connections} in @R{}.  They may be normal files, but they might also
refer to input from and output to a GUI console.  (Even with the standard
Unix @R{} interface, @code{stdin} refers to the lines submitted from
@code{readline} rather than a file.)

The three terminal connections are always open, and cannot be opened or
closed.  @code{stdout} and @code{stderr} are conventionally used for
normal output and error messages respectively.  They may normally go to
the same place, but whereas normal output can be re-directed by a call
to @code{sink}, error output is sent to @code{stderr} unless re-directed
by @code{sink, type="message")}.  Note carefully the language used here:
the connections cannot be re-directed, but output can be sent to other
connections.

@cindex Text connections
@findex textConnection
@emph{Text connections} are another source of input.  They allow @R{}
character vectors to be read as if the lines were being read from a text
file.  A text connection is created and opened by a call to
@code{textConnection}, which copies the current contents of the
character vector to an internal buffer at the time of creation.

Text connections can also be used to capture @R{} output to a character
vector.  @code{textConnection} can be asked to create a new character
object or append to an existing one, in both cases in the user's
workspace.  The connection is opened by the call to
@code{textConnection}, and at all times the complete lines output to the
connection are available in the @R{} object.  Closing the connection
writes any remaining output to a final element of the character vector.

@cindex Pipe connections
@findex pipe
@emph{Pipes} are a special form of file that connects to another
process, and pipe connections are created by the function @code{pipe}
(currently implemented on Unix and @code{Rterm} only).  Opening a pipe
connection for writing (it makes no sense to append to a pipe) runs an
OS command, and connects its standard input to whatever @R{} then writes
to that connection.  Conversely, opening a pipe connection for input
runs an OS command and makes its standard output available for @R{}
input from that connection.

@cindex URL connections
@findex url
URLs of types @code{http://}, @code{ftp://} and @code{file://} can be
read from using the function @code{url}.  For convenience, @code{file}
will also accept these as the file specification and call @code{url}.

@cindex Sockets
@findex socketConnection
Sockets can also be used as connections via function
@code{socketConnection} on platforms which support Berkeley-like sockets
(most Unix systems, Linux and Windows but not currently classic
Macintosh).  Sockets can be written to or read from, and both client and
server sockets can be used.


@node Output to connections, Input from connections, Types of connections, Connections
@section Output to connections
@cindex Connections

@findex cat
@findex write
@findex write.table
@findex sink
We have described functions @code{cat}, @code{write}, @code{write.table}
and @code{sink} as writing to a file, possibly appending to a file if 
argument @code{append = TRUE}, and this is what they did prior to @R{}
version 1.2.0.

The current behaviour is equivalent, but what actually happens is that
when the @code{file} argument is a character string, a file connection
is opened (for writing or appending) and closed again at the end of the
function call.  If we want to repeatedly write to the same file, it is
more efficient to explicitly declare and open the connection, and pass
the connection object to each call to an output function.  This also
makes it possible to write to pipes, which was implemented earlier in a
limited way via the syntax @code{file = "|cmd"} (which can still be
used).

@findex writeLines
There is a function @code{writeLines} to write complete text lines
to a connection.

Some simple examples are

@smallexample
zz <- file("ex.data", "w")  # open an output file connection
cat("TITLE extra line", "2 3 5 7", "", "11 13 17", 
    file = zz, sep = "\n")
cat("One more line\n", file = zz)
close(zz)

## convert decimal point to comma in output, using a pipe (Unix)
zz <- pipe(paste("sed s/\\./,/ >", "outfile"), "w")
cat(format(round(rnorm(100), 4)), sep = "\n", file = zz)
close(zz)
## now look at the output file:
file.show(outfile, delete.file = TRUE)

## capture R output: use examples from help(lm)
zz <- textConnection("ex.lm.out", "w")
sink(zz)
example(lm, prompt.echo = "> ")
sink()
close(zz)
## now `ex.lm.out' contains the output for futher processing.
## Look at it by, e.g.,
cat(ex.lm.out, sep = "\n")
@end smallexample

@node Input from connections, Listing and manipulating connections, Output to connections, Connections
@section Input from connections

@findex scan
@findex read.table
@findex readLines
The basic functions to read from connections are @code{scan} and
@code{readLines}.  These take a character string argument and open a
file connection for the duration of the function call, but explicitly
opening a file connection allows a file to be read sequentially in
different formats.

Other functions that call @code{scan} can also make use of connections,
in particular @code{read.table}.  As from @R{} version 1.3.0, 
@code{read.table} reads the data in a single pass and so works better
with non-file connections.

Some simple examples are

@smallexample
## read in file created in last examples
readLines("ex.data")
unlink("ex.data")

## read listing of current directory (Unix)
readLines(pipe("ls -1"))

# remove trailing commas from an input file.
# Suppose we are given a file `data' containing
450, 390, 467, 654,  30, 542, 334, 432, 421,
357, 497, 493, 550, 549, 467, 575, 578, 342,
446, 547, 534, 495, 979, 479
# Then read this by
scan(pipe("sed -e s/,$// data"), sep=",")
@end smallexample

@cindex URL connections
For convenience, if the @code{file} argument specifies a FTP or HTTP
URL, the URL is opened for reading via @code{url}.  Specifying
files via @code{file://foo.bar} is also allowed.

@menu
* Pushback::                    
@end menu

@node Pushback,  , Input from connections, Input from connections
@subsection Pushback

@findex pushBack.
@cindex Pushback on a connection
C programmers may be familiar with the @code{ungetc} function to push
back a character onto a text input stream.  @R{} connections have the
same idea in a more powerful way, in that an (essentially) arbitrary
number of lines of text can be pushed back onto a connection via a call
to @code{pushBack}.

Pushbacks operate as a stack, so a read request first uses each line
from the most recently pushbacked text, then those from earlier
pushbacks and finally reads from the connection itself.  Once a
pushbacked line is read completely, it is cleared.  The number of
pending lines pushed back can be found via a call to
@code{pushBackLength}.
@findex pushBackLength

A simple example will show the idea.

@smallexample
> zz <- textConnection(LETTERS)
> readLines(zz, 2)
[1] "A" "B"
> scan(zz, "", 4)
Read 4 items
[1] "C" "D" "E" "F"
> pushBack(c("aa", "bb"), zz)
> scan(zz, "", 4)
Read 4 items
[1] "aa" "bb" "G"  "H" 
> close(zz)
@end smallexample

Pushback is only available for connections opened for input in text mode.

@node Listing and manipulating connections, Binary connections, Input from connections, Connections
@section Listing and manipulating connections
@cindex Connections

@findex showConnections
A summary of all the connections currently opened by the user can be
found by @code{showConnections()}, and a summary of all connections,
including closed and terminal connections, by @code{showConnections(all
= TRUE)}

@findex seek
@findex isSeekable
The generic function @code{seek} can be used to read and (on some
connections) reset the current position for reading or writing.
Unfortunately it depends on OS facilities which may be unreliable
(e.g. with text files under Windows).  Function @code{isSeekable}
reports if @code{seek} can change the position on the connection
given by its argument.

@findex truncate
The function @code{truncate} can be used to truncate a file opened for
writing at its current position.  It works only for @code{file}
connections, and is not implemented on all platforms.


@node Binary connections,  , Listing and manipulating connections, Connections
@section Binary connections
@cindex Binary files

@findex readBin
@findex writeBin
@R{} version 1.2.1 added functions @code{readBin} and @code{writeBin} to
read and write from binary connections.  A connection is opened in
binary mode by appending @code{"b"} to the mode specification, that is
using mode @code{"rb"} for reading, and mode @code{"wb"} or @code{"ab"}
(where appropriate) for writing.  The functions have arguments

@smallexample
readBin(con, what, n = 1, size = NA, endian = .Platform$endian) 
writeBin(object, con, size = NA, endian = .Platform$endian) 
@end smallexample

In each case @code{con} is a connection which will be opened if
necessary for the duration of the call, and if a character string is
given it is assumed to specify a file name.

It is slightly simpler to describe writing, so we will do that first.
@code{object} should be an atomic vector object, that is a vector of
mode @code{numeric}, @code{integer}, @code{logical}, @code{character} or
@code{complex}, without attributes.  By default this is written to the
file as a stream of bytes exactly as it is represented in memory.

@code{readBin} reads a stream of bytes from the file and interprets them
as a vector of mode given by @code{what}.  This can be either an object
of the appropriate mode (e.g. @code{what=integer()}) or a character
string describing the mode (one of the five given in the previous
paragraph or @code{"double"} or @code{"int"}).  @code{size} specifies
the maximum number of vector elements to read from the connection: if
fewer are available a shorter vector will be returned.

The remaining two arguments are used to write or read data for
interchange with another program or another platform.  By default binary
data is transferred directly from memory to the connection or @emph{vice
versa}.  This will not suffice if the file is to be transferred to a
machine with a different architecture, but between almost all @R{}
platforms the only change needed is that of byte-order.  Intel
@code{ix86}-based machines, Compaq Alpha and Vaxen are
@emph{little-endian}, whereas Sun Sparc, mc680x0 series, IBM R6000, SGI
and most others are @emph{big-endian}.  (Network byte-order (as used by
XDR, eXternal Data Representation) is big-endian.)  To transfer to or
from other programs we may need to do more, for example to read 16-bit
integers or write single-precision real numbers.  This can be done using
the @code{size} argument, which (usually) allows sizes 1, 2, 4, 8 for
integers and logicals, and sizes 4, 8 and perhaps 12 or 16 for reals.
Transferring at different sizes can lose precision, and should not be
attempted for vectors containing @code{NA}'s.

@findex readChar
@findex writeChar
Character strings are read and written in C format, that is as a string
of bytes terminated by a zero byte.  Functions @code{readChar} and
@code{writeChar} (added in 1.3.0) provide greater flexibility.

@node Network interfaces, References, Connections, Top
@chapter Network interfaces

@menu
* Reading from sockets::        
* Using download.file::         
* DCOM interface::              
* CORBA interface::             
@end menu

Some limited facilities are available to exchange data at a lower level
across network connections.

@node Reading from sockets, Using download.file, Network interfaces, Network interfaces
@section Reading from sockets

@cindex Sockets
Base @R{} comes with some facilities to communicate @emph{via}
@acronym{BSD} sockets on systems that support them (including the common
Linux, Unix and Windows ports of @R{}).  One potential problem with
using sockets is that these facilities are often blocked for security
reasons or to force the use of Web caches, so these functions may be
more useful on an intranet than externally.  For new projects it
is suggested that socket connections are used instead.

@findex make.socket
@findex read.socket
@findex write.socket
@findex close.socket
The earlier low-level interface is given by functions @code{make.socket},
@code{read.socket}, @code{write.socket} and @code{close.socket}.


@node Using download.file, DCOM interface, Reading from sockets, Network interfaces
@section Using @code{download.file}

Function @code{download.file} is provided to read a file from a
Web resource via FTP or HTTP and write it to a file.  Often this can be
avoided, as functions such as @code{read.table} and @code{scan} can read
directly from a URL, either by explicitly using @code{url} to open a
connection, or implicitly using it by giving a URL as the @code{file}
argument.


@node DCOM interface, CORBA interface, Using download.file, Network interfaces
@section @acronym{DCOM} interface
@cindex DCOM

@acronym{DCOM} is a Windows protocol for communicating between different
programs, possibly on different machines.  Thomas Baier's
@code{StatConnector} program available from @acronym{CRAN} under
Software->Other->Non-standard provides an interface to the proxy DLL
which ships with the Windows version of @R{} and makes an @acronym{DCOM}
server.  This can be used to pass simple objects (vectors and matrices)
to and from @R{} and to submit commands to @R{}.

The program comes with a Visual Basic demonstration, and there is an
Excel plug-in by Erich Neuwirth available in the same area on
@acronym{CRAN}.  This interface is in the other direction to most of
those considered here in that it is another application (Excel, or
written in Visual Basic) that is the client and @R{} is the server.

@node CORBA interface,  , DCOM interface, Network interfaces
@section @acronym{CORBA} interface
@cindex CORBA

@acronym{CORBA} (Common Object Request Broker Architecture) is similar
to @acronym{DCOM}, allowing applications to call methods, or operations,
in server objects running in other applications, potentially programmed
in different languages and running on different machines.  There is a
@pkg{CORBA} package available from the Omegahat Project (at
@uref{http://www.omegahat.org/RSCORBA/}), currently for Unix but a
Windows version looks to be possible.

This package allows @R{} commands to be used to locate available
@acronym{CORBA} servers, query the methods they provide, and dynamically
invoke methods on these objects.  @R{} values given as arguments in
these calls are exported in the call and made available to that
operation invocation.  Primitive data types (vectors and lists) are
exported by default, while more complex objects are exported by
reference.  Examples of using this include communicating with the
Gnumeric (@uref{http://www.gnumeric.org}) spreadsheet, and also
interacting with the data visualization system
@uref{http://www.ggobi.org, ggobi}.

One can also create @acronym{CORBA} servers within @R{}, allowing other
applications to call these methods.  For example, one might offer access
to a particular dataset or to some of @R{}'s modelling software.  This
is done dynamically by combining @R{} data objects and functions.  This
allows one to explicitly export data and functionality from @R{}.

One can also use the @pkg{CORBA} package to achieve distributed,
parallel computing in @R{}.  One @R{} session acts as a manager and
dispatches tasks to different servers running in other @R{} worker
sessions.  This uses the ability to invoke asynchronous or background
CORBA calls in @R{}.  More information is available from the Omegahat Project,
at @uref{http://www.omegahat.org/RSCORBA/}.



@node References, Function and variable index, Network interfaces, Top
@appendix References

@noindent
R.@: A.@: Becker, J.@: M.@: Chambers and A.@: R.@: Wilks (1988)
@emph{The New S Language.  A Programming Environment for Data Analysis
and Graphics.}  Wadsworth & Brooks/Cole.

@noindent
J.@: Bowman, S.@: Emberson and M.@: Darnovsky (1996) @emph{The
Practical @acronym{SQL} Handbook.  Using Structured Query Language.}
Addison-Wesley.

@noindent
J.@: M.@: Chambers (1998) @emph{Programming with Data.  A Guide to the S
Language.} Springer-Verlag.

@noindent
P.@: Dubois (2000) @emph{MySQL.} New Riders.

@noindent
M.@: Henning and S.@: Vinoski (1999) @emph{Advanced CORBA Programming
with C++.} Addison-Wesley.

@noindent
K.@: Kline and D.@: Kline (2001) @emph{SQL in a Nutshell.} O'Reilly.

@noindent
B.@: Momjian (2000) @emph{PostgreSQL: Introduction and Concepts.}
Addison-Wesley.
Also downloadable at @uref{http://www.postgresql.org/docs/awbook.html}.

@noindent
T.@: M.@: Therneau and P.@: M.@: Grambsch (2000) @emph{Modeling Survival
Data.  Extending the Cox Model.} Springer-Verlag.

@noindent
E.@: J.@: Yarger, G.@: Reese and T.@ King (1999) @emph{MySQL & mSQL}.
O'Reilly.

@node Function and variable index, Concept index, References, Top
@unnumbered Function and variable index

@printindex vr

@node Concept index,  , Function and variable index, Top
@unnumbered Concept index

@printindex cp


@bye

@c Local Variables: ***
@c mode: TeXinfo ***
@c End: ***