-
Notifications
You must be signed in to change notification settings - Fork 315
/
R-data.texi
2193 lines (1791 loc) · 83.1 KB
/
R-data.texi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\input texinfo
@c %**start of header
@setfilename R-data.info
@settitle R Data Import/Export
@setchapternewpage on
@c %**end of header
@syncodeindex fn vr
@dircategory Programming
@direntry
* R Data: (R-data). R Data Import/Export.
@end direntry
@finalout
@include R-defs.texi
@include version.texi
@ifinfo
This is a guide to importing and exporting data to and from R.
@Rcopyright{2000}
@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore
@permission{}
@c ---------- ^- read that
@end ifinfo
@titlepage
@title R Data Import/Export
@subtitle Version @value{VERSION}
@author R Development Core Team
@page
@vskip 0pt plus 1filll
@permission{}
@Rcopyright{2000}
@value{ISBN-data}
@end titlepage
@ifnothtml
@contents
@end ifnothtml
@ifnottex
@node Top, Acknowledgements, (dir), (dir)
@top R Data Import/Export
This is a guide to importing and exporting data to and from R.
The current version of this document is @value{VERSION}.
@value{ISBN-data}
@end ifnottex
@menu
* Acknowledgements::
* Introduction::
* Spreadsheet-like data::
* Importing from other statistical systems::
* Relational databases::
* Binary files::
* Connections::
* Network interfaces::
* Reading Excel spreadsheets::
* References::
* Function and variable index::
* Concept index::
@end menu
@node Acknowledgements, Introduction, Top, Top
@unnumbered Acknowledgements
The relational databases part of this manual is based in part on an
earlier manual by Douglas Bates and Saikat DebRoy. The principal author
of this manual was Brian Ripley.
Many volunteers have contributed to the packages used here. The
principal authors of the packages mentioned are
@quotation
@multitable {foreign xxxx} {A longggggggggggggggggggggggggggggggggg description}
@item @pkg{CORBA} @tab Duncan Temple Lang
@item @pkg{foreign} @tab Thomas Lumley, Saikat DebRoy, Douglas Bates, Duncan Murdoch and Roger Bivand
@item @pkg{hdf5} @tab Marcus Daniels
@item @pkg{ncdf} @tab David Pierce
@item @pkg{ncvar} @tab Juerg Schmidli
@item @pkg{rJava} @tab Simon Urbanek
@item @pkg{RMySQL} @tab David James and Saikat DebRoy
@item @pkg{RNetCDF} @tab Pavel Michna
@item @pkg{RODBC} @tab Michael Lapsley and Brian Ripley
@item @pkg{RSPerl} @tab Duncan Temple Lang
@item @pkg{RSPython} @tab Duncan Temple Lang
@item @pkg{SJava} @tab John Chambers and Duncan Temple Lang
@item @pkg{XML} @tab Duncan Temple Lang
@end multitable
@end quotation
Brian Ripley is the author of the support for connections.
@node Introduction, Spreadsheet-like data, Acknowledgements, Top
@chapter Introduction
Reading data into a statistical system for analysis and exporting the
results to some other system for report writing can be frustrating tasks
that can take far more time than the statistical analysis itself, even
though most readers will find the latter far more appealing.
This manual describes the import and export facilities available either
in @R{} itself or via packages which are available from @acronym{CRAN}.
Some of the packages described are still under development but they
already provide useful functionality.
Unless otherwise stated, everything described in this manual is
available on all platforms running @R{}.
In general, statistical systems like @R{} are not particularly well
suited to manipulations of large-scale data. Some other systems are
better than @R{} at this, and part of the thrust of this manual is to
suggest that rather than duplicating functionality in @R{} we can make
another system do the work! (For example Therneau & Grambsch (2000)
comment that they prefer to do data manipulation in SAS and then use
@pkg{survival} in @Sl{} for the analysis.) Several recent packages
allow functionality developed in languages such as @code{Java},
@code{perl} and @code{python} to be directly integrated with @R{} code,
making the use of facilities in these languages even more
appropriate. (See the @pkg{SJava}, @pkg{RSPerl} and @pkg{RSPython}
packages from the Omegahat project, @url{http://www.omegahat.org}, and
the @pkg{rJava} package from @acronym{CRAN}.)
@cindex Unix tools
@cindex AWK
@cindex perl
It is also worth remembering that @R{} like @Sl{} comes from the Unix
tradition of small re-usable tools, and it can be rewarding to use tools
such as @code{awk} and @code{perl} to manipulate data before import or
after export. The case study in Becker, Chambers & Wilks (1988, Chapter
9) is an example of this, where Unix tools were used to check and
manipulate the data before input to @Sl{}. @R{} itself takes that
approach, using @code{perl} to manipulate its databases of help files
rather than @R{} itself, and the function @code{read.fwf} used a call to
a @code{perl} script until it was decided not to require @code{perl} at
run-time. The traditional Unix tools are now much more widely
available, including on Windows.
@menu
* Imports::
* Export to text files::
* XML::
@end menu
@node Imports, Export to text files, Introduction, Introduction
@section Imports
@findex scan
The easiest form of data to import into @R{} is a simple text file, and
this will often be acceptable for problems of small or medium scale.
The primary function to import from a text file is @code{scan}, and this
underlies most of the more convenient functions discussed in
@ref{Spreadsheet-like data}.
However, all statistical consultants are familiar with being presented
by a client with a floppy disc or CD-R of data in some proprietary
binary format, for example `an Excel spreadsheet' or `an SPSS file'.
Often the simplest thing to do is to use the originating application to
export the data as a text file (and statistical consultants will have
copies of the most common applications on their computers for that
purpose). However, this is not always possible, and @ref{Importing from
other statistical systems} discusses what facilities are available to
access such files directly from @R{}. For Excel spreadsheets, the
available methods are summarized in @ref{Reading Excel spreadsheets}.
In a few cases, data have been stored in a binary form for compactness
and speed of access. One application of this that we have seen several
times is imaging data, which is normally stored as a stream of bytes as
represented in memory, possibly preceded by a header. Such data formats
are discussed in @ref{Binary files} and @ref{Binary connections}.
For much larger databases it is common to handle the data using a
database management system (DBMS). There is once again the option of
using the DBMS to extract a plain file, but for many such DBMSs the
extraction operation can be done directly from an @R{} package:
@xref{Relational databases}. Importing data via network connections is
discussed in @ref{Network interfaces}.
@node Export to text files, XML, Imports, Introduction
@section Export to text files
@cindex Exporting to a text file
Exporting results from @R{} is usually a less contentious task, but
there are still a number of pitfalls. There will be a target
application in mind, and normally a text file will be the most
convenient interchange vehicle. (If a binary file is required, see
@ref{Binary files}.)
@findex cat
Function @code{cat} underlies the functions for exporting data. It
takes a @code{file} argument, and the @code{append} argument allows a
text file to be written via successive calls to @code{cat}. Better,
especially if this is to be done many times, is to open a @code{file}
connection for writing or appending, and @code{cat} to that connection,
then @code{close} it.
@findex write
@findex write.table
The most common task is to write a matrix or data frame to file as a
rectangular grid of numbers, possibly with row and column labels. This
can be done by the functions @code{write.table} and @code{write}.
Function @code{write} just writes out a matrix or vector in a specified
number of columns (and transposes a matrix). Function
@code{write.table} is more convenient, and writes out a data frame (or
an object that can be coerced to a data frame) with row and column
labels.
There are a number of issues that need to be considered in writing out a
data frame to a text file.
@enumerate
@findex format
@item @strong{Precision}
Most of the conversions of real/complex numbers done by these functions
is to full precision, but those by @code{write} are governed by the
current setting of @code{options(digits)}. For more control, use
@code{format} on a data frame, possibly column-by-column.
@item @strong{Header line}
@R{} prefers the header line to have no entry for the row names, so the
file looks like
@example
dist climb time
Greenmantle 2.5 650 16.083
...
@end example
@noindent
Some other systems require a (possibly empty) entry for the row names, which
is what @code{write.table} will provide if argument @code{col.names = NA}
is specified. Excel is one such system.
@item @strong{Separator}
@cindex CSV files
@cindex comma separated values
@findex write.csv
@findex write.csv2
A common field separator to use in the file is a comma, as that is
unlikely to appear in any of the fields, in English-speaking countries.
Such files are known as CSV (comma separated values) files, and wrapper
function @code{write.csv} provides appropriate defaults. In some locales
the comma is used as the decimal point (set this in @code{write.table}
by @code{dec = ","}) and there CSV files use the semicolon as the field
separator: use @code{write.csv2} for appropriate defaults.
Using a semicolon or tab (@code{sep = "\t"}) are probably the safest
options.
@item @strong{Missing values}
@cindex Missing values
By default missing values are output as @code{NA}, but this may be
changed by argument @code{na}. Note that @code{NaN}s are treated as
@code{NA} by @code{write.table}, but not by @code{cat} nor @code{write}.
@item @strong{Quoting strings}
@cindex Quoting strings
By default strings are quoted (including the row and column names).
Argument @code{quote} controls quoting of character and factor variables.
Some care is needed if the strings contain embedded quotes. Three
useful forms are
@example
> df <- data.frame(a = I("a \" quote"))
> write.table(df)
"a"
"1" "a \" quote"
> write.table(df, qmethod = "double")
"a"
"1" "a "" quote"
> write.table(df, quote = FALSE, sep = ",")
a
1,a " quote
@end example
@noindent
The second is the form of escape commonly used by spreadsheets.
@end enumerate
@findex write.matrix
Function @code{write.matrix} in package @pkg{MASS} provides a
specialized interface for writing matrices, with the option of writing
them in blocks and thereby reducing memory usage.
@findex sink
It is possible to use @code{sink} to divert the standard @R{} output to
a file, and thereby capture the output of (possibly implicit)
@code{print} statements. This is not usually the most efficient route,
and the @code{options(width)} setting may need to be increased.
@findex write.foreign
Function @code{write.foreign} in package @pkg{foreign} uses
@code{write.table} to produce a text file and also writes a code file
that will read this text file into another statistical package. There is
currently support for export to @code{SPSS} and @code{Stata}.
@node XML, , Export to text files, Introduction
@section XML
@cindex XML
When reading data from text files, it is the responsibility of the user
to know and to specify the conventions used to create that file,
e.g. the comment character, whether a header line is present, the value
separator, the representation for missing values (and so on) described
in @ref{Export to text files}. A markup language which can be used to
describe not only content but also the structure of the content can
make a file self-describing, so that one need not provide these details
to the software reading the data.
The eXtensible Markup Language -- more commonly known simply as
@acronym{XML} -- can be used to provide such structure, not only for
standard datasets but also more complex data structures.
@acronym{XML} is becoming extremely popular and is emerging as a
standard for general data markup and exchange. It is being used by
different communities to describe geographical data such as maps,
graphical displays, mathematics and so on.
The @pkg{XML} package provides general facilities for reading and
writing @acronym{XML} documents within both @R{} and S-PLUS in the hope
that we can easily make use of this technology as it evolves. Several
people are exploring how we can use @acronym{XML} for, amongst other
things, representing datasets to be shared across different
applications; storing @R{} and S-PLUS objects so they can be shared by
both systems; representing plots via @acronym{SVG} (Scalable Vector
Graphics, a dialect of @acronym{XML}); representing function
documentation; generating ``live'' analyses/reports that contain text,
data and code.
A description of the facilities of the @pkg{XML} package is outside the
scope of this document: see the package's Web page at
@url{http://www.omegahat.org/RSXML} for details and examples. Package
@pkg{StatDataML} on CRAN is one example building on @pkg{XML}.
@node Spreadsheet-like data, Importing from other statistical systems, Introduction, Top
@chapter Spreadsheet-like data
@cindex Spreadsheet-like data
@menu
* Variations on read.table::
* Fixed-width-format files::
* Data Interchange Format (DIF)::
* Using scan directly::
* Re-shaping data::
* Flat contingency tables::
@end menu
In @ref{Export to text files} we saw a number of variations on the
format of a spreadsheet-like text file, in which the data are presented
in a rectangular grid, possibly with row and column labels. In this
section we consider importing such files into @R{}.
@node Variations on read.table, Fixed-width-format files, Spreadsheet-like data, Spreadsheet-like data
@section Variations on @code{read.table}
@findex read.table
The function @code{read.table} is the most convenient way to read in a
rectangular grid of data. Because of the many possibilities, there are
several other functions that call @code{read.table} but change a group
of default arguments.
Beware that @code{read.table} is an inefficient way to read in
very large numerical matrices: see @code{scan} below.
Some of the issues to consider are:
@enumerate
@item @strong{Encoding}
If the file contains non-ASCII character fields, ensure that it is read
in the correct encoding. This is mainly an issue for reading Latin-1
files in a UTF-8 locale, which can be done by something like
@example
read.table(file("file.dat", encoding="latin1"))
@end example
@noindent
Note that this will work in any locale which can represent Latin-1 names.
@item @strong{Header line}
We recommend that you specify the @code{header} argument explicitly,
Conventionally the header line has entries only for the columns and not
for the row labels, so is one field shorter than the remaining lines.
(If @R{} sees this, it sets @code{header = TRUE}.) If presented with a
file that has a (possibly empty) header field for the row labels, read
it in by something like
@example
read.table("file.dat", header = TRUE, row.names = 1)
@end example
Column names can be given explicitly via the @code{col.names}; explicit
names override the header line (if present).
@item @strong{Separator}
Normally looking at the file will determine the field separator to be
used, but with white-space separated files there may be a choice between
the default @code{sep = ""} which uses any white space (spaces, tabs or
newlines) as a separator, @code{sep = " "} and @code{sep = "\t"}. Note
that the choice of separator affects the input of quoted strings.
If you have a tab-delimited file containing empty fields be sure to use
@code{sep = "\t"}.
@item @strong{Quoting}
@cindex Quoting strings
By default character strings can be quoted by either @samp{"} or
@samp{'}, and in each case all the characters up to a matching quote are
taken as part of the character string. The set of valid quoting
characters (which might be none) is controlled by the @code{quote}
argument. For @code{sep = "\n"} the default is changed to @code{quote =
""}.
If no separator character is specified, quotes can be escaped within
quoted strings by immediately preceding them by @samp{\}, C-style.
If a separator character is specified, quotes can be escaped within
quoted strings by doubling them as is conventional in spreadsheets. For
example
@example
'One string isn''t two',"one more"
@end example
@noindent
can be read by
@example
read.table("testfile", sep = ",")
@end example
@noindent
This does not work with the default separator.
@item @strong{Missing values}
@cindex Missing values
By default the file is assumed to contain the character string @code{NA}
to represent missing values, but this can be changed by the argument
@code{na.strings}, which is a vector of one or more character
representations of missing values.
Empty fields in numeric columns are also regarded as missing values.
In numeric columns, the values @code{NaN}, @code{Inf} and @code{-Inf} are
accepted.
@item @strong{Unfilled lines}
It is quite common for a file exported from a spreadsheet to have all
trailing empty fields (and their separators) omitted. To read such
files set @code{fill = TRUE}.
@item @strong{White space in character fields}
If a separator is specified, leading and trailing white space in
character fields is regarded as part of the field. To strip the space,
use argument @code{strip.white = TRUE}.
@item @strong{Blank lines}
By default, @code{read.table} ignores empty lines. This can be changed
by setting @code{blank.lines.skip = FALSE}, which will only be useful in
conjunction with @code{fill = TRUE}, perhaps to use blank rows to
indicate missing cases in a regular layout.
@item @strong{Classes for the variables}
Unless you take any special action, @code{read.table} tries to select a
suitable class for each variable in the data frame. It tries in turn
@code{logical}, @code{integer}, @code{numeric} and @code{complex},
moving on if any entry is not missing and cannot be
converted.@footnote{This is normally fast as looking at the first entry
rules out most of the possibilities.} If all of these fail, the
variable is converted to a factor.
Arguments @code{colClasses} and @code{as.is} provide greater control.
@code{as.is} suppresses conversion of character vectors to factors
(only). Using @code{colClasses} allows the desired class to be set for
each column in the input.
Note that @code{colClasses} and @code{as.is} are specified @emph{per}
column, not @emph{per} variable, and so include the column of row names
(if any).
@item @strong{Comments}
By default, @code{read.table} uses @samp{#} as a comment character,
and if this is encountered (except in quoted strings) the rest of the
line is ignored. Lines containing only white space and a comment are
treated as blank lines.
If it is known that there will be no comments in the data file, it is
safer (and may be faster) to use @code{comment.char = ""}.
@item @strong{Escapes}
Many OSes have conventions for using backslash as an escape character in
text files, but Windows does not (and uses backslash in path names).
It is optional in @R{} whether such conventions are applied to data files.
Both @code{read.table} and @code{scan} have a logical argument
@code{allowEscapes}. As from @R{} 2.2.0 this is false by default, and
backslashes are then only interpreted as (under circumstances described
above) escaping quotes. If this set to be true, C-style escapes are
interpreted, namely the control characters @code{\a, \b, \f, \n, \r, \t,
\v} and octal and hexadecimal representations like @code{\040} and
@code{\0x2A}. Any other escaped character is treated as itself,
including backslash.
@end enumerate
@findex read.csv
@findex read.csv2
@findex read.delim
@findex read.delim2
@cindex CSV files
@findex Sys.localeconv
@cindex locales
Convenience functions @code{read.csv} and @code{read.delim} provide
arguments to @code{read.table} appropriate for CSV and tab-delimited
files exported from spreadsheets in English-speaking locales. The
variations @code{read.csv2} and @code{read.delim2} are appropriate for
use in countries where the comma is used for the decimal point.
@c Currently R overrides the setting of LC_NUMERIC
@c You can find the current settings of the computer in use by calling
@c @code{Sys.localeconv()}.
If the options to @code{read.table} are specified incorrectly, the error
message will usually be of the form
@example
Error in scan(file = file, what = what, sep = sep, :
line 1 did not have 5 elements
@end example
@noindent
or
@example
Error in read.table("files.dat", header = TRUE) :
more columns than column names
@end example
@findex count.fields
@noindent
This may give enough information to find the problem, but the auxiliary
function @code{count.fields} can be useful to investigate further.
Efficiency can be important when reading large data grids. It will help
to specify @code{comment.char = ""}, @code{colClasses} as one of the
atomic vector types (logical, integer, numeric, complex, character or
perhaps raw) for each column, and to give @code{nrows}, the number of
rows to be read (and a mild over-estimate is better than not specifying
this at all). See the examples below.
@node Fixed-width-format files, Data Interchange Format (DIF), Variations on read.table, Spreadsheet-like data
@section Fixed-width-format files
@cindex Fixed-width-format files
Sometimes data files have no field delimiters but have fields in
pre-specified columns. This was very common in the days of punched
cards, and is still sometimes used to save file space.
@findex read.fwf
Function @code{read.fwf} provides a simple way to read such files,
specifying a vector of field widths. The function reads the file into
memory as whole lines, splits the resulting character strings, writes
out a temporary tab-separated file and then calls @code{read.table}.
This is adequate for small files, but for anything more complicated we
recommend using the facilities of a language like @code{perl} to
pre-process the file.
@cindex perl
@findex read.fortran
Function @code{read.fortran} is a similar function for fixed-format files,
using Fortran-style column specifications.
@node Data Interchange Format (DIF), Using scan directly, Fixed-width-format files, Spreadsheet-like data
@section Data Interchange Format (DIF)
@cindex Data Interchange Format (DIF)
An old format sometimes used for spreadsheet-like data is DIF, or Data Interchange
format.
@findex read.DIF
Function @code{read.DIF} provides a simple way to read such files. It takes
arguments similar to @code{read.table} for assigning types to each of the columns.
In Windows, spreadsheets often store spreadsheet data on the clipboard in this
format; @code{read.DIF("clipboard")} can read it from there directly.
It is slightly more robust than @code{read.table("clipboard")} in
handling spreadsheets with empty cells.
@node Using scan directly, Re-shaping data, Data Interchange Format (DIF), Spreadsheet-like data
@section Using @code{scan} directly
@findex scan
Both @code{read.table} and @code{read.fwf} use @code{scan} to read the
file, and then process the results of @code{scan}. They are very
convenient, but sometimes it is better to use @code{scan} directly.
Function @code{scan} has many arguments, most of which we have already
covered under @code{read.table}. The most crucial argument is
@code{what}, which specifies a list of modes of variables to be read
from the file. If the list is named, the names are used for the
components of the returned list. Modes can be numeric, character or
complex, and are usually specified by an example, e.g. @code{0},
@code{""} or @code{0i}. For example
@example
cat("2 3 5 7", "11 13 17 19", file="ex.dat", sep="\n")
scan(file="ex.dat", what=list(x=0, y="", z=0), flush=TRUE)
@end example
@noindent
returns a list with three components and discards the fourth column in
the file.
@findex readLines
There is a function @code{readLines} which will be more convenient if
all you want is to read whole lines into @R{} for further processing.
One common use of @code{scan} is to read in a large matrix. Suppose
file @file{matrix.dat} just contains the numbers for a 200 x 2000
matrix. Then we can use
@c write.table(matrix(rnorm(200*2000), 200), "matrix.dat", row.names=F, col.names=F)
@example
A <- matrix(scan("matrix.dat", n = 200*2000), 200, 2000, byrow = TRUE)
@end example
@noindent
On one test this took 1 second (under Linux, 3 seconds under Windows on
the same machine) whereas
@example
A <- as.matrix(read.table("matrix.dat"))
@end example
@noindent
took 10 seconds (and more memory), and
@example
A <- as.matrix(read.table("matrix.dat", header = FALSE, nrows = 200,
comment.char = "", colClasses = "numeric"))
@end example
@noindent
took 7 seconds. The difference is almost entirely due to the overhead
of reading 2000 separate short columns: were they of length 2000,
@code{scan} took 9 seconds whereas @code{read.table} took 18 if used
efficiently (in particular, specifying @code{colClasses}) and 125 if
used naively.
Note that timings can depend on the type read and the data.
Consider reading a million distinct integers:
@example
writeLines(as.character((1+1e6):2e6), "ints.dat")
xi <- scan("ints.dat", what=integer(0), n=1e6) # 0.77s
xn <- scan("ints.dat", what=numeric(0), n=1e6) # 0.93s
xc <- scan("ints.dat", what=character(0), n=1e6) # 0.85s
xf <- as.factor(xc) # 2.2s
DF <- read.table("ints.dat") # 4.5s
@end example
@noindent
and a million examples of a small set of codes:
@example
code <- c("LMH", "SJC", "CHCH", "SPC", "SOM")
writeLines(sample(code, 1e6, replace=TRUE), "code.dat")
y <- scan("code.dat", what=character(0), n=1e6) # 0.44s
yf <- as.factor(y) # 0.21s
DF <- read.table("code.dat") # 4.9s
DF <- read.table("code.dat", nrows=1e6) # 3.6s
@end example
Note that these timings depend heavily on the operating system (the
basic reads in Windows take at least as twice as long as these Linux
times) and on the precise state of the garbage collector.
@node Re-shaping data, Flat contingency tables, Using scan directly, Spreadsheet-like data
@section Re-shaping data
@cindex Re-shaping data
Sometimes spreadsheet data is in a compact format that gives the
covariates for each subject followed by all the observations on that
subject. @R{}'s modelling functions need observations in a single
column. Consider the following sample of data from repeated MRI brain
measurements
@example
Status Age V1 V2 V3 V4
P 23646 45190 50333 55166 56271
CC 26174 35535 38227 37911 41184
CC 27723 25691 25712 26144 26398
CC 27193 30949 29693 29754 30772
CC 24370 50542 51966 54341 54273
CC 28359 58591 58803 59435 61292
CC 25136 45801 45389 47197 47126
@end example
@noindent
There are two covariates and up to four measurements on each subject.
The data were exported from Excel as a file @file{mr.csv}.
@findex stack
We can use @code{stack} to help manipulate these data to give a single
response.
@example
zz <- read.csv("mr.csv", strip.white = TRUE)
zzz <- cbind(zz[gl(nrow(zz), 1, 4*nrow(zz)), 1:2], stack(zz[, 3:6]))
@end example
@noindent
with result
@example
Status Age values ind
X1 P 23646 45190 V1
X2 CC 26174 35535 V1
X3 CC 27723 25691 V1
X4 CC 27193 30949 V1
X5 CC 24370 50542 V1
X6 CC 28359 58591 V1
X7 CC 25136 45801 V1
X11 P 23646 50333 V2
...
@end example
@findex unstack.
Function @code{unstack} goes in the opposite direction, and may be
useful for exporting data.
@findex reshape
Another way to do this is to use the function
@code{reshape}, by
@example
> reshape(zz, idvar="id",timevar="var",
varying=list(c("V1","V2","V3","V4")),direction="long")
Status Age var V1 id
1.1 P 23646 1 45190 1
2.1 CC 26174 1 35535 2
3.1 CC 27723 1 25691 3
4.1 CC 27193 1 30949 4
5.1 CC 24370 1 50542 5
6.1 CC 28359 1 58591 6
7.1 CC 25136 1 45801 7
1.2 P 23646 2 50333 1
2.2 CC 26174 2 38227 2
...
@end example
The @code{reshape} has a more complicated syntax than @code{stack} but
can be used for data where the `long' form has more than the one column
in this example. With @code{direction="wide"}, @code{reshape} can also
perform the opposite transformation.
@node Flat contingency tables, , Re-shaping data, Spreadsheet-like data
@section Flat contingency tables
@cindex Flat contingency tables
Displaying higher-dimensional contingency tables in array form typically
is rather inconvenient. In categorical data analysis, such information
is often represented in the form of bordered two-dimensional arrays with
leading rows and columns specifying the combination of factor levels
corresponding to the cell counts. These rows and columns are typically
``ragged'' in the sense that labels are only displayed when they change,
with the obvious convention that rows are read from top to bottom and
columns are read from left to right. In @R{}, such ``flat'' contingency
tables can be created using @code{ftable},
@findex ftable
which creates objects of class @code{"ftable"} with an appropriate print
method.
As a simple example, consider the @R{} standard data set
@code{UCBAdmissions} which is a 3-dimensional contingency table
resulting from classifying applicants to graduate school at UC Berkeley
for the six largest departments in 1973 classified by admission and sex.
@example
> data(UCBAdmissions)
> ftable(UCBAdmissions)
Dept A B C D E F
Admit Gender
Admitted Male 512 353 120 138 53 22
Female 89 17 202 131 94 24
Rejected Male 313 207 205 279 138 351
Female 19 8 391 244 299 317
@end example
@noindent
The printed representation is clearly more useful than displaying the
data as a 3-dimensional array.
There is also a function @code{read.ftable} for reading in flat-like
contingency tables from files.
@findex read.ftable
This has additional arguments for dealing with variants on how exactly
the information on row and column variables names and levels is
represented. The help page for @code{read.ftable} has some useful
examples. The flat tables can be converted to standard contingency
tables in array form using @code{as.table}.
Note that flat tables are characterized by their ``ragged'' display of
row (and maybe also column) labels. If the full grid of levels of the
row variables is given, one should instead use @code{read.table} to read
in the data, and create the contingency table from this using
@code{xtabs}.
@node Importing from other statistical systems, Relational databases, Spreadsheet-like data, Top
@chapter Importing from other statistical systems
@cindex Importing from other statistical systems
In this chapter we consider the problem of reading a binary data file
written by another statistical system. This is often best avoided, but
may be unavoidable if the originating system is not available.
@menu
* EpiInfo Minitab SAS S-PLUS SPSS Stata Systat::
* Octave::
@end menu
@node EpiInfo Minitab SAS S-PLUS SPSS Stata Systat, Octave, Importing from other statistical systems, Importing from other statistical systems
@section EpiInfo, Minitab, S-PLUS, SAS, SPSS, Stata, Systat
The recommended package @pkg{foreign} provides import facilities for
files produced by these statistical systems, and for export to Stata. In
some cases these function may require substantially less memory than
@code{read.table} would. @code{write.foreign} (See @ref{Export to text files}) provides an export mechanism with support currently for @code{SPSS} and @code{Stata}.
@cindex EpiInfo
@cindex EpiData
@findex read.epiinfo
EpiInfo versions 5 and 6 stored data in a self-describing fixed-width
text format. @code{read.epiinfo} will read these @file{.REC} files into
an @R{} data frame. EpiData also produces data in this format.
@cindex Minitab
@findex read.mtp
Function @code{read.mtp} imports a `Minitab Portable Worksheet'. This
returns the components of the worksheet as an @R{} list.
@cindex SAS
@findex read.xport
Function @code{read.xport} reads a file in SAS Transport (XPORT) format
and return a list of data frames. If SAS is available on your system,
function @code{read.ssd} can be used to create and run a SAS script that
saves a SAS permanent dataset (@file{.ssd} or @file{.sas7bdat}) in
Transport format. It then calls @code{read.xport} to read the resulting
file. Package @pkg{Hmisc} has a similar function @code{sas.get}, also
running SAS.
@cindex S-PLUS
@findex read.S
@findex data.restore
Function @code{read.S} which can read binary objects produced by S-PLUS
3.x, 4.x or 2000 on (32-bit) Unix or Windows (and can read them on a
different OS). This is able to read many but not all @Sl{} objects: in
particular it can read vectors, matrices and data frames and lists
containing those.
Function @code{data.restore} reads S-PLUS data dumps (created by
@code{data.dump}) with the same restrictions (except that dumps from the
Alpha platform can also be read). It should be possible to read data
dumps from S-PLUS 5.x and 6.x written with @code{data.dump(oldStyle=T)}.
If you have access to S-PLUS, it is usually more reliable to @code{dump}
the object(s) in S-PLUS and @code{source} the dumpfile in @R{}. For
S-PLUS 5.x and 6.x you may need to use @code{dump(..., oldStyle=T)}, and
to read in very large objects it may be preferable to use the dumpfile
as a batch script rather than use the @code{source} function.
@cindex SPSS
@cindex SPSS Data Entry
@findex read.spss
Function @code{read.spss} can read files created by the `save' and
`export' commands in @acronym{SPSS}. It returns a list with one
component for each variable in the saved data set. @acronym{SPSS}
variables with value labels are optionally converted to @R{} factors.
@acronym{SPSS} Data Entry is an application for creating data entry
forms. By default it creates data files with extra formatting
information that @code{read.spss} cannot handle, but it is possible to
export the data in an ordinary @acronym{SPSS} format.
@cindex Stata
@findex read.dta
@findex write.dta
Stata @file{.dta} files are a binary file format. Files from versions 5,
6, 7/SE and 8 of Stata can be read and written by functions
@code{read.dta} and @code{write.dta}. Stata variables with value labels
are optionally converted to (and from) @R{} factor.
@cindex Systat
@findex read.systat
@code{read.systat} reads those Systat @code{SAVE} files that are
rectangular data files (@code{mtype = 1}) written on little-endian
machines (such as from Windows). These have extension @file{.sys}
or (more recently) @file{.syd}.
@node Octave, , EpiInfo Minitab SAS S-PLUS SPSS Stata Systat, Importing from other statistical systems
@section Octave
@cindex Octave
@findex read.octave
Octave is a numerical linear algebra system
(@url{http://www.octave.org}), and function @code{read.octave} in
package @pkg{foreign} can read in files in Octave text data format
created using the Octave command @command{save -ascii}, with support for
most of the common types of variables, including the standard atomic
(real and complex scalars, matrices, and @math{N}-d arrays, strings,
ranges, and boolean scalars and matrices) and recursive (structs, cells,
and lists) ones.
@node Relational databases, Binary files, Importing from other statistical systems, Top
@chapter Relational databases
@cindex Relational databases
@cindex DBMS
@menu
* Why use a database?::
* Overview of RDBMSs::
* R interface packages::
@end menu
@node Why use a database?, Overview of RDBMSs, Relational databases, Relational databases
@section Why use a database?
There are limitations on the types of data that @R{} handles well.
Since all data being manipulated by @R{} are resident in memory, and
several copies of the data can be created during execution of a
function, @R{} is not well suited to extremely large data sets. Data
objects that are more than a (few) hundred megabytes in size can cause
@R{} to run out of memory.
@R{} does not easily support concurrent access to data. That is, if
more than one user is accessing, and perhaps updating, the same data,
the changes made by one user will not be visible to the others.
@R{} does support persistence of data, in that you can save a data
object or an entire worksheet from one session and restore it at the
subsequent session, but the format of the stored data is specific to
@R{} and not easily manipulated by other systems.
Database management systems (DBMSs) and, in particular, relational
DBMSs (RDBMSs) @emph{are} designed to do all of these things well.
Their strengths are
@enumerate
@item
To provide fast access to selected parts of large databases.
@item
Powerful ways to summarize and cross-tabulate columns in databases.
@item
Store data in more organized ways than the rectangular grid model of