Releases: SebKrantz/collapse
collapse version 1.7.6
-
Corrected a C-level bug in
gsplit
that could lead R to crash in some instances (gsplit
is used internally infsummarise
,fmutate
,BY
andcollap
to perform computations with base R (non-optimized) functions). -
Ensured that
BY.grouped_df
always (by default) returns grouping columns in aggregations i.e.iris |> gby(Species) |> nv() |> BY(sum)
now gives the same asiris |> gby(Species) |> nv() |> fsum()
. -
A
.
was added to the first argument of functionsfselect
,fsubset
,colorder
andfgroup_by
, i.e.fselect(x, ...) -> fselect(.x, ...)
. The reason for this is that over time I added the option to select-rename columns e.g.fselect(mtcars, cylinders = cyl)
, which was not offered when these functions were created. This presents problems if columns should be renamed intox
, e.g.fselect(mtcars, x = cyl)
failed, see #221. Renaming the first argument to.x
somewhat guards against such situations. I think this change is worthwhile to implement, because it makes the package more robust going forward, and usually the first argument of these functions is never invoked explicitly. I really hope this breaks nobody's code. -
Added a function
GRPN
to make it easy to add a column of group sizes e.g.mtcars %>% fgroup_by(cyl,vs,am) %>% ftransform(Sizes = GRPN(.))
ormtcars %>% ftransform(Sizes = GRPN(list(cyl, vs, am)))
orGRPN(mtcars, by = ~cyl+vs+am)
. -
Added
[.pwcor
and[.pwcov
, to be able to subset correlation/covariance matrices without loosing the print formatting.
collapse version 1.7.5
collapse 1.7.5
-
In the development version on GitHub, a
.
was added to the first argument of functionsfselect
,fsubset
,colorder
andfgroup_by
, i.e.fselect(x, ...) -> fselect(.x, ...)
. The reason for this is that over time I added the option to select-rename columns e.g.fselect(mtcars, cylinders = cyl)
, which was not offered when these functions were created. This presents problems if columns should be renamed intox
, e.g.fselect(mtcars, x = cyl)
fails, see e.g. #221 . Renaming the first argument to.x
somewhat guards against such situations. I think this API change is worthwhile to implement, because it makes the package more robust going forward, and usually the first argument of these functions is never invoked explicitly. For now it remains in the development version which you can install usingremotes::install_github("SebKrantz/collapse")
. If you have strong objections to this change (because it will break your code or you know of people that have a programming style where they explicitly set the first argument of data manipulation functions), please let me know! -
Also ensuring tidyverse examples are in
\donttest{}
and building without the dplyr testing file to avoid issues with static code analysis on CRAN. -
20-50% Speed improvement in
gsplit
(and therefore infsummarise
,fmutate
,collap
andBY
when invoked with base R functions) when grouping withGRP(..., sort = TRUE, return.order = TRUE)
. To enable this by default, the default for argumentreturn.order
inGRP
was set tosort
, which retains the ordering vector (needed for the optimization). Retaining the ordering vector uses up some memory which can possibly adversely affect computations with big data, but with big datasort = FALSE
usually gives faster results anyway, and you can also always setreturn.order = FALSE
(also infgroup_by
,collap
), so this default gives the best of both worlds.
- An ancient depreciated argument
sort.row
(replaced bysort
in 2020) is now removed fromcollap
. Also argumentsreturn.order
andmethod
were added tocollap
providing full control of the grouping that happens internally.
collapse 1.7.4
-
Tests needed to be adjusted for the upcoming release of dplyr 1.0.8 which involves an API change in
mutate
.fmutate
will not take over these changes i.e.fmutate(..., .keep = "none")
will continue to work likedplyr::transmute
. Furthermore, no more tests involving dplyr are run on CRAN, and I will also not follow along with any future dplyr API changes. -
The C-API macro
installTrChar
(used in the newmassign
function) was replaced withinstallChar
to maintain backwards compatibility with R versions prior to 3.6.0. Thanks @tedmoorman #213. -
Minor improvements to
group()
, providing increased performance for doubles and also increased performance when the second grouping variable is integer, which turned out to be very slow in some instances.
collapse version 1.7.3
-
Removed tests involving the weights package (which is not available on R-devel CRAN checks).
-
fgroup_by
is more flexible, supporting computing columns e.g.fgroup_by(GGDC10S, Variable, Decade = floor(Year / 10) * 10)
and various programming options e.g.fgroup_by(GGDC10S, 1:3)
,fgroup_by(GGDC10S, c("Variable", "Country"))
, orfgroup_by(GGDC10S, is.character)
. You can also use column sequences e.g.fgroup_by(GGDC10S, Country:Variable, Year)
, but this should not be mixed with computing columns. Compute expressions may also not include the:
function. -
More memory efficient attribute handling in C/C++ (using C-API macro
SHALLOW_DUPLICATE_ATTRIB
instead ofDUPLICATE_ATTRIB
) in most places.
collapse version 1.7.2
- Ensured that the base pipe
|>
is not used in tests or examples, to avoid errors on CRAN checks with older versions of R.
collapse version 1.7.1
collapse 1.7.1
-
Fixed minor C/C++ issues flagged in CRAN checks.
-
Added option
ties = "last"
tofmode
. -
Added argument
stable.algo
toqsu
. Settingstable.algo = FALSE
toggles a faster calculation of the standard deviation, yielding 2x speedup on large datasets. -
Fast Statistical Functions now internally use
group
for grouping data if bothg
andTRA
arguments are used, yielding efficiency gains on unsorted data. -
Ensured that
fmutate
andfsummarise
can be called if collapse is not attached.
collapse version 1.7.0
collapse 1.7.0
collapse 1.7.0, released mid January 2022, brings major improvements in the computational backend of the package, it's data manipulation capabilities, and a whole set of new functions that enable more flexible and memory efficiency R programming - significantly enhancing the language itself. For the vast majority of codes, updating to 1.7 should not cause any problems.
Changes to functionality
-
num_vars
is now implemented in C, yielding a massive performance increase over checking columns usingvapply(x, is.numeric, logical(1))
. It selects columns where(is.double(x) || is.integer(x)) && !is.object(x)
. This provides the same results for most common classes found in data frames (e.g. factors and date columns are not numeric), however it is possible for users to define methods foris.numeric
for other objects, which will not be respected bynum_vars
anymore. A prominent example are base R's 'ts' objects i.e.is.numeric(AirPassengers)
returnsTRUE
, butis.object(AirPassengers)
is alsoTRUE
so the above yieldsFALSE
, implying - if you happened to work with data frames of 'ts' columns - thatnum_vars
will now not select those anymore. Please make me aware if there are other important classes that are found in data frames and whereis.numeric
returnsTRUE
.num_vars
is also used internally incollap
so this might affect your aggregations. -
In
flag
,fdiff
andfgrowth
, if a plain numeric vector is passed to thet
argument such thatis.double(t) && !is.object(t)
, it is coerced to integer usingas.integer(t)
and directly used as time variable, rather than applying ordered grouping first. This is to avoid the inefficiency of grouping, and owes to the fact that in most data imported into R with various packages, the time (year) variables are coded as double although they should be integer (I also don't know of any cases where time needs to be indexed by a non-date variable with decimal places). Note that the algorithm internally handles irregularity in the time variable so this is not a problem. Should this break any code, kindly raise an issue on GitHub. -
The function
setrename
now truly renames objects by reference (without creating a shallow copy). The same is true forvlabels<-
(which was rewritten in C) and a new functionsetrelabel
. Thus additional care needs to be taken (with use inside functions etc.) as the renaming will take global effects unless a shallow copy of the data was created by some prior operation inside the function. If in doubt, better usefrename
orrelabel
which do create a shallow copy. -
Some improvements to the
BY
function, both in terms of performance and security. Performance is enhanced through a new C functiongsplit
, providing split-apply-combine computing speeds competitive with dplyr on a much broader range of R objects. Regarding Security: if the result of the computation has the same length as the original data, names / rownames and grouping columns (for grouped data) are only added to the result object if known to be valid, i.e. if the data was originally sorted by the grouping columns (information recorded byGRP.default(..., sort = TRUE)
, which is called internally on non-factor/GRP/qG objects). This is becauseBY
does not reorder data after the split-apply-combine step (unlikedplyr::mutate
); data are simply recombined in the order of the groups. Because of this, in general,BY
should be used to compute summary statistics (unless data are sorted before grouping). The added security makes this explicit. -
Added a method
length.GRP
giving the length of a grouping object. This could break code callinglength
on a grouping object before (which just returned the length of the list). -
Functions renamed in collapse 1.6.0 will now print a message telling you to use the updated names. The functions under the old names will stay around for 1-3 more years.
-
The passing of argument
order
instead ofsort
in functionGRP
(from a very early version of collapse), is now disabled.
Bug Fixes
- Fixed a bug in some functions using Welfords Online Algorithm (
fvar
,fsd
,fscale
andqsu
) to calculate variances, occurring when initial or final zero weights caused the running sum of weights in the algorithm to be zero, yielding a division by zero andNA
as output although a value was expected. These functions now skip zero weights alongside missing weights, which also implies that you can pass a logical vector to the weights argument to very efficiently calculate statistics on a subset of data (e.g. usingqsu
).
Additions
Basic Computational Infrastructure
-
Function
group
was added, providing a low-level interface to a new unordered grouping algorithm based on hashing in C and optimized for R's data structures. The algorithm was heavily inspired by the greatkit
package of Morgan Jacob, and now feeds into the package through multiple central functions (includingGRP
/fgroup_by
,funique
andqF
) when invoked with argumentsort = FALSE
. It is also used in internal groupings performed in data transformation functions such asfwithin
(when no factor or 'GRP' object is provided to theg
argument). The speed of the algorithm is very promising (often superior toradixorder
), and it could be used in more places still. I welcome any feedback on it's performance on different datasets. -
Function
gsplit
provides an efficient alternative tosplit
based on grouping objects. It is used as a new backend torsplit
(which also supports data frame) as well asBY
,collap
,fsummarise
andfmutate
- for more efficient grouped operations with functions external to the package. -
Added multiple functions to facilitate memory efficient programming (written in C). These include elementary mathematical operations by reference (
setop
,%+=%
,%-=%
,%*=%
,%/=%
), supporting computations involving integers and doubles on vectors, matrices and data frames (including row-wise operations viasetop
) with no copies at all. Furthermore a set of functions which check a single value against a vector without generating logical vectors:whichv
,whichNA
(operators%==%
and%!=%
which return indices and are significantly faster than==
, especially inside functions likefsubset
),anyv
andallv
(allNA
was already added before). Finally, functionssetv
andcopyv
speed up operations involving the replacement of a value (x[x == 5] <- 6
) or of a sequence of values from a equally sized object (x[x == 5] <- y[x == 5]
, orx[ind] <- y[ind]
whereind
could be pre-computed vectors or indices) in vectors and data frames without generating any logical vectors or materializing vector subsets. -
Function
vlengths
was added as a more efficient alternative tolengths
(without method dispatch, simply coded in C). -
Function
massign
provides a multivariate version ofassign
(written in C, and supporting all basic vector types). In addition the operator%=%
was added as an efficient multiple assignment operator. (It is called%=%
and not%<-%
to facilitate the translation of Matlab or Python codes into R, and because the zeallot package already provides multiple-assignment operators (%<-%
and%->%
), which are significantly more versatile, but orders of magnitude slower than%=%
)
High-Level Features
-
Fully fledged
fmutate
function that provides functionality analogous todplyr::mutate
(sequential evaluation of arguments, including arbitrary tagged expressions andacross
statements).fmutate
is optimized to work together with the packages Fast Statistical and Data Transformation Functions, yielding fast, vectorized execution, but also benefits fromgsplit
for other operations. -
across()
function implemented for use insidefsummarise
andfmutate
. It is also optimized for Fast Statistical and Data Transformation Functions, but performs well with other functions too. It has an additional arguments.apply = FALSE
which will apply functions to the entire subset of the data instead of individual columns, and thus allows for nesting tibbles and estimating models or correlation matrices by groups etc..across()
also supports an arbitrary number of additional arguments which are split and evaluated by groups if necessary. Multipleacross()
statements can be combined with tagged vector expressions in a single call tofsummarise
orfmutate
. Thus the computational framework is pretty general and similar to data.table, although less efficient with big datasets. -
Added functions
relabel
andsetrelabel
to make interactive dealing with variable labels a bit easier. Note that both functions operate by reference. (Throughvlabels<-
which is implemented in C. Taking a shallow copy of the data frame is useless in this case because variable labels are attributes of the columns, not of the frame). The only difference between the two is thatsetrelabel
returns the result invisibly. -
function shortcuts
rnm
andmtt
added forfrename
andfmutate
.across
can also be abbreviated usingacr
. -
Added two options that can be invoked before loading of the package to change the namespace:
options(collapse_mask = c(...))
can be set to export copies of selected (or all) functions in the package that start withf
removing the leadingf
e.g.fsubset
->subset
(bothfsubset
andsubset
will be exported). This allows masking base R and dplyr functions (even basic functions such assum
,mean
,unique
etc. if desired) with collapse's fast functions, facilitating the optimization of existing codes and allowing you to work with collapse using a more natural namespace. The package has been internally insulated against such changes, but of course they might have major effects on...
collapse version 1.6.5
collapse 1.6.5
-
Use of
VECTOR_PTR
in C API now gives an error on R-devel even ifUSE_RINTERNALS
is defined. Thus this patch gets rid of all remaining usage of this macro to avoid errors on CRAN checks using the development version of R. -
The print method for
qsu
now uses an apostrophe (') to designate million digits, instead of a comma (,). This is to avoid confusion with the decimal point, and the typical use of (,) for thousands (which I don't like).
collapse version 1.6.4
collapse 1.6.4
A patch for 1.6.0 which fixes (minor) issues flagged by CRAN and adds a few handy extras.
Bug Fixes
-
Puts examples using the new base pipe
|>
inside\donttest{}
so that they don't fail CRAN tests on older R versions. -
Fixes a LTO issue caused by a small mistake in a header file (which does not have any implications to the user but was detected by CRAN checks).
-
Checks on the gcc11 compiler flagged an additional issue with a pointer pointing to element -1 of a C array (which I had done on purpose to index it with an R integer vector).
-
Fixes a valgrind issue because of comparing an uninitialized value to something.
Additions
-
Added a function
fcomputev
, which allows selecting columns and transforming them with a function in one go. Thekeep
argument can be used to add columns to the selection that are not transformed. -
Added a function
setLabels
as a wrapper aroundvlabels<-
to facilitate setting variable labels inside pipes. -
Function
rm_stub
now has an argumentregex = TRUE
which triggers a call togsub
and allows general removing of character sequences in column names on the fly.
Improvements
vlabels<-
andsetLabels
now support list of variable labels or other attributes (i.e. thevalue
is internally subset using[[
, not[
). Thus they are now general functions to attach a vector or list of attributes to columns in a list / data frame.
Other Changes
- CRAN maintainers have asked me to remove a line in a Makevars file intended to reduce the size of Rcpp object files (which has been there since version 1.4). So the installed size of the package may now be larger.
collapse version 1.6.0
collapse 1.6.0
collapse 1.6.0, released end of June 2021, presents some significant improvements in the user-friendliness, compatibility and programmability of the package, as well as a few function additions.
Changes to Functionality
-
ffirst
,flast
,fnobs
,fsum
,fmin
andfmax
were rewritten in C. The former three now also support list columns (whereNULL
or empty list elements are considered missing values whenna.rm = TRUE
), and are extremely fast for grouped aggregation ifna.rm = FALSE
. The latter three also support and return integers, with significant performance gains, even compared to base R. Code using these functions expecting an error for list-columns or expecting double output even if the input is integer should be adjusted. -
collapse now directly supports sf data frames through functions like
fselect
,fsubset
,num_vars
,qsu
,descr
,varying
,funique
,roworder
,rsplit
,fcompute
etc., which will take along the geometry column even if it is not explicitly selected (mirroring dplyr methods for sf data frames). This is mostly done internally at C-level, so functions remain simple and fast. Existing code that explicitly selects the geometry column is unaffected by the change, but code of the formsf_data %>% num_vars %>% qDF %>% ...
, where columns excluding geometry were selected and the object later converted to a data frame, needs to be rewritten assf_data %>% qDF %>% num_vars %>% ...
. A short vignette was added describing the integration of collapse and sf. -
I've received several requests for increased namespace consistency. collapse functions were named to be consistent with base R, dplyr and data.table, resulting in names like
is.Date
,fgroup_by
orsettransformv
. To me this makes sense, but I've been convinced that a bit more consistency is advantageous. Towards that end I have decided to eliminate the '.' notation of base R and to remove some unexpected capitalizations in function names giving some people the impression I was using camel-case. The following functions are renamed:
fNobs
->fnobs
,fNdistinct
->fndistinct
,pwNobs
->pwnobs
,fHDwithin
->fhdwithin
fHDbetween
->fhdbetween
,as.factor_GRP
->as_factor_GRP
,as.factor_qG
->as_factor_qG
,is.GRP
->is_GRP
,is.qG
->is_qG
,is.unlistable
->is_unlistable
,is.categorical
->is_categorical
,is.Date
->is_date
,as.numeric_factor
->as_numeric_factor
,as.character_factor
->as_character_factor
,
Date_vars
->date_vars
.
This is done in a very careful manor, the others will stick around for a long while (end of 2022), and the generics offNobs
,fNdistinct
,fHDbetween
andfHDwithin
will be kept in the package for an indeterminate period, but their core methods will not be exported beyond 2022. I will start warning about these renamed functions in 2022. In the future I will undogmatically stick to a function naming style with lowercase function names and underslashes where words need to be split. Other function names will be kept. To say something about this: The quick-conversion functionsqDF
qDT
,qM
,qF
,qG
are consistent and in-line with data.table (setDT
etc.), and similarly the operatorsL
,F
,D
,Dlog
,G
,B
,W
,HDB
,HDW
. I'll keepGRP
,BY
andTRA
, for lack of better names, parsimony and because they are central to the package. The camel case will be kept in helper functionssetDimnames
etc. because they work like statssetNames
and do not modify the argument by reference (likesettransform
orsetrename
and various data.table functions). FunctionscopyAttrib
andcopyMostAttrib
are exports of like-named functions in the C API and thus kept as they are. Finally, I want to keepfFtest
the way it is because the F-distribution is widely recognized by a capital F. -
I've updated the
wlddev
dataset with the latest data from the World Bank, and also added a variable giving the total population (which may be useful e.g. for population-weighted aggregations across regions). The extra column could invalidate codes used to demonstrate something (I had to adjust some examples, tests and code in vignettes).
Additions
-
Added a function
fcumsum
(written in C), permitting flexible (grouped, ordered) cumulative summations on matrix-like objects (integer or double typed) with extra methods for grouped data frames and panel series and data frames. Apart from the internal grouping, and an ordering argument allowing cumulative sums in a different order than data appear,fcumsum
has 2 options to deal with missing values. The default (na.rm = TRUE
) is to skip (preserve) missing values, whereas settingfill = TRUE
allows missing values to be populated with the previous value of the cumulative sum (starting from 0). -
Added a function
alloc
to efficiently generate vectors initialized with any value (faster thanrep_len
). -
Added a function
pad
to efficiently pad vectors / matrices / data.frames with a value (default isNA
). This function was mainly created to make it easy to expand results coming from a statistical model fitted on data with missing values to the original length. For example letdata <- na_insert(mtcars); mod <- lm(mpg ~ cyl, data)
, then we can dosettransform(data, resid = pad(resid(mod), mod$na.action))
, or we could dopad(model.matrix(mod), mod$na.action)
orpad(model.frame(mod), mod$na.action)
to receive matrices and data frames from model data matching the rows ofdata
.pad
is a general function that will also work with mixed-type data. It is also possible to pass a vector of indices matching the rows of the data topad
, in which casepad
will fill gaps in those indices with a value/row in the data.
Improvements
-
Full data.table support, including reference semantics (
set*
,:=
)!! There is some complex C-level programming behind data.table's operations by reference. Notably, additional (hidden) column pointers are allocated to be able to add columns without taking a shallow copy of the data.table, and an".internal.selfref"
attribute containing an external pointer is used to check if any shallow copy was made using base R commands like<-
. This is done to avoid even a shallow copy of the data.table in manipulations using:=
(and is in my opinion not worth it as even large tables are shallow copied by base R (>=3.1.0) within microseconds and all of this complicates development immensely). Previously, collapse treated data.table's like any other data frame, using shallow copies in manipulations and preserving the attributes (thus ignoring how data.table works internally). This produced a warning whenever you wanted to use data.table reference semantics (set*
,:=
) after passing the data.table through a collapse function such ascollap
,fselect
,fsubset
,fgroup_by
etc. From v1.6.0, I have adopted essential C code from data.table to do the overallocation and generate the".internal.selfref"
attribute, thus seamless workflows combining collapse and data.table are now possible. This comes at a cost of about 2-3 microseconds per function, as to do this I have to shallow copy the data.table again and add extra column pointers and an".internal.selfref"
attribute telling data.table that this table was not copied (it seems to be the only way to do it for now). This integration encompasses all data manipulation functions in collapse, but not the Fast Statistical Functions themselves. Thus you can doagDT <- DT %>% fselect(id, col1:coln) %>% collap(~id, fsum); agDT[, newcol := 1]
, but you would need to do add aqDT
after a function likefsum
if you want to use reference semantics without incurring a warning:agDT <- DT %>% fselect(id, col1:coln) %>% fgroup_by(id) %>% fsum %>% qDT; agDT[, newcol := 1]
. collapse appears to be the first package that attempts to account for data.table's internal working without importing data.table, andqDT
is now the fastest way to create a fully functional data.table from any R object. A global option"collapse_DT_alloccol"
was added to regulate how many columns collapse overallocates when creating data.table's. The default is 100, which is lower than the data.table default of 1024. This was done to increase efficiency of the additional shallow copies, and may be changed by the user. -
Programming enabled with
fselect
andfgroup_by
(you can now pass vectors containing column names or indices). Note that instead offselect
you should useget_vars
for standard eval programming. -
fselect
andfsubset
support in-place renaming e.g.fselect(data, newname = var1, var3:varN)
,
fsubset(data, vark > varp, newname = var1, var3:varN)
. -
collap
supports renaming columns in the custom argument, e.g.collap(data, ~ id, custom = list(fmean = c(newname = "var1", "var2"), fmode = c(newname = 3), flast = is_date))
. -
Performance improvements:
fsubset
/ss
return the data or perform a simple column subset without deep copying the data if all rows are selected through a logical expression.fselect
andget_vars
,num_vars
etc. are slightly faster through data frame subsetting done fully in C.ftransform
/fcompute
usealloc
instead ofbase::rep
to replicate a scalar value which is slightly more efficient. -
fcompute
now has akeep
argument, to preserve several existing columns when computing columns on a data frame. -
replace_NA
now has acols
argument, so we can doreplace_NA(data, cols = is.numeric)
, to replaceNA
's in numeric columns. I note that for big numeric datadata.table::setnafill
is the most efficient solution. -
fhdbetween
andfhdwithin
have aneffect
argument in plm methods, allowing centering on selected identifiers. The default is still to center on all panel identifiers.
...
collapse version 1.5.3
Changes to Functionality
-
The first argument of
ftransform
was renamed to.data
fromX
. This was done to enable the user to transform columns named "X". For the same reason the first argument offrename
was renamed to.x
fromx
(not.data
to make it explicit that.x
can be any R object with a "names" attribute). It is not possible to depreciateX
andx
without at the same time undoing the benefits of the argument renaming, thus this change is immediate and code breaking in rare cases where the first argument is explicitly set. -
The function
is.regular
to check whether an R object is atomic or list-like is depreciated and will be removed before the end of the year. This was done to avoid a namespace clash with the zoo package (#127).
Bug Fixes
-
For reasons of efficiency, most statistical and transformation functions used the C macro
SHALLOW_DUPLICATE_ATTRIB
to copy column attributes in a data frame. Since this macro does not copy S4 object bits, it caused some problems with S4 object columns such as POSIXct (e.g. computing lags/leads, first and last values on these columns). This is now fixed, all statistical functions (apart fromfvar
andfsd
) now useDUPLICATE_ATTRIB
and thus preserve S4 object columns (#91). -
unlist2d
produced a subsetting error if an empty list was present in the list-tree. This is now fixed, empty orNULL
elements in the list-tree are simply ignored (#99).
Additions
-
A function
fsummarise
was added to facilitate translating dplyr / data.table code to collapse. Likecollap
, it is only very fast when used with the Fast Statistical Functions. -
A function
t_list
is made available to efficiently transpose lists of lists.
Improvements
- C files are compiled -O3 on Windows, which gives a boost of around 20% for the grouping mechanism applied to character data.