Skip to content
This repository has been archived by the owner on May 4, 2019. It is now read-only.

Commit

Permalink
Deprecate PooledDataArray
Browse files Browse the repository at this point in the history
Recommend either CategoricalArrays (for categorical data) or PooledArrays (for compressed storage) as replacements. Do not ajust tests for now.
  • Loading branch information
nalimilan committed Oct 21, 2017
1 parent 8a7003b commit 9dfc1c5
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 20 deletions.
21 changes: 2 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,8 @@ Documentation:
[![](https://img.shields.io/badge/docs-stable-blue.svg)](https://JuliaStats.github.io/DataArrays.jl/stable)
[![](https://img.shields.io/badge/docs-latest-blue.svg)](https://JuliaStats.github.io/DataArrays.jl/latest)

The DataArrays package provides array types for working efficiently with [missing data](https://en.wikipedia.org/wiki/Missing_data)
The DataArrays package provides the `DataArray` type for working efficiently with [missing data](https://en.wikipedia.org/wiki/Missing_data)
in Julia, based on the `null` value from the [Nulls.jl](https://github.com/JuliaData/Nulls.jl) package.
In particular, it provides the following:

* `DataArray{T}`: An array-like data structure that can contain values of type `T`, but can also contain missing values.
* `PooledDataArray{T}`: A variant of `DataArray{T}` optimized for representing arrays that contain many repetitions of a small number of unique values -- as commonly occurs when working with categorical data.

# DataArray's

Most Julian arrays cannot contain `null` values: only `Array{Union{T, Null}}` and more generally `Array{>:Null}` can contain `null` values.

Expand All @@ -28,15 +22,4 @@ For example, a `DataArray{Int}` can contain integers and `null` values. We can c

da = @data([1, 2, null, 4])

# PooledDataArray's

As noted above, the `DataArray` type provides an efficient array-like data structure that contain potentially missing values. When working with categorical data sets in which a large number of data points occur, but only take on a limited set of unique values, we provide an analog to `DataArray` that is optimized for efficient memory usage: `PooledDataArray`.

For example, we can use a PooledDataArray to represent the following data efficiently:

d = repeat(["Group A", "Group B"], inner = [1000, 1000])
pda = PooledDataArray(d)

Instead of maintaining an array of 2,000 distinct strings, the PooledDataArray will maintain an array of 2,000 distinct references to the 2 unique values: `"Group A"` and `"Group B"`.

In addition to using less memory, PooledDataArray's also maintain a cached record of the unique values in an array: for applications, like the creation of dummy variables, in which the unique values are frequently needed, this allows computations to avoid recalculating the unique values.
This package used to provide the `PooledDataArray` type, a variant of `DataArray{T}` optimized for representing arrays that contain many repetitions of a small number of unique values. `PooledDataArray` has been deprecated in favor of [`CategoricalArray`](https://github.com/JuliaData/CategoricalArrays.jl) or [`PooledArray`](https://github.com/JuliaComputing/PooledArrays.jl).
2 changes: 1 addition & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ In particular, it provides the following:

* `DataArray{T}`: An array type that can house both values of type `T` and missing values (of type `Null`)
* `PooledDataArray{T}`: An array type akin to `DataArray` but optimized for arrays with a smaller set of unique
values, as commonly occurs with categorical data
values, as commonly occurs with categorical data. This type is now deprecated in favor of [`CategoricalArray`](https://github.com/JuliaData/CategoricalArrays.jl) or [`PooledArray`](https://github.com/JuliaComputing/PooledArrays.jl).

## Installation

Expand Down
3 changes: 3 additions & 0 deletions src/pooleddataarray.jl
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ mutable struct PooledDataArray{T, R<:Integer, N} <: AbstractDataArray{T, N}
pool::Vector{T}

function PooledDataArray{T,R,N}(rs::RefArray{R, N}, p::Vector{T}) where {T,R,N}
Base.depwarn("PooledDataArray is deprecated, use CategoricalArray " *
"from the CategoricalArrays package or PooledArray " *
"from the PooledArrays package instead.", :PooledDataArray)
# refs mustn't overflow pool
if length(rs.a) > 0 && maximum(rs.a) > prod(size(p))
throw(ArgumentError("Reference array points beyond the end of the pool"))
Expand Down

0 comments on commit 9dfc1c5

Please sign in to comment.