Update to DataFrames v0.11 (CategoricalArrays + Missings) #28

alyst · 2017-09-15T17:14:17Z

I suppose now it's the time for #19:

update to the latest DataFrames.jl ~~v0.10~~v0.11 and switch to ~~Nulls.jl~~Missings.jl and CategoricalArrays.jl from DataArrays.jl (NAs are representing as Union{T, Null}).
drop compatibility with Julia v0.5
fix R logical vector conversion

alyst · 2017-09-15T17:43:34Z

Oops, still a little bit too early. It needs DataFrames.jl master.

alyst · 2017-09-16T09:34:03Z

The CI pass on v0.6, but not on nightly (some error in DataStructures.jl)

nalimilan

Great!

nalimilan · 2017-09-16T08:56:16Z

NEWS.md

+##### Changes
+* R logical vectors converted to `Vector{Bool}` (instead of `Vector{Int32}`)
+* R factors converted into `CategoricalVector` (instead of `PooledDataArray`) ([#19])
+* switched from DataVector to Vector{Union{T,Null}} for NAs ([#19])


Missing backticks.

nalimilan · 2017-09-16T08:57:16Z

NEWS.md

@@ -15,5 +25,6 @@ Initial release based on `DataFrames.read_rda()` ([JuliaStats/DataFrames.jl#1031
 [#9]: https://github.com/JuliaStats/RData.jl/issues/9
 [#10]: https://github.com/JuliaStats/RData.jl/issues/10
 [#15]: https://github.com/JuliaStats/RData.jl/issues/15
+[#19]: https://github.com/JuliaStats/RData.jl/issues/19


Wrong place?

nalimilan · 2017-09-16T08:57:44Z

REQUIRE

-DataFrames 0.7
-DataArrays 0.3
+julia 0.6
+DataFrames 0.10


That will be 0.11.

I guess I should keep it 0.10 for now, otherwise the package won't install.

nalimilan · 2017-09-16T08:59:37Z

src/context.jl

 int2ver(v::Integer) = VersionNumber(v >> 16, (v >> 8) & 0xff, v & 0xff)
-function RDAContext{T<:RDAIO}(io::T, kwoptions::Vector{Any}=Any[])
+
+function RDAContext(io::T, kwoptions::Vector{Any}=Any[]) where {T<:RDAIO}


No need to define T.

nalimilan · 2017-09-16T09:03:54Z

src/convert.jl

 # if re or im is NA, the whole complex number is NA
-# FIXME avoid temporary Vector{Bool}
-namask(rc::RComplexVector) = BitArray([isna_float64(v.re) || isna_float64(v.im) for v in reinterpret(Complex{UInt64}, rc.data)])
+namask(rc::RComplexVector) = [isna_float64(v.re) || isna_float64(v.im) for v in reinterpret(Complex{UInt64}, rc.data)]


Better use broadcast with an anonymous function so that the result is a BitArray. Though I see you switched to Vector{Bool} in the RNullableVector struct. Why change that? Also, other namask methods return a BitArray, so that change may force an inefficient conversion.

That was a legacy from a switch to DataTables and the fact that I forgot that vectorizing logical operation gives BitVector.
I'll fix namask() to return BitVector, but should RLogicalVector be converted into Vector{Bool} or BitVector given that it will most likely be a column in a data frame?

I don't think the difference matters a lot, given that the conversion to Array{Union{Null, T}} won't be able to reuse the mask directly. Array{Bool} should be faster, but will also use 8× more memory.

In the revision I've removed namask(), because with Union{T,Null} we don't actually need NA mask. That was the legacy from DataArray/NullableArray times.

nalimilan · 2017-09-16T09:08:23Z

src/convert.jl

+# convert R logical vector (uses Int32 to store values) into [Nullable]Vector{Bool}
+function jlvec(rl::RLogicalVector, force_nullable::Bool=true)
+    na_mask = namask(rl)
+    vals = rl.data .!= zero(eltype(rl.data))


.!= 0 should be enough here.

nalimilan · 2017-09-16T09:14:49Z

src/convert.jl

-    return PooledDataArray(DataArrays.RefArray(refs), pool)
+    # FIXME set ordered flag
+    refs = na2zero(REFTYPE, ri.data)
+    anyna = findfirst(refs, zero(REFTYPE)) > 0


Just any(iszero, refs)

nalimilan · 2017-09-16T09:15:48Z

src/convert.jl

+end
+
+# converts Vector{Int32} into Vector{R} replacing R_NA_INT32 with 0
+# it's assumed that v fits into R


How safe is that assumption?

The only caller so far is jlvec() method below, and it calculates R to fit all the factor levels.

nalimilan · 2017-09-16T09:19:58Z

src/convert.jl

-        # FIXME remove Any type assertion workaround
-        DataFrame(Any[data(col) for col in rl.data], map(identifier, names(rl)))
+        # FIXME add forceNullable option to control whether always convert to Union{T, Null}
+        DataFrame(Any[jlvec(col, true) for col in rl.data], identifier.(names(rl)))


DataFrames no longer automatically converts columns to nullable, probably better keep them non-nullable when there are no nulls.

nalimilan · 2017-09-16T09:23:54Z

src/io/XDRIO.jl

@@ -25,7 +25,11 @@ function readfloatorNA(io::XDRIO, n::RVecLength)
    reinterpret(Float64, map!(ntoh, v, v))
 end

-readuint8(io::XDRIO, n::RVecLength) = readbytes(io.sub, n)
+function readuint8(io::XDRIO, n::RVecLength)


Wouldn't read(io, UInt8, n) be enough?

nalimilan · 2017-09-18T12:13:59Z

src/convert.jl

-    na_mask = namask(rv)
-    anyna = any(na_mask)
-    if force_nullable || anyna
+function jlvec(::Type{T}, rv::RVEC, forceNullable::Bool=true) where T


force_nullable was a more Julian name. Same below.

nalimilan · 2017-09-18T12:18:04Z

src/convert.jl

        res = convert(Vector{Union{T,Null}}, rv.data)
        if anyna
-            @inbounds res[na_mask] = null
+            for (i,x) in enumerate(rv.data)
+                isna(x) && @inbounds res[i] = null


I think putting @inbounds before for will generate more efficient code since it will also apply to enumerate.

Probably you're right, and I'll fix it.
But it's natural to assume that for x in rv.data would do @inbounds access, and so should do for (i,x) in enumerate(rv.data). Maybe something could be tweaked in the Base.

Yes, that's definitely how it should work, but currently AFAIK @inbounds still makes a difference.

nalimilan

Before merging we'll have to clean the Travis config and bump the DataFrames dependency to 0.11.

alyst · 2017-09-18T14:33:22Z

@nalimilan Thanks for the review. I've just squashed the fixups into appropriate commits, but the cumulative changeset is identical to what you have just reviewed. So I'll wait until DataFrames 0.11.

alyst · 2017-11-25T22:17:32Z

I'm waiting for JuliaLang/METADATA.jl#12198 to see if this PR works on nightlies.
Then it should be ready to merge if there are no objections.

- switch to Missings.jl/CategoricalArrays.jl - add jlvec() methods handling conversion logic - fix conversion of RLogicalVector into Vector{Bool} - remove DataArrays.jl dependency

(::Type{XXX})(...) -> XXX(...)

reinterpret() returns ReinterpretArray in 0.7 instead of Vector, whereas RVector{} only accepts Vector

allows to fix the result of readcomplex() from ReinterpretArray to Vector on 0.7

alyst force-pushed the update_to_0.6 branch 3 times, most recently from def7de9 to 24fc1bf Compare September 15, 2017 17:30

alyst force-pushed the update_to_0.6 branch from 3d2dfd5 to e969026 Compare September 15, 2017 17:57

nalimilan reviewed Sep 16, 2017

View reviewed changes

nalimilan mentioned this pull request Sep 16, 2017

Port dependencies to Missing and CategoricalArray JuliaData/DataFrames.jl#1232

Closed

alyst changed the title ~~Update to Julia v0.6 & DataFrames v0.10~~ Update to Julia v0.6 & DataFrames v0.11 Sep 18, 2017

nalimilan reviewed Sep 18, 2017

View reviewed changes

nalimilan approved these changes Sep 18, 2017

View reviewed changes

alyst force-pushed the update_to_0.6 branch from 575b9f9 to 1e06c2c Compare September 18, 2017 14:30

alyst force-pushed the update_to_0.6 branch from 1e06c2c to 3e31894 Compare October 30, 2017 09:35

This was referenced Oct 30, 2017

Switch from GZip to CodecZlib #31

Merged

Update to Julia v0.6 #32

Merged

alyst force-pushed the update_to_0.6 branch 2 times, most recently from c313f78 to eaf82ff Compare November 4, 2017 12:47

alyst changed the title ~~Update to Julia v0.6 & DataFrames v0.11~~ Update to DataFrames v0.11 (CategoricalArrays + Nulls) Nov 4, 2017

alyst force-pushed the update_to_0.6 branch from eaf82ff to e964507 Compare November 4, 2017 13:30

alyst force-pushed the update_to_0.6 branch from 00f89f0 to 0a0d2d0 Compare November 25, 2017 18:55

alyst changed the title ~~Update to DataFrames v0.11 (CategoricalArrays + Nulls)~~ Update to DataFrames v0.11 (CategoricalArrays + Missings) Nov 25, 2017

alyst force-pushed the update_to_0.6 branch from 0a0d2d0 to 8dac8c5 Compare November 25, 2017 19:01

alyst mentioned this pull request Nov 25, 2017

Support for R Dates and POSIXct but no support for handling timezone support #35

Merged

nalimilan closed this Nov 25, 2017

nalimilan reopened this Nov 25, 2017

alyst added 3 commits November 26, 2017 01:20

switch to DataFrames 0.11

cbbdeb9

- switch to Missings.jl/CategoricalArrays.jl - add jlvec() methods handling conversion logic - fix conversion of RLogicalVector into Vector{Bool} - remove DataArrays.jl dependency

remove v0.5-isms from ctors

66aeb9b

(::Type{XXX})(...) -> XXX(...)

readfloat[orNA](io): avoid reinterpret()

f8a8b9b

reinterpret() returns ReinterpretArray in 0.7 instead of Vector, whereas RVector{} only accepts Vector

add readfloatorNA!(io, vec::AbstractVector)

73204b7

allows to fix the result of readcomplex() from ReinterpretArray to Vector on 0.7

alyst force-pushed the update_to_0.6 branch from 2168169 to 73204b7 Compare November 26, 2017 00:21

alyst merged commit 7e17dcf into master Nov 26, 2017

alyst deleted the update_to_0.6 branch November 26, 2017 01:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to DataFrames v0.11 (CategoricalArrays + Missings) #28

Update to DataFrames v0.11 (CategoricalArrays + Missings) #28

alyst commented Sep 15, 2017 •

edited

Loading

alyst commented Sep 15, 2017

alyst commented Sep 16, 2017

nalimilan left a comment

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

alyst Sep 16, 2017

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

alyst Sep 16, 2017

nalimilan Sep 16, 2017

alyst Sep 18, 2017

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

alyst Sep 16, 2017

nalimilan Sep 16, 2017

nalimilan Sep 16, 2017

nalimilan Sep 18, 2017

nalimilan Sep 18, 2017

alyst Sep 18, 2017

nalimilan Sep 18, 2017

nalimilan left a comment

alyst commented Sep 18, 2017

alyst commented Nov 25, 2017

Update to DataFrames v0.11 (CategoricalArrays + Missings) #28

Update to DataFrames v0.11 (CategoricalArrays + Missings) #28

Conversation

alyst commented Sep 15, 2017 • edited Loading

alyst commented Sep 15, 2017

alyst commented Sep 16, 2017

nalimilan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nalimilan left a comment

Choose a reason for hiding this comment

alyst commented Sep 18, 2017

alyst commented Nov 25, 2017

alyst commented Sep 15, 2017 •

edited

Loading