empty ByRow test failure in Julia_1.11.0-DEV.1238

[George9000]

Under test/select.jl, several of the empty ByRow tests fail between lines 1579 and 1610.

Errors

empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1579
  Expression: select(df, [] => (ByRow((()->begin
                            #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1579 =#
                            [1, "1"]
                        end)) => AsTable)) == DataFrame()
   Evaluated: 0×2 DataFrame
 Row │ ref        size
     │ GenericM…  Tuple…
─────┴─────────────────── == 0×0 DataFrame

Stacktrace:
 [1] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1579 [inlined]
 [3] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [4] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1580
  Expression: select(df, [] => (ByRow((()->begin
                        #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1580 =#
                        [1, "1"]
                    end)) => [:p, :q]))
    Expected: ArgumentError
  No exception thrown

Stacktrace:
 [1] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1580 [inlined]
 [2] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [3] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1579
  Expression: select(df, [] => (ByRow((()->begin
                            #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1579 =#
                            [1, "1"]
                        end)) => AsTable)) == DataFrame()
   Evaluated: 0×2 DataFrame
 Row │ ref        size
     │ GenericM…  Tuple…
─────┴─────────────────── == 0×0 DataFrame

Stacktrace:
 [1] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1579 [inlined]
 [3] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [4] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1580
  Expression: select(df, [] => (ByRow((()->begin
                        #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1580 =#
                        [1, "1"]
                    end)) => [:p, :q]))
    Expected: ArgumentError
  No exception thrown

Stacktrace:
 [1] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1580 [inlined]
 [2] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [3] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1609
  Expression: select(df, [] => (ByRow((()->begin
                            #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1609 =#
                            [1, "1"]
                        end)) => AsTable)) == DataFrame()
   Evaluated: 0×2 DataFrame
 Row │ ref        size
     │ GenericM…  Tuple…
─────┴─────────────────── == 0×0 DataFrame

Stacktrace:
 [1] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1609 [inlined]
 [3] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [4] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1610
  Expression: select(df, [] => (ByRow((()->begin
                        #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1610 =#
                        [1, "1"]
                    end)) => [:p, :q]))
    Expected: ArgumentError
  No exception thrown

Stacktrace:
 [1] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1610 [inlined]
 [2] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [3] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1609
  Expression: select(df, [] => (ByRow((()->begin
                            #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1609 =#
                            [1, "1"]
                        end)) => AsTable)) == DataFrame()
   Evaluated: 0×2 DataFrame
 Row │ ref        size
     │ GenericM…  Tuple…
─────┴─────────────────── == 0×0 DataFrame

Stacktrace:
 [1] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1609 [inlined]
 [3] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [4] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
empty ByRow: Test Failed at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1610
  Expression: select(df, [] => (ByRow((()->begin
                        #= /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1610 =#
                        [1, "1"]
                    end)) => [:p, :q]))
    Expected: ArgumentError
  No exception thrown

Stacktrace:
 [1] macro expansion
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1610 [inlined]
 [2] macro expansion
   @ ~/applications/julia11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
 [3] top-level scope
   @ ~/Documents/julia/dev/DataFrames/test/select.jl:1555
Test Summary: | Pass  Fail  Total  Time
empty ByRow   |   46     8     54  1.5s
	FAILED: select.jl
LoadError: Some tests did not pass: 46 passed, 8 failed, 0 errored, 0 broken.
in expression starting at /Users/foo/Documents/julia/dev/DataFrames/test/select.jl:1

versioninfo and full test results

[bkamins]

Thank you for reporting. I will look into it.

[bkamins]

I have investigated the issue. It is related to Tables.jl not DataFrames.jl.

The problematic line is:

elseif isconcretetype(WT) && fieldcount(WT) > 0

in fallbacks.jl (line 216).

The change in Julia that causes this is as follows.

On Julia 1.10:

julia> fieldcount(Vector{Any})
0

julia> fieldnames(Vector{Any})
()

On Julia Version 1.11.0-DEV.1346:

julia> fieldcount(Vector{Any})
2

julia> fieldnames(Vector{Any})
(:ref, :size)

I have asked on Slack if the change is intentional or not.

[bkamins]

CC @quinnj - as we need to decide what to do with this.

[bkamins]

The change leading to this problem is JuliaLang/julia#51319

[bkamins]

In general probably we shoudl special case AbstractArray (and document it. As currently we have (Julia 1.10):

julia> Tables.columns(typeof(@view [1,2][:])[])
Tables.CopiedColumns{@NamedTuple{parent::Vector{Vector{Int64}}, indices::Vector{Tuple{Base.Slice{Base.OneTo{Int64}}}}, offset1::Vector{Int64}, stride1::Vector{Int64}}} with 0 rows, 4 columns, and schema:
 :parent   Vector{Int64} (alias for Array{Int64, 1})
 :indices  Tuple{Base.Slice{Base.OneTo{Int64}}}
 :offset1  Int64
 :stride1  Int64

julia> DataFrame(Tables.columns([@view [1,2][:]]))
1×4 DataFrame
 Row │ parent  indices                       offset1  stride1
     │ Array…  Tuple…                        Int64    Int64
─────┼────────────────────────────────────────────────────────
   1 │ [1, 2]  (Base.Slice(Base.OneTo(2)),)        0        1

julia> DataFrame(Tables.columns([[1,2]])) # this is expected
ERROR: ArgumentError: 'Vector{Vector{Int64}}' iterates 'Vector{Int64}' values, which doesn't satisfy the Tables.jl `AbstractRow` interface

[bkamins]

Also, a basic case that is probably unwanted:

julia> Tables.columntable(Vector{Any}[])
(ref = MemoryRef{Any}[], size = Tuple{Int64}[])