Add PrettyTables.jl as an alternative backend for display in DataFrames.jl

[bkamins]

It would be nice to add PrettyTables.jl as an alternative display backend for DataFrames.jl to the one we currently have (probably not to replace it in the short term, as it would be too hard to ensure a smooth transition).

If we want to do it the steps would be the following:

1. Test package load time impact
2. choose appropriate defaults for printing
3. prepare backend switching mechanics (in particular mapping kwargs where appropriate)
4. ensure all types that currently use custom printing (datata frame, DataFrameRow, GroupedDataFrame, DataFrameRows, DataFrameColumns are correctly handled)
5. decide if we re-export PrettyTables.jl from DataFrames.jl

CC @ronisbr

[ronisbr]

Just to report the progress, I am trying to, at least, print the common structures just like DataFrames is currently printing. Right now, I am having a problem with undefined references when accessing DataFrames using Tables.row. I can easily solve this if I assume that the table is a DataFrame. However, I want to make something more general that works for any Tables.jl. I asked for help here:

https://discourse.julialang.org/t/help-to-check-if-an-element-in-a-tables-jl-is-defined/44110

After that, I think I just need to add option to adjust the vertical cropping. AFAIK, this would be the last feature I really need to implement. The rest will be only configurations and tweaks using existing interface (I hope!).

[bkamins]

I have commented on the issue there and for completeness I repost here:

@quinnj - I think for Tables.row result we should add a requirement that isassigned method should be defined for it.

Also in Tables.jl the Base.show(io::IO, x::T) where {T <: RorC} method has show(io, NamedTuple(x)) line which throws an error on #undef.

[pdeffebach]

I don't really see why data frames needs to implement printing at all. Unlike joins etc. where the storage of the columns matters, i think all of this could be relegated to a Tables.jl-type package.

Would be great to excise this huge chunk of code!

[bkamins]

This is the ultimate objective and this package is PrettyTables.jl. The only problem is that currently DataFrames.jl has some functionality that PrettyTables.jl has to catch up with (but it seems to be very close towards this goal).

Then we will have a choice:

1. allow switching of backends (safe approach)
2. drop legacy printing (radical approach)

I would prefer to do 2, but there is a risk that user's code depends on how printing is done currently and it would be very breaking.

So probably the way we will approach it will be in three stages:

1. add PrettyTables.jl as a secondary backend (experimental)
2. switch to make PrettyTables.jl a default backend
3. remove printing code from DataFrames.jl to LegacyDataFramesShow.jl or something like this and allow to optionally load this backend

Anyway - we will have time to discuss this when @ronisbr tells us that he is ready for the stress tests of PrettyTables.jl against:

  show([io::IO,] df::AbstractDataFrame;
       allrows::Bool = !get(io, :limit, false),
       allcols::Bool = !get(io, :limit, false),
       allgroups::Bool = !get(io, :limit, false),
       splitcols::Bool = get(io, :limit, false),
       rowlabel::Symbol = :Row,
       summary::Bool = true,
       eltypes::Bool = true)

[ronisbr]

So probably the way we will approach it will be in three stages:

1. add PrettyTables.jl as a secondary backend (experimental)
2. switch to make PrettyTables.jl a default backend
3. remove printing code from DataFrames.jl to LegacyDataFramesShow.jl or something like this and allow to optionally load this backend

If I can provide an opinion, this is the perfect approach for me! I have been studying the printing system of DataFrames and it is very complex. There are a lot of cases that we can break things easily in the first attempt to move to PrettyTables.jl :)

Anyway - we will have time to discuss this when @ronisbr tells us that he is ready for the stress tests of PrettyTables.jl against:

I am getting there! I found that problem with #undef and some minor things. I just need to reach a point that we need no other feature inside PrettyTables.jl, but a correct configuration for it.

[pdeffebach]

Thanks for the update to make prettytables performant with tables! Thats a lot of progress right there as well.

[bkamins]

@ronisbr - thank you for all your work on this! I will open an issue for #undef in Tables.jl

[ronisbr]

Ok! Now we can start to do some tests and I need help to tweak everything. I propose to start with the Text backend before trying to make HTML and LaTeX work. Using master branch of PrettyTables.jl and this code snippet:

function h1_f(data,i,j)
    try
        return ismissing(data[i,j]) ||
            data[i,j] == nothing ||
            typeof(data[i,j]) <: Union{AbstractDataFrame, GroupedDataFrame,
                                       DataFrames.DataFrameRow,
                                       DataFrames.DataFrameRows,
                                       DataFrames.DataFrameColumns}
    catch e
        if isa(e, UndefRefError)
            return true
        else
            rethrow(e)
        end
    end
end

h1 = Highlighter(h1_f, crayon"dark_gray")

function f1(v,i,j)
    if typeof(v) <: Union{AbstractDataFrame, GroupedDataFrame,
                          DataFrames.DataFrameRow, DataFrames.DataFrameRows,
                          DataFrames.DataFrameColumns}
        str = sprint(print, v, context = :compact => true)
        str = split(str, '\n')[1]
        return str
    elseif ismissing(v)
        return "missing"
    elseif v == nothing
        return ""
    else
        return v
    end
end

function print_dataframe(df)
    sch = Tables.schema(df)

    names = reshape( [sch.names...], (1,:) )
    types = DataFrames.compacttype.(reshape( [sch.types...], (1,:) ))

    pretty_table(df,
                 vcat(names,types),
                 alignment = :l,
                 formatters = (f1,),
                 highlighters = (h1,),
                 maximum_columns_width = 30,
                 row_number_alignment = :l,
                 show_row_number = true,
                 tf = dataframe)
end

I am managed to get a lot of printing like the default output of DataFrames (I am already limiting the column width size to 30). For example:

julia> df = DataFrame(A=Int64.(1:9), B = Vector{Any}(undef, 9));

julia> df.B[1:8] = [df, # DataFrame
                        df[1,:], # DataFrameRow
                        view(df,1:1, :), # SubDataFrame
                        eachrow(df), # DataFrameColumns
                        eachcol(df), # DataFrameRows
                        groupby(df, :A),missing,nothing] # GroupedDataFrame;

julia> df = DataFrame(A = Int64[1:4;], B = ["x\"", "∀ε>0: x+ε>x", "z\$", "A\nC"],
                         C = Float32[1.0, 2.0, 3.0, 4.0]);

Hence, I think this is a good baseline to start. Now, we need to tweak the edge cases and, for this, I need help because I do not know what they are :D

[bkamins]

OK - I will try to list the cases.

[bkamins]

I think apart from what you covered this would be good to check:

DataFrame(x=[pi,
big(pi),
Rational(float(pi)),
Rational(big(pi)),
:pi,
Symbol(1),
Symbol(""),
Symbol("a\bx"),
"a\bx",
"̂", # note this is " then \hat then <TAB> then "
String([0x7c,0xf8, 0x82, 0x86, 0xe1, 0xf4, 0xc4, 0xa6, 0xd0, 0x40]),
'😄'
])

[bkamins]

• please test Markdown values (this is not merged yet but as you know will be added soon)

[ronisbr]

Good! For Markdown, it will not work now. We need to wait @NicholasWMRitchie update his PR because some conflicts.

[ronisbr]

@bkamins for your last example, I am getting this with DataFrames:

Is this expected?