Make it easier to work with missing data

[ryancerf]

Useful for working with missing data. Similar to pandas fillna, but for a single column at a time.

Usage:

myDoubleColumn.fillMissingWith(1.0);

I am not sure what the easiest way to do this is right now.

Will send a PR if we think this is worthwhile.

[benmccann]

You should be able to do that today with col.set(NumberPredicates.isMissing, 1.0) . This is just off the top of my head, so the syntax might be slightly off

[ryancerf]

Another method that might be useful for working with missing data is a getOrDefault method on Row.

// Proposed
row.getLongOrDefault("col1", 1L);

It could simply iterating over a table with missing data.

    // Current
    for (Row row : table) {
        long value1 = row.getLong("col1");
        int value2 = row.getInt("col2");
        value1 = LongColumn.valueIsMissing(value1) ? 1L : value1;
        value2 = IntColumn.valueIsMissing(value2) ? 1 : value2;

        // do something with value1 and value 2;
    }

    // Proposed
    for (Row row : table) {
        long value1 = row.getLongOrDefault("col1", 1L);
        int value2 = row.getIntOrDefault("col2", 1);
        // do something with value1 and value2
    }

[lwhite1]

I have mixed feelings on whether it's worth having a new method.
As @benmccann points out, it's not too hard.

col.set(col.isMissing(), 1.0);
will do it.

On the other hand, there's a reason why they made it easier in pandas. I feel like it's customary in data analysis tools and libraries to provide very good support for missing values. I would not be averse to:

Column<?>::setMissing(? value)...
I wouldn't call it "fill" though, I think fill in Tablesaw is more likely to refer to a series of appends than a series of updates. Update methods generally start with set.

[ryancerf]

Using set is fine with me, but I would like to point out we do have a fillWith method on DoubleColumn (I am not sure I like this method. map is a good alternative).

// This method was added recently.
 @Override
    public DoubleColumn fillWith(double d) {
        for (int r = 0; r < size(); r++) {
            set(r, d);
        }
        return this;
    }

[lwhite1]

@ryancerf You're right about fillWith. I forgot how it was implemented.

I think the intent there was to create columns where every value is set in this method, and that it is intended for populating new columns. Although it uses set() instead of append, I think the intent is the same.

I do think set() is more appropriate for a method that is being selective about what it's updating, where fill suggests more of a bulk/batch process.

[lwhite1]

closing. @ryancerf please lmk if you have issues with the resolution.