[Feature Request] Array operation callbacks

[alexcjohnson]

In a lot of cases you need to make a relatively small change to a large array. For example, adding a row to a table, or expanding one row into multiple rows as in drill-down interactions; adding a new section to a page, or new options to a dropdown menu. Any array prop might benefit from this - children, DataTable.data, Checklist.options etc...

The way we do this today is to provide the whole array as State, modify it, and return the whole modified thing as Output. If you could just specify an alteration, it cut down data transfer a huge amount, and possibly improve rendering performance.

Seems like the most general operation we'd need to support is what JS calls Array.splice:

splice(start_index, n_to_remove, subarray_to_add)

That covers append (if we define a start_index for "the end" - null/None?), prepend, extend (what's pre-extend, pretend??), insert, delete, replace, and generic splice == replace-with-a-different-length - at least as long as the modifications are contiguous. If we need to support multi-region splice, we could support all three of those args being arrays.

I'd propose the renderer only support splice itself, which the back end could turn into all the above variants. For an API, how about:

@app.callback(Output("my-table", "data.append"), ...)
def add_row(...):
    return {"id1": val1, "id2": val2, ...}

@app.callback(Output("my-table", "data.extend"), ...)
def add_rows(...):
    return [
        {"id1": val1_1, "id2": val2_1, ...},
        {"id1": val1_2, "id2": val2_2, ...}
    ]

@app.callback(Output("my-table", "data.splice"), ...)
def splice_rows(...):
    new_rows = [
        {"id1": val1_1, "id2": val2_1, ...},
        {"id1": val1_2, "id2": val2_2, ...}
    ]
    return [start_index, n_to_remove, new_rows]

Now, what if you need some information about the existing array in order to figure out what to return? You can store that info separately from the array - and in some cases the info you'll need isn't in the array at all, like if you're looking for new events in a database, you might want to store the server timestamp when the query was last run (don't use a global var for this!!!) which could be stashed in a Store and used as both State and Output. But in other cases you might want info that's already in the array and don't want to duplicate it. Seems to me there's always an out here, so this could be omitted from the initial feature, if it's included at all. But for completeness, here are items that occur to me as possibly useful, with proposed API:

• the array length - as an extreme shorthand for which items are already included
  • State("my-table", "data.length")
• one or more items from the array - the last row, for example, so you know what to load for next
  • State("my-table", "data.slice(-1, 1)")
• some value plucked from each item - the ID of each row, perhaps, so you can
  tell which row to operate on).
  • State("my-table", "data.pluck('id')")

Split out from #475 (Wildcard callbacks) where @chriddyp started discussing this a little - will likely be used together with wildcards but the implementation should be independent.

[chriddyp]

Love this idea 🎉

Looking at the API like data.append, would this ever conflict with an API that would allow for partial updating of nested objects? e.g. only updating figure.layout.

I suppose that could be rewritten as an object operation like figure.mergeDeep if we extended this particular property.operation API.

[Marc-Andre-Rivet]

I wonder if we could generalize and expend on this and have the output be an operation on the output prop, thinking of the Ramda prototype presentation @chriddyp did a while back, before clientside callbacks. The back-end provides a syntax, the renderer reconciles the action on the data instead of data directly.

Providing the API in the BE would make it easier to write and debug than using operation strings.

For example, an input could become
State("my-component", "data", pluck("id")) // (id, prop, transform)
Output("my-component-2, "data") // (id, prop) -> transform

With the callback returning, say: append(datum) or insert(2, datum) or some other more complex series of operations. The renderer would then know to (a) append datum at the end of data or insert at 2, the callback could return different operations (or an array of operations) on the target prop, as needed.

The input/state part could be made to work for free with clientside callbacks, the output would need operations of its own in JS.

I think we could cover both partial props and array operations with a single feature.

[Marc-Andre-Rivet]

Building on the transforms. If components could define special transformations props => derived_prop we might be able to eliminate the derived_* props entirely from the table's API. It would also clear up the confusion about derived vs. non-derived props (readonly / read-write)

We could get something like
Input('my-data-table', DataTable.derived.selected_rows) replacing
Input('my-data-table', 'derived_selected_rows')

DataTable.derived.selected_rows would be a Py facing flag, part of the generated table Python component wrapper that would map to a specific props transform defined in JS