Add ability to trace mutation form from each data source during query

[tgrabiec]

Knowing mutation form returned by each of the data source touched by a query would be very helpful in debugging issues related to incorrect query results.

The user wouldn't have to share whole sstables with us, so the process would be easier and faster. Sensitive data could be easily removed. We could add an option to do this automatically, removing cell values and obfuscate keys. The user could keep the obfuscation translation map for reference.

It could be reported via CQL tracing and enabled with a new query syntax (like "bypass cache"), e.g.:

cqlsh> tracing on
cqlsh> select * from my_table where ... TRACE MUTATIONS 

\cc @bhalevy

[bhalevy]

Printing the mutation as json seems like a good idea so analyzing it could be automated.

[denesb]

We already have a mutation -> json converter in https://github.com/scylladb/scylla/blob/81e20ceaab8f54938aa77842ee5c9d9e62639432/tools/scylla-sstable.cc#L324.

[avikivity]

When a mutation reader creates subordinate readers, then we could create subordinate trace_state objects that name the subordinate and its parent. This would recreate the reader tree in runtime without much effort.

[denesb]

I'm, thinking a virtual table would be a better fit than tracing. The potential amount of data is huge, and it can be duplicated in all the different data sources, or even multiple times in a single data source. A virtual table naturally lends itself to delivering large amounts of data, and if done right, advanced filtering/redacting should be easily possible.

[denesb]

I opened an RFC PR using the virtual table approach: #14083.

[mykaul]

@bhalevy - this looks like an important improvement to our ability to debug issues - can we push it forward?