[SUPPORT] how fast (total time) to update 1 row?

[tooptoop4]

from slack

me
how fast can u update a single row? including time to trigger spark-submit, unless there is faster way

Sudha
there is a performance page to give you idea - https://hudi.apache.org/docs/performance.html . Hope thats useful.
hudi.apache.orghudi.apache.org
Performance
In this section, we go over some real world performance numbers for Hudi upserts, incremental pull and compare them againstthe conventional alternatives for achieving these tasks.

me
@sudha it doesn't help, I want to know if Hudi can update 1 single row in less than 10 seconds?

Sudha
Not sure if there is a straight answer to your question. Especially when you are considering spark submit trigger times etc. Usually spark-submit uploads jars and if there are many jars, it could take in the order of minutes. It would be useful to create a Github issue what is your exact use case so we can point you to the right set of configs in Hudi to achieve that.

[vinothchandar]

@tooptoop4

including time to trigger spark-submit, unless there is faster way

if you are targetting something like spark-submit, then its not under hudi's control. spark-submit alone can take anywhere from 30 seconds to minutes depending on various factors on your cluster, size of the jar you are uploading etc..

on Hudi upsert it self, using Hudi to upsert 1 row at a time, is not an intended design. I recommend atleast batching input data for a minute or so, and issue upsert as a batch. In other words, Hudi (or anything like Hudi) is not really designed like MySQL.. Hope that helps..

[tooptoop4]

how about via livy could it avoid the 30 seconds? i believe delta promise fast 1 row update via sql

[vinothchandar]

That's for the delta folks to answer :) .. If you are rewriting parquet files or generating new parquet file on each write, there is nothing fundamentally different any other system can do here.. All databases or even data warehouses you are comparing to, have long running servers with some metadata/data loaded into memory, to help with such fast updates..

livy is a long running server.. which already has a spark application running, unlike issuing spark-submit everytime.. ofc if you use livy or zeppelin, that overhead goes away.