[SPARK-41685][UI] Support Protobuf serializer for the KVStore in History server #39202
Conversation
| @@ -111,7 +122,7 @@ private[spark] object KVUtils extends Logging { | |||
| // The default serializer is slow since it is using JSON+GZip encoding. | |||
| Some(new KVStoreProtobufSerializer()) | |||
| } else { | |||
| None | |||
| Some(serializerForHistoryServer(conf)) | |||
| } | |||
There was a problem hiding this comment.
QQ: If it is always non empty, change from Option ?
There was a problem hiding this comment.
Thanks, I just did some refactoring. Please take another look
docs/monitoring.md
Outdated
| <td> | ||
| Serializer for writing/reading in-memory UI objects to/from disk-based KV Store; JSON or PROTOBUF. | ||
| JSON serializer is the only choice before Spark 3.4.0, thus it is the default value. | ||
| PROTOBUF serializer is fast and compact, and it is the default serializer for disk-based KV store of live UI. |
There was a problem hiding this comment.
Users cannot leverage/access the live db right ? (Unlike history db, which is used across restarts).
Do we want to expose the impl details ? If we change it tomorrow to something else, it should be completely transparent to users.
gengliangwang
changed the title
| } | ||
|
|
||
| val LOCAL_STORE_SERIALIZER = ConfigBuilder("spark.history.store.serializer") | ||
| .doc("Serializer for writing/reading in-memory UI objects to/from disk-based KV Store; " + |
There was a problem hiding this comment.
disk-based KV Store; , ; or :?
There was a problem hiding this comment.
It follows
val HYBRID_STORE_DISK_BACKEND = ConfigBuilder("spark.history.store.hybridStore.diskBackend")
.doc("Specifies a disk-based store used in hybrid store; LEVELDB or ROCKSDB.")
Will this be supported in the future or has it been supported now? |
gengliangwang
changed the title
|
I think SHS can read the written RocksDB after this PR, if the file path/DB backend/serializer configurations are set properly. |
| .doc("Serializer for writing/reading in-memory UI objects to/from disk-based KV Store; " + | ||
| "JSON or PROTOBUF. JSON serializer is the only choice before Spark 3.4.0, thus it is the " + | ||
| "default value. PROTOBUF serializer is fast and compact, and it is the default " + | ||
| "serializer for disk-based KV store of live UI.") |
There was a problem hiding this comment.
I'm a bit confused about this config. SHS only reads the rocksdb files, but not write them. So it should support whatever format of the rocksdb values. Is there a way to encode the format in rocksdb as well?
There was a problem hiding this comment.
SHS writes RocksDB files during replaying event logs. The default serializer is JSON+GZIP, which is slower than the Protobuf serializer. Imagine that there are 300GB of event logs to replay...
|
+CC @thejdeep, @shardulm94 |
|
@techaddict @mridulm @LuciferYang @cloud-fan thanks for the review. |
What changes were proposed in this pull request?
Support configurable(
spark.history.store.serializer) serializer for the KVStore in History server. The value can beJSONorPROTOBUF.Why are the changes needed?
Support the fast and compact protobuf serializer in Spark history server(SHS), so that:
Does this PR introduce any user-facing change?
Yes, introduce a new configuration
spark.history.store.serializerfor setting the serializer for the KVStore in History server, which can beJSONorPROTOBUFHow was this patch tested?
UT