why pulsar sink so much slower than sink kafka？

[littlejoyo]

Hi,my vector configuration is as follows:

[sources.syslog]
type = "syslog"
address = "0.0.0.0:524"
mode = "tcp"
path = "/path/to/socket"

# Parse Syslog logs
[transforms.parse_syslogs]
type = "remap"
inputs = ["syslog"]
source = '''
. |= parse_syslog!(.message)
.timestamp = to_timestamp!(to_unix_timestamp!(.timestamp)+3600*8)
'''
# Print parsed logs to stdout
[sinks.printsys]
type = "pulsar"
inputs = ["parse_syslogs"]
endpoint = "pulsar://10.65.172.58:6650"
topic = "public/default/vector"
encoding.codec = "json"
buffer.type = "memory"
buffer.when_full = "block"
buffer.max_events = 5000000

for example：
A(syslog) -> B（vector） ->C（pulsar）

by vector top can get：
A to B ：60k/s （source/transform）
B to C ：2k/s (pulsar sink)

vector server is 4C 8G centos8，
and if i chance to kafka sink can get：
A to B ：60k/s （source/transform）
B to C ：60k/s (kafka sink)

for example：

[sources.syslog]
type = "syslog"
address = "0.0.0.0:524"
mode = "tcp"
path = "/path/to/socket"

# Parse Syslog logs
[transforms.parse_syslogs]
type = "remap"
inputs = ["syslog"]
source = '''
. |= parse_syslog!(.message)
.timestamp = to_timestamp!(to_unix_timestamp!(.timestamp)+3600*8)
'''
# Print parsed logs to stdout
[sinks.printsys]
type = "pulsar"
inputs = ["parse_syslogs"]
bootstrap_servers = "10.65.172.58:9092/kafka"
topic = "mykafka"
key_field = "user_id"
compression = "none"
encoding.codec = "json"
buffer.type = "memory"
buffer.when_full = "block"
buffer.max_events = 5000000

ps：buffer set 5000000 is more than client syslog events，so is not exist block

I want to know if there is a configuration problem for pulsar sink？
I don't think 2k/s is normal for pulsar sink，and why doesn't pulsar have batch configuration like kafka?
Thank you very much！Hope to receive your help soon！

[neuronull]

Hi @littlejoyo, are the k/s referenced here for event count or for byte size ?

[littlejoyo]

@neuronull event count through vector top

[jszwedko]

Looks like you opened an issue for this too. We can center the discussion there.