[General] Ingestion via Vector

[jorgehermo9]

What would you like to discuss?

Altought it was briefly discussed at #1632 (comment), I would like to be more involved in ParadeDB and want to discuss this and help on it if you want.

At Vector I recently addressed the PostgreSQL sink feature vectordotdev/vector#15765 via vectordotdev/vector#21248 and will land vector 0.46.0. I didn't tested it yet with ParadeDB (It does work with regular postgres), but I think it would be interesting to test how it behaves with ParadeDB (because maybe it would work out-of-the-box right now). Vector's postgres sink allows to ingest metrics, logs and traces from a lot of different sources which includes kafka, s3, prometheus...

I think this would allow to ingest data from a lot of sources into ParadeDB. I'm opening this discussion to know if this is interesting enough, and to suggest opening a tracking issue for this, and if this works, maybe it would be worth to document it into ParadeDB docs so people knows that can ingest from there? (very similar to #2175)

There is currently insterest from Timescale vectordotdev/vector#939 and RisingWave vectordotdev/vector#21308 to have a Vector integration.

I would love to hear your opinions about this.

Full Name:

Jorge Hermo

Affiliation:

None

[philippemnoel]

Hi @jorgehermo9, thank you for this issue. We have a tracking issue here #1632.

That being said, this is more of a documentation issue. ParadeDB is "just Postgres" and if there is a Vector sink for Postgres, it will work with ParadeDB. We would love your help testing and documenting this feature.

[jorgehermo9]

Hi @philippemnoel

Thank you for your response. I just did a quick test and it worked!

Although in my test I just used a bm25 index and searched over it. It would be interesting to ingest metrics & oltp data and test it too, but I'm pretty sure it would work out of the box. I can take a further look these weeks and propose a PR documentating this.
This is the configuration I used:

ParadeDB

> docker run -e POSTGRES_PASSWORD=password -p 5432:5432 paradedb/paradedb
> psql -h localhost -U postgres
# CREATE DATABASE test;
# \c test

CREATE TABLE logs (id UUID, message TEXT, timestamp TIMESTAMPTZ);
CREATE INDEX search_idx ON logs USING bm25 (id,message) WITH (key_field='id');

Vector Config (currently, postgres sink is only supported on master branch or nightly builds, not released yet)

[sources.demo_logs]
type = "demo_logs"
format = "apache_common"

[transforms.remap]
type = "remap"
inputs = ["demo_logs"]
source = """
.id = uuid_v7()
"""
[sinks.postgres]
type = "postgres"
endpoint = "postgresql://postgres:password@localhost:5432/test"
table = "logs"
inputs = ["remap"]

Test

SELECT id,message FROM logs WHERE message @@@ 'put' ORDER BY message LIMIT 5;

[philippemnoel]

That looks awesome. Thank you for doing this! Excited to see your PR