Fetch instead of pull?

[malhoudmouad]

Question

Hey,

I know it's not the standard or the common practice, but is there a way to fetch/pull from the agregator (agregator -> multiple agents) instead of having agents pushing (multiples agents -> agregator) ?

In a context where security policy inforces that the domain is the one that goes to reach outside (instead of the outside reaching to the inside of the network)..

Thanks

Vector Config

No response

Vector Logs

No response

[thomasqueirozb]

Hi @malhoudmouad, does the http_client source fit your use case? It will make an HTTP request every scrape_interval_secs (15 seconds by default).

[malhoudmouad]

I'm trying to wrap my head around how this would work for our setup. In our use case, our applications write logs directly to flat text files on the local disk.

Doesn't the http_client source only work if you already have a REST API or an HTTP server running to serve these logs?

If I understand correctly, wouldn't this mean we'd have to either build our own custom API just to read and serve these local files, or expose our log directories over HTTP using a web server (which sounds like a major security risk)?

[malhoudmouad]

Hey ! Any news about this ?

[thomasqueirozb]

Doesn't the http_client source only work if you already have a REST API or an HTTP server running to serve these logs?

Yes, you'd need to implement some http queuing system for this to work. This is not a scenario we support out of the box.

You may want to consider other alternatives that may or may not work.

Even with ingress completely disabled, a protocol where the agent initiates the connection may still work. Once a connection is established, data should be able to flow in both directions - otherwise HTTP wouldn't work for example. It is possible that another protocol that requires a connection to be established beforehand can work. You can try out the kafka source.

Alternatively you can have a separate process reach out and dump contents into a file or stdin or similar and use the other sources to ingest that data.