Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an api source operator #3630

Merged
merged 7 commits into from Nov 14, 2023
Merged

Add an api source operator #3630

merged 7 commits into from Nov 14, 2023

Conversation

dominiklohmann
Copy link
Member

@dominiklohmann dominiklohmann commented Nov 7, 2023

This operator makes it possible to use Tenzir's REST API without going through the web plugin's locally spun up server. This is mostly motivated by a testing use case: Right now, all of our internal tests use curl to test API endpoints, which can be very hard to test, and always adds an implicit dependency to the web plugin.

The syntax is very simple:

api <endpoint> [<key=value>...]

For example, to create a pipeline in the pipeline manager API:

api /pipeline/create '{"name": "Suricata Import", "definition": "load file /tmp/eve.sock | read suricata | import"}'

The operator parses the API response and returns it as an event, which makes it possible to interact with it just like any other data:

api /pipeline/list
| write yaml

@dominiklohmann dominiklohmann added feature New functionality operator Source, transformation, and sink labels Nov 7, 2023
@dominiklohmann dominiklohmann marked this pull request as ready for review November 7, 2023 15:55
@dominiklohmann dominiklohmann force-pushed the topic/api-operator branch 3 times, most recently from f6562c1 to eb935bb Compare November 8, 2023 17:49
libtenzir/builtins/operators/api.cpp Outdated Show resolved Hide resolved
web/docs/operators/sources/api.md Outdated Show resolved Hide resolved
web/docs/operators/sources/api.md Outdated Show resolved Hide resolved
This operator makes it possible to use Tenzir's REST API without going
through the `web` plugin's locally spun up server. This is mostly
motivated by a testing use case: Right now, all of our internal tests
use `curl` to test API endpoints, which can be very hard to test, and
always adds an implicit dependency to the `web` plugin.

The syntax is very simple:

```
api <endpoint> [<key=value>...]
```

For example, to create a pipeline in the pipeline manager API:

```
api /pipeline/create "name=Suricata Import" "definition=load file /tmp/eve.sock | read suricata | import"
```

The operator parses the API response and returns it as an event, which
makes it possible to interact with it just like any other data:

```
api /pipeline/list
| write yaml
```
This makes it easier to set labels, as interacting with lists of record
parameters was not easily possible with the previous API.
demo-node/setup.bash Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
@rdettai
Copy link
Contributor

rdettai commented Nov 14, 2023

I tested a build of the latest demo node against the platform integration test and it failed, I need a moment to understand why

Copy link
Contributor

@rdettai rdettai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the failing tests on the platform side, it's just that we were using a custom entrypoint that was overriding this new one with copying.

@dominiklohmann dominiklohmann merged commit 467a610 into main Nov 14, 2023
39 of 41 checks passed
@dominiklohmann dominiklohmann deleted the topic/api-operator branch November 14, 2023 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality operator Source, transformation, and sink
Projects
None yet
3 participants