Skip to content

logzio/logzio-api-fetcher

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Fetcher to send API data to Logz.io

The API Fetcher offers way to configure fetching data from APIs every defined scrape interval.

Setup

Pull Docker Image

Download the logzio-api-fetcher image:

docker pull logzio/logzio-api-fetcher

Configuration

Create a local config file config.yaml.
Configure your API inputs under apis. For every API, mention the input type under type field:

General API

For structuring custom API calls use type general API with the parameters below.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional the defined url
url The request URL Required -
headers The request Headers Optional {}
body The request body Optional -
method The request method (GET or POST) Optional GET
pagination Pagination settings if needed (see options below) Optional -
next_url If needed to update the URL in the next request based on the last response. Supports using variables (see below) Optional -
next_body If needed to update the body in the next request based on the last response. Supports using variables (see below) Optional -
response_data_path The path to the data inside the response Optional response root
additional_fields Additional custom fields to add to the logs before sending to logzio Optional Add type as api-fetcher
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)

Pagination Configuration Options

If needed, you can configure pagination.

Parameter Name Description Required/Optional Default
type The pagination type (url, body or headers) Required -
url_format If pagination type is url, configure the URL format used for the pagination. Supports using variables (see below). Required if pagination type is url -
update_first_url True or False; If pagination type is url, and it's required to append new params to the first request URL and not reset it completely. Optional if pagination type is url False
headers_format If pagination type is headers, configure the headers format used for the pagination. Supports using variables (see below). Required if pagination type is headers -
body_format If pagination type is body, configure the body format used for the pagination. Supports using variables (see below). Required if pagination type is body -
stop_indication When should the pagination end based on the response. (see options below). Optional (if not defined will stop on max_calls) -
max_calls Max calls that the pagination can make. (Supports up to 1000) Optional 1000

Pagination Stop Indication Configuration

Parameter Name Description Required/Optional Default
field The name of the field in the response body, to search the stop indication at Required -
condition The stop condition (empty, equals or contains) Required -
value If condition is equals or contains, the value of the field that we should stop at Required if condition is equals or contains -

Using Variables

Using variables allows taking values from the response of the first request, to structure the request after it.
Mathematical operations + and - are supported, in order to add or reduce a number from the variable value.

Use case examples for variable usage:

  1. Update a date filter at every call
  2. Update a page number in pagination

To use variables:

  • Wrap the variable name in curly brackets
  • Provide the full path to that variable in the response
  • Add res. prefix to the path.

Example: Say this is my response:

{
  "field": "value",
  "another_field": {
    "nested": 123
  },
  "num_arr": [1, 2, 3],
  "obj_arr": [
    {
      "field2": 345
    },
    {
      "field2": 567
    }
  ]
}

Paths to fields values are structured like so:

  • {res.field} = "value"
  • {res.another_field.nested} = 123
  • {res.num_arr.[2]} = 3
  • {res.obj_arr.[0].field2} = 345

Using the fields values in the next_url for example like so:

next_url: https://logz.io/{res.field}/{res.obj_arr[0].field2}

Would update the URL at every call to have the value of the given fields from the response, in our example the url for the next call would be:

https://logz.io/value/345

And in the call after it, it would update again according to the response and the next_url structure, and so on.

OAuth API

For structuring custom OAuth calls use type oauth API with the parameters below.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional the defined url
token_request Nest here any detail relevant to the request to get the bearer access token. (Options in General API) Required -
data_request Nest here any detail relevant to the data request. (Options in General API) Required -
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
additional_fields Additional custom fields to add to the logs before sending to logzio Optional Add type as api-fetcher
Azure Graph

For Azure Graph, use type azure_graph with the below parameters.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional azure api
azure_ad_tenant_id The Azure AD Tenant id Required -
azure_ad_client_id The Azure AD Client id Required -
azure_ad_secret_value The Azure AD Secret value Required -
date_filter_key The name of key to use for the date filter in the request URL params Optional createdDateTime
data_request.url The request URL Required -
additional_fields Additional custom fields to add to the logs before sending to logzio Optional -
days_back_fetch The amount of days to fetch back in the first request Optional 1 (day)
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
Azure Mail Reports

For Azure Mail Reports, use type azure_mail_reports with the below parameters.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional azure api
azure_ad_tenant_id The Azure AD Tenant id Required -
azure_ad_client_id The Azure AD Client id Required -
azure_ad_secret_value The Azure AD Secret value Required -
start_date_filter_key The name of key to use for the start date filter in the request URL params. Optional startDate
end_date_filter_key The name of key to use for the end date filter in the request URL params. Optional EndDate
data_request.url The request URL Required -
additional_fields Additional custom fields to add to the logs before sending to logzio Optional -
days_back_fetch The amount of days to fetch back in the first request Optional 1 (day)
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
Azure General API

For structuring custom general Azure API calls use type azure_general API with the parameters below.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional azure api
azure_ad_tenant_id The Azure AD Tenant id Required -
azure_ad_client_id The Azure AD Client id Required -
azure_ad_secret_value The Azure AD Secret value Required -
data_request Nest here any detail relevant to the data request. (Options in General API) Required -
additional_fields Additional custom fields to add to the logs before sending to logzio Optional -
days_back_fetch The amount of days to fetch back in the first request Optional 1 (day)
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
Cloudflare

For Cloudflare API, use type as cloudflare.
By default cloudflare API type:

  • has built in pagination settings
  • sets the response_data_path to result field.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional the defined url
cloudflare_account_id The CloudFlare Account ID Required -
cloudflare_bearer_token The Cloudflare Bearer token Required -
url The request URL Required -
next_url If needed to update the URL in next requests based on the last response. Supports using variables (see General API) Optional -
additional_fields Additional custom fields to add to the logs before sending to logzio Optional -
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
pagination_off True if builtin pagination should be off, False otherwise Optional False
1Password

By default 1password API type has built in pagination settings and sets the response_data_path to items field.

Configuration Options

Parameter Name Description Required/Optional Default
name Name of the API (custom name) Optional the defined url
onepassword_bearer_token The 1Password Bearer token Required -
url The request URL Required -
method The request method (GET or POST) Optional GET
additional_fields Additional custom fields to add to the logs before sending to logzio Optional -
days_back_fetch The amount of days to fetch back in the first request. Applies a filter on 1password start_time parameter. Optional -
scrape_interval Time interval to wait between runs (unit: minutes) Optional 1 (minute)
onepassword_limit 1Password limit for number of events to return in a single request (allowed range: 100 to 1000) Optional 100
pagination_off True if builtin pagination should be off, False otherwise Optional False

And your logzio output under logzio:

Parameter Name Description Required/Optional Default
url The logzio Listener address Optional https://listener.logz.io:8071
token The logzio shipping token Required -

Example

apis:
  - name: random
    type: general
    additional_fields:
      type: random
    ...
  - name: another api
    type: oauth
    additional_fields:
      type: oauth-api
    ...
logzio:
  url: https://listener-ca.logz.io:8071  # for us-east-1 region delete url param (default)
  token: <<SHIPPING_TOKEN>>

Run The Docker Container

In the path where you saved your config.yaml, run:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher 

Change logging level

The default logging level is INFO. To change it, add --level flag to the command:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher \
--level DEBUG

Available Options: INFO, WARN, ERROR, DEBUG

Stopping the container

When you stop the container, the code will run until completion of the iteration. To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop command:

docker stop -t 30 logzio-api-fetcher

Changelog:

  • 0.2.2:
    • Resolve Azure mail reports bug
  • 0.2.1:
    • Add 1Password Support
    • Add next_body support to allow more customization in general settings
    • Support integers and boolean as values in pagination 'equals' stop condition
  • 0.2.0:
    • Breaking changes!!
      • Deprecate configuration fields:
        • credentials, start_date_name, end_date_name, json_paths, settings.days_back_fetch
        • settings.time_interval >> scrape_interval
        • OAuth Changes:
          • token_http_request >> token_request
          • data_http_request >> data_request
        • All APIs now nested in the config under apis and supported types are:
          • general, oauth, azure_general, azure_graph, azure_mail_reports, cloudflare.
      • Upgrade General API settings to allow more customization
        • Adding Authorization Header is required, if needed, due to credentials field deprecation.
        • Adding Content-Type may be required based on the API
      • Deprecate Cisco SecureX
    • New Features!!
      • Add Pagination support
      • Add variables support in next_url and pagination settings
        • Supports + and - mathematical operations on values
        • Supports arrays
      • Upgrade python version 3.9 >> 3.12
      • Add Cloudflare Support
      • Add option to control logging level with --level flag.
  • 0.1.3:
    • Support another date format for azure_graph.
  • 0.1.2:
    • Support another date format for security_alerts.
  • 0.1.1:
    • Bug fix for security_alerts task fails on second cycle.
  • 0.1.0:
    • Added azure_mail_reports type.
  • 0.0.6:
    • Improved documentation.
    • Added error log.
  • 0.0.5:
    • Bug fix for azure_graph task fails on second cycle.
    • Changed start date filter mechanics for auth_api.
  • 0.0.4:
    • Bug fix for azure_graph task fails when trying to get start date from file.