Skip to content

Commit

Permalink
Merge branch 'develop' into hardcode_host
Browse files Browse the repository at this point in the history
  • Loading branch information
@mdazam1942
mdazam1942 committed Mar 30, 2023
2 parents 048fda1 + 4b85c73 commit 7188432
Show file tree
Hide file tree
Showing 3 changed files with 281 additions and 12 deletions.
248 changes: 240 additions & 8 deletions OVERVIEW.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -470,11 +470,11 @@ An asynchronous data source will typically return a search ID supplied by the AP

### Status

Uses the data source API to look up the query status based on the `search_id` that is returned from the query call. This is only used for asynchronous data sources where the results are not returned right after making a query call.
Uses the data source API to look up the query status based on the `search_id` that is returned from the query call. This is only used for asynchronous data sources where the results are not returned right after making a query call. If the connector supports, you can specify `metadata` parameter which may contain extra information to make the status api call.

#### CLI Command

`stix-shifter transmit <MODULE NAME> '<CONNECTION OBJECT>' '<CONFIGURATION OBJECT>' status <SEARCH ID>`
`stix-shifter transmit <MODULE NAME> '<CONNECTION OBJECT>' '<CONFIGURATION OBJECT>' status <SEARCH ID> <METADATA(optional)>`

#### OUTPUT:

Expand All @@ -484,17 +484,53 @@ The status can be one of: `COMPLETED`, `ERROR`, `CANCELLED`, `TIMEOUT`, or `RUNN

### Results

Uses the data source API to fetch the query results based on the search ID, offset, and length.
Uses the data source API to fetch the query results based on the search ID, offset, and length.

If the connector supports, you can specify `metadata` parameter which may contain extra information to fetch the next batch of results from the datasource. This is a recomended parameter for the datasource that supports pagination.

#### CLI Command

`stix-shifter transmit <MODULE NAME> '<CONNECTION OBJECT>' '<CONFIGURATION OBJECT>' results <SEARCH ID> <OFFSET> <LENGTH>`
`stix-shifter transmit <MODULE NAME> '<CONNECTION OBJECT>' '<CONFIGURATION OBJECT>' results <SEARCH ID> <OFFSET> <LENGTH> <METADATA(optional)>`

The `OFFSET` and `LENGTH` control what pages/rows of data are returned in the query results.

#### OUTPUT:

`{'success': True, 'data': [<QUERY RESULTS>]}`

The `OFFSET` and `LENGTH` control what pages/rows of data are returned in the query results.
#### OUTPUT(with metadata):

`{'success': True, 'data': [<QUERY RESULTS>], 'metadata': <metadata values>}`

#### Example:
```
{
"success": true,
"data": [
{
"event": {
"securityEvent": {
"eventTimestamp": "2022-06-13T14:36:54.216539700Z",
"eventType": "FILE_CREATION",
"vendorName": "Microsoft",
"productEventType": "DeviceFileEvents",
"ingestedTimestamp": "2022-06-13T15:36:26.275010Z"
},
"securityResult": [
{
"summary": "FileCreated",
"category": "alert"
}
]
}
}
],
"metadata": {
"result_count": 2,
"next_page_token": "CgwIlqLjoAYQ2NfggwESCwiGl52VBhC0xKB"
}
}
```

### Results as STIX

Expand Down Expand Up @@ -554,6 +590,10 @@ You can redirect the output of your CLI command to a file to save the STIX resul

`stix-shifter execute <TRANSMISSION MODULE NAME> <TRANSLATION MODULE NAME> '<STIX IDENTITY OBJECT>' '<CONNECTION OBJECT>' '<CONFIGURATION OBJECT>' '<STIX PATTERN>' > results.json`

### OUTPUT:

A bundle of STIX objects

## Modules

The `modules` command will return a JSON of the existing connectors along with their dialects and supported languages that are used in query translation.
Expand All @@ -562,7 +602,7 @@ The `modules` command will return a JSON of the existing connectors along with t

`python main.py modules`

returns
#### output
```
{
"qradar": {
Expand All @@ -589,12 +629,204 @@ returns
}
```

This command can also be used to get the dialects of a specific connector.

`python main.py modules <module name>`

### CLI Command

`python main.py modules qradar`

#### output
```
{
"qradar": {
"flows": {
"language": "stix",
"default": true
},
"events": {
"language": "stix",
"default": true
},
"aql": {
"language": "aql",
"default": true
}
}
}
```

In the above example, the QRadar connector can use three dialects: `flows`, `events`, and `aql`. When a connector only has a `default` dialect, such as with Security Advisor, only one dialect is used by the connector. Most dialects will use the `stix` language since they translate STIX patterns into native queries. QRadar's `aql` dialect uses the `aql` language since it is meant to accept an AQL query rather than a STIX pattern. See the [QRadar connector README](stix_shifter_modules/qradar/README.md) for more information on AQL passthrough.

### OUTPUT:
## configs

A bundle of STIX objects
The `configs` command returns the configuration pararmetes of the existing connectors. It basically returns a JSON of the existing connectors along with their connections and configuation objects that are specified in config.json.

### CLI Command

`python main.py configs`
#### output
```
{
"alertflex": {
"connection": {
"type": {
"type": "connectorType",
"displayName": "Alertflex"
},
"options": {
"type": "fields",
"async_call": {
"type": "text",
"hidden": true,
"optional": true
},
"result_limit": {
"default": 10000,
"min": 1,
"max": 500000,
"type": "number",
"previous": "connection.resultSizeLimit"
},
"time_range": {
"default": 5,
"min": 1,
"max": 10000,
"type": "number",
"previous": "connection.timerange",
"nullable": true
},
.....
}
}
}
}
```

Specifying the connector module name will return the configuration parameters of a specific connector.

### CLI Command

`python main.py configs `
#### output
```
{
"qradar": {
"connection": {
"type": {
"type": "connectorType",
"displayName": "IBM\u00ae QRadar and QRadar On Cloud",
"group": "qradar"
},
"options": {
"type": "fields",
"async_call": {
"type": "text",
"hidden": true,
"optional": true
},
"result_limit": {
"default": 10000,
"min": 1,
"max": 500000,
"type": "number",
"previous": "connection.resultSizeLimit"
},
"time_range": {
"default": 5,
"min": 1,
"max": 10000,
"type": "number",
"previous": "connection.timerange",
"nullable": true
},
"timeout": {
"default": 30,
"min": 1,
"max": 60,
"hidden": true,
"type": "number",
"previous": "connection.timeoutLimit"
},
"dialects": {
"type": "array",
"hidden": true,
"optional": true
},
"language": {
"type": "string",
"default": "stix",
"optional": true,
"hidden": true
},
"validate_pattern": {
"type": "boolean",
"optional": true,
"hidden": true,
"previous": "connection.validate_pattern",
"default": false
},
"stix_validator": {
"type": "boolean",
"default": false,
"optional": true,
"hidden": true,
"previous": "connection.stix_validator"
},
"mapping": {
"type": "json",
"optional": true,
"previous": "connection.mapping"
},
"unmapped_fallback": {
"type": "boolean",
"default": true,
"optional": true,
"hidden": true
},
"stix_2.1": {
"type": "boolean",
"default": false,
"optional": true,
"hidden": true
}
},
"host": {
"type": "text",
"regex": "^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9_:/\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9_:/\\-]*[A-Za-z0-9])$"
},
"port": {
"type": "number",
"default": 443,
"min": 1,
"max": 65535
},
"help": {
"default": "data-sources-qradar.html",
"type": "link"
},
"sni": {
"type": "text",
"optional": true
},
"selfSignedCert": {
"type": "password",
"optional": true
}
},
"configuration": {
"auth": {
"type": "fields",
"sec": {
"type": "password",
"previous": "configuration.auth.SEC"
}
}
}
}
}
```
## Limitations

STIX-Shifter has limitations on the length of a pattern that can be translated into a native query. As the pattern length increases, the translation time increases exponentially due to how ANTLR 4 parses the pattern. See [STIX-Shifter Limitations](adapter-guide/stix-shifter-limitations.md) for more details.
Expand Down
21 changes: 20 additions & 1 deletion adapter-guide/develop-transmission-module.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,21 @@ Results from the data source need to be returned as an array of JSON objects bef

For asynchronous sources, the search id that gets passed into the status, delete, and results methods is the ID returned by the data source when making the query API call. This is used to keep track of the original query, allowing the status and results to be fetched. However, in the case a synchronous data source, the search id is the entire query string; this is what gets passed into the results and delete methods.

### Returning metadata in the return object

Additional values can be returned as a metadata paremeter in the status and results return object. The data type of the `metadata` parameter can be anything based on the requirements of the connector. The recomended types are python dictionary and string. Ideal use case for this paramater is pagination query. For example, if the connector needs to store the next page token or url and the previous results count to fetch next batch of results from the datasource then the metadata can look like this:

```
{
"result_count": 1,
"next_page_token": "CgwImdHioAYQqKmUuQMSDAiHl52VBhD8g4"
}
```
Here's an example of the return object of the results with metadata parameter:

`{'success': True, 'data': [<QUERY RESULTS>], 'metadata': <metadata values>}`


[Back to top](#create-a-transmission-module)

## Step 4. Edit the error mapper file
Expand Down Expand Up @@ -143,7 +158,11 @@ The connection and configuration objects are explained in the transmission secti
```

2. You can set the offset and length command line arguments to 1.
3. Visually confirm that query results are returned as JSON objects. These results can be compared to what is returned when running the query string used in test C directly on the data source API, either through a UI or the CLI.
3. Optionally, you can set the metadata parameter if the connector supports it. Ideally used for pagination query.
```
python main.py transmit abc '{"host":"www.example.com", "port":1234}' '{"auth": {"username": "some_user_name", "password": "some_password"}}' results "<Query UUID from query test>" <Offset Integer> <Length Integer> '<metadata>'
```
4. Visually confirm that query results are returned as JSON objects. These results can be compared to what is returned when running the query string used in test C directly on the data source API, either through a UI or the CLI.

### Test the transmission **delete** method.

Expand Down
Loading

0 comments on commit 7188432

Please sign in to comment.