Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long running migration docs #29846

Merged
merged 28 commits into from Oct 12, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
e2642b3
Update WebFileRepository and XSOARWebServer integration files (#29468)
ShirleyDenkberg Sep 20, 2023
8b8aeaa
Update for EDL (#29367)
ShirleyDenkberg Sep 20, 2023
e1f494d
Update TAXII2 Server for EDL (#29358)
ShirleyDenkberg Sep 20, 2023
66bed64
Update EDL for Long running integration in XSOAR 8 (#29127)
ShirleyDenkberg Sep 20, 2023
8b24e4b
Update Generic Webhook and TAXIIServer for EDL (#29356)
ShirleyDenkberg Sep 20, 2023
8ef9a22
Update Packs/EDL/Integrations/EDL/EDL_description.md
ShahafBenYakir Oct 1, 2023
d3dca90
Update Packs/EDL/Integrations/EDL/README.md
ShahafBenYakir Oct 1, 2023
32a0eb3
Update Packs/TAXIIServer/Integrations/TAXII2Server/README.md
ShirleyDenkberg Oct 1, 2023
d7d20a2
Update Packs/TAXIIServer/Integrations/TAXII2Server/README.md
ShirleyDenkberg Oct 1, 2023
9543da5
Update Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server_descr…
ShirleyDenkberg Oct 1, 2023
729fa54
Update Packs/WebFileRepository/Integrations/WebFileRepository/README.md
ShirleyDenkberg Oct 1, 2023
027f492
Update Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server_descr…
ShirleyDenkberg Oct 1, 2023
9c4c7aa
Update Packs/TAXIIServer/Integrations/TAXIIServer/README.md
ShirleyDenkberg Oct 1, 2023
547a204
Update Packs/TAXIIServer/Integrations/TAXIIServer/README.md
ShirleyDenkberg Oct 1, 2023
b27430a
Update Packs/TAXIIServer/Integrations/TAXIIServer/README.md
ShirleyDenkberg Oct 1, 2023
2907c83
Update Packs/TAXIIServer/Integrations/TAXIIServer/README.md
ShirleyDenkberg Oct 1, 2023
13cba65
Update Packs/TAXIIServer/Integrations/TAXIIServer/TAXIIServer_descrip…
ShirleyDenkberg Oct 1, 2023
630d02c
Update Packs/TAXIIServer/Integrations/TAXIIServer/TAXIIServer_descrip…
ShirleyDenkberg Oct 1, 2023
63f5ad0
Update Packs/TAXIIServer/Integrations/TAXIIServer/TAXIIServer_descrip…
ShirleyDenkberg Oct 1, 2023
d7ddcb8
Formatted
ShahafBenYakir Oct 2, 2023
205a6dc
Formatted
ShahafBenYakir Oct 2, 2023
93a2fa6
Added rn
ShahafBenYakir Oct 2, 2023
4611054
Merge branch 'master' into long-running-migration-docs
ShahafBenYakir Oct 5, 2023
c8940a7
Merge branch 'master' into long-running-migration-docs
ShahafBenYakir Oct 8, 2023
b867433
Merge branch 'master' into long-running-migration-docs
ShahafBenYakir Oct 8, 2023
abf9c1f
Merge branch 'master' into long-running-migration-docs
ShahafBenYakir Oct 12, 2023
5fbf776
updated from master
ShahafBenYakir Oct 12, 2023
08c3d08
Merge branch 'long-running-migration-docs' of github.com:demisto/cont…
ShahafBenYakir Oct 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 4 additions & 4 deletions Packs/EDL/Integrations/EDL/EDL.yml
@@ -1,4 +1,4 @@
category: Data Enrichment & Threat Intelligence

Check failure on line 1 in Packs/EDL/Integrations/EDL/EDL.yml

View workflow job for this annotation

GitHub Actions / pre-commit

Validation Error DO106

The docker image tag is not the latest numeric tag, please update it. The docker image tag in the yml file is: 1.0.0.74096 The latest docker image tag in docker hub is: 1.0.0.75066 You can check for the most updated version of demisto/flask-nginx here: https://hub.docker.com/r/demisto/flask-nginx/tags To update the docker image run: demisto-sdk format -ud -i Packs/EDL/Integrations/EDL/EDL.yml
sectionOrder:
- Connect
- Collect
Expand Down Expand Up @@ -56,27 +56,27 @@
section: Collect
advanced: true
required: false
- additionalinfo: "Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances.\nNote: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use."
- additionalinfo: "Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances.\nNote: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use. (For Cortex XSOAR 8 and Cortex XSIAM) If you do not enter a Listen Port, an unused port for the EDL will automatically be generated when the instance is saved. However, if using an engine, you must enter a Listen Port. "
display: Listen Port
name: longRunningPort
type: 0
section: Connect
required: false
- additionalinfo: For use with HTTPS - the certificate that the service should use.
- additionalinfo: (For Cortex XSOAR 6.x) For use with HTTPS - the certificate that the service should use. (For Cortex XSOAR 8 and Cortex XSIAM) Custom certificates are not supported.
display: Certificate (Required for HTTPS)
name: certificate
type: 12
section: Connect
advanced: true
required: false
- additionalinfo: For use with HTTPS - the private key that the service should use.
- additionalinfo: (For Cortex XSOAR 6.x) For use with HTTPS - the private key that the service should use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) When using an engine, configure a private API key. Not supported on the Cortex XSOAR​​ 8 or Cortex XSIAM server.
display: Private Key (Required for HTTPS)
name: key
type: 14
section: Connect
advanced: true
required: false
- additionalinfo: Uses basic authentication for accessing the list. If empty, no authentication is enforced.
- additionalinfo: Uses basic authentication for accessing the list. If empty, no authentication is enforced. (For Cortex XSOAR 8 and Cortex XSIAM) Optional for engines, otherwise mandatory.
display: Username
name: credentials
type: 9
Expand Down
9 changes: 6 additions & 3 deletions Packs/EDL/Integrations/EDL/EDL_description.md
Expand Up @@ -40,8 +40,11 @@ For a Cortex XSOAR version lower than 8, the `Listen Port` parameter is required
To access the Generic Export Indicators Service by instance name, make sure *Instance execute external* is enabled.

1. In Cortex XSOAR, go to **Settings > About > Troubleshooting**.
2. In the **Server Configuration** section, verify that the `instance.execute.external.<instance_name>` key is set to `true`. If this key does not exist, click **+ Add Server Configuration** and add the `instance.execute.external.<instance_name>` and set the value to `true`. See [this documentation](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.
3. In a web browser, go to `https://<cortex-xsoar-address>/instance/execute/<instance_name>/`.
2. (For Cortex XSOAR 6.x only) In the **Server Configuration** section, verify that the `instance.execute.external.<instance_name>` key is set to `true`. If this key does not exist, click **+ Add Server Configuration** and add the `instance.execute.external.<instance_name>` and set the value to `true`. See [this documentation](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.
3. In a web browser, go to:
(For Cortex XSOAR 6.x) `https://*<xsoar_address>*/instance/execute/*<instance_name>*`
(For Cortex XSOAR 8) `https://ext-<XSOAR-tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>`
(For Cortex XSIAM) `https://ext-<XSIAM-tenant>.crtx.<region>.paloaltonetworks.com/xsiam/instance/execute/<instance-name>`
ShahafBenYakir marked this conversation as resolved.
Show resolved Hide resolved

In Multi Tenant environments, go to `https://<cortex-xsoar-address>/acc-<account name>/instance/execute/<instance_name>/`

Expand Down Expand Up @@ -79,4 +82,4 @@ Make sure you run the ***!export-indicators-list-update*** command for the first

### Important Notes:
- If constantly using different queries for the same EDL instance through the *q* inline argument, it is recommended to use different instances of the EDL (one for each query), and set each one with a default query for better performance.
- When using the *q* inline argument, the number of exported indicators is limited to 100,000 due to performance reasons. To export more than 100,000 indicators, create a new instance of the integration with the desired Indicator Query and List Size.
- When using the *q* inline argument, the number of exported indicators is limited to 100,000 due to performance reasons. To export more than 100,000 indicators, create a new instance of the integration with the desired Indicator Query and List Size.
25 changes: 14 additions & 11 deletions Packs/EDL/Integrations/EDL/README.md
Expand Up @@ -15,14 +15,13 @@ Unlike `PAN-OS EDL Management`, this integration hosts the EDL on the Cortex XSO

***Important Notes:***
- EDL is designed to spawn on two processes: NGNIX and Python. NGNIX is the process that listens on the configured port, while the Python process listens on the configured port + 1. This means that if an integration was configured for port 9009, the NGNIX process will listen on port 9009 and Python on port 9010. When running without --network=host, the Python port is not exposed to the machine.
-
- If constantly using different queries for the same EDL instance through the *q* inline argument, it is recommended to use different instances of the EDL (one for each query), and set each one with a default query for better performance.
- When using the *q* inline argument, the number of exported indicators is limited to 100,000 due to performance reasons. To export more than 100,000 indicators, create a new instance of the integration with the desired Indicator Query and List Size.
- Note: After a successful configuration of an instance, if the 'test button' is clicked again, it may result in a failure due to an incorrect assumption that the port is already in use. Nevertheless, it is important to highlight that despite this issue, the instance will continue to function correctly.
## Troubleshooting
- If you are encountering an 504 Gateway error:
1. Increase the NGINX Read Timeout in the instance configuration (for 1,000,000 indicators, it is recommended to increase the timeout up to 1 hour).
2. If the issue persists, try to increase the Load Balancer timeout through the Devops team (for 800,000 indicators, it is recommended to increase the timeout up to 1 hour (depends on the indicator query)).
If you are encountering an 504 Gateway error:
1. Increase the NGINX Read Timeout in the instance configuration (for 1,000,000 indicators, it is recommended to increase the timeout up to 1 hour).
2. If the issue persists, try to increase the Load Balancer timeout through the Devops team (for 800,000 indicators, it is recommended to increase the timeout up to 1 hour (depends on the indicator query)).
## Use Cases
---
1. Export a list of malicious IPs to block via a firewall.
Expand All @@ -46,11 +45,11 @@ Unlike `PAN-OS EDL Management`, this integration hosts the EDL on the Cortex XSO
| Exported Fields | For use with JSON and CSV formats - select specific Cortex XSOAR fields to export. If given the value 'all' - all Cortex XSOAR fields are exported. If empty - only value and type are exported. | False |
| List Size | Maximum number of items in the list. | True |
| Refresh Rate | How often to refresh the list (e.g., less than 1 minute, 5 minutes, 12 hours, 7 days, 3 months, 1 year). For performance reasons, we do not recommend setting this value at less than 1 minute. | False |
| Listen Port | Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances. Note: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use." | True |
| Certificate (Required for HTTPS) | For use with HTTPS - the certificate that the service should use. | False |
| Private Key (Required for HTTPS) | For use with HTTPS - the private key that the service should use. | False |
| Username | Uses basic authentication for accessing the list. If empty, no authentication is enforced. | False |
| Password | Uses basic authentication for accessing the list. If empty, no authentication is enforced. | False |
| Listen Port | Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances. <br>Note: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) If you do not enter a Listen Port, an unused port for the EDL will automatically be generated when the instance is saved. However, if using an engine, you must enter a Listen Port. | True |
| Certificate (Required for HTTPS) | (For Cortex XSOAR 6.x) For use with HTTPS - the certificate that the service should use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) Custom certificates are not supported. | False |
| Private Key (Required for HTTPS) | For Cortex XSOAR 6.x) For use with HTTPS - the private key that the service should use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) When using an engine, configure a private API key. Not supported on the Cortex XSOAR​​ or Cortex XSIAM server. | False |
| Username | Uses basic authentication for accessing the list. If empty, no authentication is enforced. | (For Cortex XSOAR 6.x) False <br> (For Cortex XSOAR 8 and Cortex XSIAM) Optional for engines, otherwise mandatory. |
| Password | Uses basic authentication for accessing the list. If empty, no authentication is enforced. | (For Cortex XSOAR 6.x) False <br> (For Cortex XSOAR 8 and Cortex XSIAM) Optional for engines, otherwise mandatory. |
| Add comment to empty list | If selected, add to an empty list the comment "# Empty list". | False |
| Strip ports from URLs | If selected, ports in URLs are removed. For example, 'www.example.com:9999/path' becomes 'www.example.com/path'. | False |
| Strip protocols from URLs | If selected, strips the protocol from URLs (http/https)/. | False |
Expand Down Expand Up @@ -144,8 +143,12 @@ In order to get the list of all available fields to search by, you can configure
To access the Export Indicators service by instance name, make sure ***Instance execute external*** is enabled.

1. In Cortex XSOAR, go to **Settings > About > Troubleshooting**.
2. In the **Server Configuration** section, verify that the ***instance.execute.external*** key is set to *true*. If this key does not exist, click **+ Add Server Configuration** and add the *instance.execute.external* and set the value to *true*. See [this documentation](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.
3. In a web browser, go to `https://*<demisto_address>*/instance/execute/*<instance_name>*` .
2. (Cortex XSOAR 6.x only) In the **Server Configuration** section, verify that the ***instance.execute.external*** key is set to *true*. If this key does not exist, click **+ Add Server Configuration** and add the *instance.execute.external* and set the value to *true*. See [this documentation](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.
3. In a web browser, go to:
(For Cortex XSOAR 6.x) `https://*<xsoar_address>*/instance/execute/*<instance_name>*`
(For Cortex XSOAR 8) `https://ext-<XSOAR-tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>`
(For Cortex XSIAM) `https://ext-<XSIAM-tenant>.crtx.<region>.paloaltonetworks.com/xsiam/instance/execute/<instance-name>`
ShahafBenYakir marked this conversation as resolved.
Show resolved Hide resolved


### URL Inline Arguments
Use the following arguments in the URL to change the request:
Expand Down
@@ -1,4 +1,4 @@
category: Utilities

Check failure on line 1 in Packs/GenericWebhook/Integrations/GenericWebhook/GenericWebhook.yml

View workflow job for this annotation

GitHub Actions / pre-commit

Validation Error BA102

File Integrations/GenericWebhook/GenericWebhook.yml is not supported in the validate command. The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
sectionOrder:
- Connect
- Collect
Expand All @@ -14,7 +14,7 @@
section: Connect
advanced: true
required: false
- additionalinfo: Runs the web server on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances.
- additionalinfo: Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances. Note: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use. (For Cortex XSOAR 8 and Cortex XSIAM) If you do not enter a Listen Port, an unused port for Generic Webhook will automatically be generated when the instance is saved. However, if using an engine, you must enter a Listen Port.
display: Listen Port
name: longRunningPort
type: 0
Expand All @@ -26,14 +26,14 @@
section: Connect
required: false
- display: Certificate (Required for HTTPS)
additionalinfo: Required for HTTPS, if not using server rerouting
additionalinfo: (For Cortex XSOAR 6.x) For use with HTTPS - the certificate that the service should use. (For Cortex XSOAR 8 and Cortex XSIAM) Custom certificates are not supported.
name: certificate
type: 12
section: Connect
advanced: true
required: false
- display: Private Key (Required for HTTPS)
additionalinfo: Required for HTTPS, if not using server rerouting
additionalinfo: (For Cortex XSOAR 6.x) For use with HTTPS - the private key that the service should use. (For Cortex XSOAR 8 and Cortex XSIAM) When using an engine, configure a private API key. Not supported on the Cortex XSOAR​​ or Cortex XSIAM server.
name: key
type: 14
section: Connect
Expand Down
@@ -1,7 +1,7 @@
We recommend that you use Cortex XSOAR server rerouting when using this integration:

1. Navigate to **Settings > About > Troubleshooting**.
2. In the **Server Configuration** section, verify that the value for the ***instance.execute.external.\<INTEGRATION-INSTANCE-NAME\>*** key is set to *true*. If this key does not exist, click **+ Add Server Configuration** and add *instance.execute.external.\<INTEGRATION-INSTANCE-NAME\>* and set the value to *true*. See the following [reference article](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.
2. (For Cortex XSOAR 6.x) In the **Server Configuration** section, verify that the value for the ***instance.execute.external.\<INTEGRATION-INSTANCE-NAME\>*** key is set to *true*. If this key does not exist, click **+ Add Server Configuration** and add *instance.execute.external.\<INTEGRATION-INSTANCE-NAME\>* and set the value to *true*. See the following [reference article](https://xsoar.pan.dev/docs/reference/articles/long-running-invoke) for further information.

**Note**: The ***Listen Port*** needs to be available, which means it has to be unique for each integration instance. It cannot be used by other long-running integrations.

Expand Down
10 changes: 5 additions & 5 deletions Packs/GenericWebhook/Integrations/GenericWebhook/README.md
Expand Up @@ -8,11 +8,11 @@ The Generic Webhook integration is used to create incidents on event triggers. T

| **Parameter** | **Description** | **Required** |
| --- | --- | --- |
| longRunningPort | Listen Port | True |
| username | Username (see [Security](#security) for more details) | False |
| password | Password (see [Security](#security) for more details) | False |
| certificate | Certificate (Required for HTTPS, in case not using the server rerouting) | False |
| key | Private Key (Required for HTTPS, in case not using the server rerouting) | False |
| Listen Port | Runs the service on this port from within Cortex XSOAR. Requires a unique port for each long-running integration instance. Do not use the same port for multiple instances. <br>Note: If you click the test button more than once, a failure may occur mistakenly indicating that the port is already in use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) If you do not enter a Listen Port, an unused port for the EDL will automatically be generated when the instance is saved. However, if using an engine, you must enter a Listen Port. | True |
| username | Username (see [Security](#security) for more details) | (For Cortex XSOAR 6.x) False <br> (For Cortex XSOAR 8 and Cortex XSIAM) Optional for engines, otherwise mandatory. |
| password | Password (see [Security](#security) for more details) | (For Cortex XSOAR 6.x) False <br> (For Cortex XSOAR 8 and Cortex XSIAM) Optional for engines, otherwise mandatory. |
| certificate | (For Cortex XSOAR 6.x) For use with HTTPS - the certificate that the service should use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) Custom certificates are not supported. | False |
| Private Key | For Cortex XSOAR 6.x) For use with HTTPS - the private key that the service should use. <br> (For Cortex XSOAR 8 and Cortex XSIAM) When using an engine, configure a private API key. Not supported on the Cortex XSOAR​​ or Cortex XSIAM server. | False |
| incidentType | Incident type | False |
| store_samples | Store sample events for mapping (Because this is a push-based integration, it cannot fetch sample events in the mapping wizard). | False |

Expand Down