Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Devo Integration #4419

Closed
wants to merge 4 commits into from
Closed

Update Devo Integration #4419

wants to merge 4 commits into from

Conversation

ansfan
Copy link
Contributor

@ansfan ansfan commented Sep 17, 2019

New version of Devo integration with enhanced features. Includes new features for fetch-incidents, querying alerts, multi-table querying, and writing back to Devo. Using newer version of the Devo SDK which has been updated to use Devo's most recent APIs.

Status

Ready

Related Issues

Description

New version of Devo integration with enhanced features. Includes new features for fetch-incidents, querying alerts, multi-table querying, and writing back to Devo.

Refer to README.md found here: https://github.com/devods/demisto-devo

Screenshots

image
image

Related PRs

List related PRs against other branches:

demisto/dockerfiles#462 <- This is assuming demisto/devo:1.0.0 docker image name to be built.

Required version of Demisto

4.x.x
5.x.x

Does it break backward compatibility?

  • Yes
    • We are upgrading this integration to use the newest Devo HTTP APIs. Also using a different authentication method for enhanced security.

Must have

Dependencies

demisto/dockerfiles#462 <- Dockerfiles to create image needed for Devo's Python SDK.

Additional changes

Re-write of entire existing Devo integration.

Technical writer review

Mention and link to the files that require a technical writer review.

@welcome
Copy link

welcome bot commented Sep 17, 2019

Hi and welcome to the Demisto Content project! Thank you and congrats on your first pull request, we will review it soon! Until then you can check out our documentation for more details. We would be thrilled to see you get involved in our Slack DFIR community for discussions. Hope you have a great time here :)

@guardrails
Copy link

guardrails bot commented Sep 17, 2019

⚠️ We detected security issues in this pull request:

Hard-Coded Secrets (1)

https://github.com/demisto/content/blob/6fc8aa6c12ca5b16638be77c0d20e6501f69aed2/Integrations/integration-Devo.yml#L7

More info on how to fix Hard-Coded Secrets in General.

Happy with the results? Give your feedback.

@content-bot content-bot added the Contribution Thank you! Contributions are always welcome! label Sep 18, 2019
@content-bot content-bot changed the base branch from master to devods_devo_base September 18, 2019 08:54
@content-bot
Copy link
Collaborator

Thank you for your contribution. Your generosity and caring are unrivaled! Rest assured - our content wizard @anara123 will very shortly look over your proposed changes.

@CLAassistant
Copy link

CLAassistant commented Sep 18, 2019

CLA assistant check
All committers have signed the CLA.

@ansfan
Copy link
Contributor Author

ansfan commented Sep 20, 2019

⚠️ We detected security issues in this pull request:

Hard-Coded Secrets (1)
Happy with the results? Give your feedback.

Guardrails - Base64 encoded image not a secret.

Also for the circleci might have to whitelist the default devo.com domains so not to be categorized as secrets.

@anara123
Copy link
Contributor

anara123 commented Sep 23, 2019

@ansfan Hi. Thank you for the contribution.
We changed some the way we store our integration. Please take a look at https://github.com/demisto/content/tree/master/docs/package_directory_structure

  1. Navigate to content folder (the folder you cloned to your local env)

  2. Installation prerequisites:
    python3
    python2
    pip
    pipenv
    pyyaml
    ruamel.yaml
    docker
    autopep8
    flake8
    mypy

  3. You should run in order to convert yml to package format (replace the old integration-Devo.yml with the new Devo folder):

PYTHONPATH="`pwd`:${PYTHONPATH}" python3 package_extractor.py -i /Integrations/integration-Devo.yml -o Integrations/Devo -m -t integration
  1. Then run linting
PYTHONPATH="`pwd`:${PYTHONPATH}" python3 Tests/scripts/pkg_dev_test_tasks.py -d Integrations/Devo
  1. Validate files before you commit your changes
PYTHONPATH="`pwd`:${PYTHONPATH}" python2 Tests/scripts/validate_files.py
  1. Add CHANGELE.md file to the package

You can take a look at the following integration example:
https://github.com/demisto/content/tree/master/Integrations/ProofpointTAP_v2

@ansfan ansfan closed this Sep 23, 2019
@ansfan ansfan reopened this Sep 23, 2019
…features for fetch-incidents, querying alerts, multi-table querying, and writing back to Devo. Using newer version of the Devo SDK which has been updated to use Devo's most recent APIs.
@ansfan
Copy link
Contributor Author

ansfan commented Sep 24, 2019

@anara123 Please rebase/fast forward the demisto:devods_devo_base branch to the master git commit: 857896ea12c0c360defe1f559390def192fbc4de.

I have done this in order to make sure I had the latest testing environment setup. I believe the rebasing is blocking the license/cla check from passing.

Let me know if this looks okay to you and I also have included pytest unit tests but have set myself to mimic the proofpoint integration by skipping the playbook test.

@anara123 anara123 changed the base branch from devods_devo_base to master September 24, 2019 17:03
Comment on lines +55 to +78
### Devo (Preview) Commands ###

`devo-run-query`

Runs a linq query and returns results from the specified timeframe.

*inputs*

| Argument | Required | Type | Description |
|----------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| query | TRUE | string | A linq query that you wish to run against Devo |
| from | TRUE | float | Start datetime for specified Query. Expects a unix timestamp in seconds with milliseconds allowed after decimal points. |
| to | FALSE | float | End datetime for specified Query. If no value is provided will default to now. Expects a unix timestamp in seconds with milliseconds allowed after decimal points |
| writeToContext | FALSE | string | String of `true` or `false` to pass on the results context. |

*outputs*

| ContextPath | Type | Description |
|-------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Devo.QueryResults | list(objects) | Will return a list of python dictionary objects. Each object in the list will be a row from the query. If no rows are returned result will be null. |
| Devo.QueryLink | object | Will return an JSON object with a single key called DevoTableLink and a URL link back to Devo for the results just queried. |

*current caveats*
- None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_description file should only contain instructions on how to configure the integration in order to connect to Devo instance.
For each parameter means like you explained above.
Remove the documentation about the commands. For commands you need to create README.md file in this folder and generate documentation file and fill it with data.
Follow: https://github.com/demisto/content/blob/master/docs/integration_documentation/README.MD

@anara123 anara123 changed the base branch from master to devods_devo_base October 7, 2019 10:08
@content-bot
Copy link
Collaborator

This PR is starting to get a little stale. @ansfan are there any changes you wanted to make since @anara123's last comment?

demisto.results(write_to_table_command())

except Exception as e:
LOG(e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove LOG(e) and LOG.print_log(True)
return_error(e) will do them both

except Exception as e:
LOG(e)
LOG.print_log(True)
return_error(e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return_error(e)
return_error('Failed to execute command {}. Error: {}'.format(demisto.command(), str(e)))

@@ -0,0 +1,145 @@
commonfields:
id: Devo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to Devo v2

commonfields:
id: Devo
version: -1
name: Devo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to Devo v2

id: Devo
version: -1
name: Devo
display: Devo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to Devo v2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This integration breaking backward compatibility with the previous integration. The instance parameters are different. In this case we should create new integration v2 and deprecate the old one.

name: Devo
display: Devo
category: Analytics & SIEM
image: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHgAAAAyCAYAAACXpx/YAAAAAXNSR0IArs4c6QAAFAtJREFUeAHtWwl0HMWZrqrunhkdyIewOSKMMWCDDT408qHbEFuHDY45RAgYkgcb7n27YV9MQghxSB6EZCGAIV7W4LBseAbEETAgS7bjS7Iko5F8YCB+2BZgfOAL2zpG091V+/09amk0npHkgwfsm3pqVXfVX39V/Vf99VcNY4mUoECCAgkKJCiQoECCAgkKJCiQoECCAgkKJCiQoECCAgkK/D+hAD/ReWTllUxljI/oYOLNzdXvHurEI5q2sRJdZ+Msm8XFjXomFdtat4+9dXsWM/szhkmTJqVb3sE3MyWTGddUjzbK5pyzI8rUmjXBP29bl/LhFlYe6gET48OfNy2PCc9UpZSMUd2ziPoQzNaCbW+sX1+91e+fNoynGddLS+pC6Vwqa01jTeXano3if03IK54mhDZFKVtyrkIe07O4tvadL2K18OeUns+FzJWCz+ZMFTDFOAjwMZdsKZPa3wO1722O1Y7KQOrjT1k5xT/gGn9N6LruCZmlwPAjPNbGZnaH4WHPEEatN8xgvSEYm3g6exCgvyP4vpLtPS3fMPTHY/OiszMSHGkHfQVHt/pV8WuszbMgEFiyPy5uod2pG9oNStpxQbordMaFTtLoRdlv7VQ1ysP4o0LXGEmXMvmnY8YUZ27ZUnmwu03sNxJWKdgiTRfnqE5RNaXVAOgeDPZPKc5UBv85E7IEwjCQuOrMH/QTXORwneXYNvuFv6CkmnHzD4HVK1ZG9wgyH39SBvNruqE7hOEqNzd3VpKDRbE8H6ZvgQrQ4PgP6kEXxmxW2N/elTSElBIM7P2BVfFpmhiL8T3EUkO1mbnF18XrAwTTaA594XTryeworjk027B6xWpbmtWE27YtCLQ41ztYXBOvr8hy2zPwh0IzzqF2lCzbXqPa99VEwvjzi/6Ve7UVEOrrBdcG0hgihZveqYxzmYz5FnFlLPUXFj8EHJ5IPCfEYK1d+5tpdQSUVLshU4/X1LzdQkgtxRa0trOPbMkOgMn74z22Yvvb29l2COQTkYM5vncF6qggaN5BD7UVQuDRQQjSZAtSrl3ANW2xP7/417Fwwy7vA41CLo6+clvJDmGrw524QlzqT7oa6PTP1Z0ZGRlhYY/VIcoyMsqSwJXbXGZRe6HU/EAg0LVU+QunP6zp+lMwDANdIaC50RoBeMfchOcaLiNGw7x4NGH8OjO/5H8uKC0lK+Ok3gypC3NMvn79e1vHjh1baBhnpQUCVWByOPlHsLUN21i26GADpRcrRZwk2hj/ymJH8scyd+2OAxm7WIP9t0zzBYj+U7xTYC1mnSY0MYoLcypmOxtSnUYTByEgxNpDmfnT9jSuXb4wEqOp9AcNaS/klog71kh4qdv8yB7e7JaFDn9RYQw8exO6GBsmMh9/xnmjinfu3Pl3FyY6HzKsZabQtHHEYGKYtO3GA59vfceF8+cV3aUJ/ZfEM4ipI7S2tFuZxV9WWugtJvVdzGK6ZYiLOZMz4enMwFyTqX96SOPTWqydaPxzwhDXEaLKb1PKzC25WjPE60QYh8Gh0O8D1VUxNXNcdukEOHIvaDp3CO8QUtmHWUjL780hOZH5+vNn3AHzvMCxGJAl27aXBdZklzA2z2FRD5xlZZp/z5Hlmq5N7RQ+ZprWrU3VVYsIbsqUmRebXqsWlmcAzZNkkwQAduPuxvqKuh64Oj8y84r8WDWe0ITII5w0V6i5ZUlV0rR26Qp94sQZZ9o+9St4ciN6GPlY2I63DL1pGt9vtsrHm9ZXbqTmm/7JLpI+dh/GMlRABB2UGBcsawgmverwCPbcZRwy2lfSGK3iMdPG2oqmsQUFs5lMXoaJn08T1zVjgK1bc9HgppiNTrAwpPgrHsuci4meFyYwu8yfV50TqGbO+hyJdsL+ljxQJN9lhG3LrcInX3dhTMOa64wTa7PDXGl/AJgrm+ord7kw0XljdVXg0ryZs7zKegdjyCHcWKZ0IZ25/kO3k+xXDd3Id9eEaAQn+40JMc0XGl5WVnZ5eXm5LT3sz2mprCTorJrd2LHeMENnV7EdjCbzdnfNib1tWrNmx4Tcknvhab4BDBpNXAl1xfgpJcM31C1tPjGsx7aiLSK06Hkw5PdgBoir6Vgk7wLkMQwWUt0toJ4YSljTpHw2sHy5s6ZfMrHoHMX5D2iclODfBG0h7mmqic9cBxD/aAwTJhXfxXxqDVQFS5MF/Cx/3MTi0fBIWJoL+LXkzurGk8Bc7CWwJOxgg8ixiHROnH5RRlsrxVmvTsrxjLGpZul7mQUlTbomsohwGrYayjDzgKOZ8Pj9VybrujWY3vtOB836+vovAefMKBKet/NFlmbdg63LmZ3zuhLEHbPx/cotLtzYnKmXQMJmumuvbZu7dNP7klvv8fBsjHNQWANh6i17adPqytVufV85WcjMvOLXdMO4xcGhiSQtSZbp0uSzLW7Ng0yNgtic2jUZ2PC3BxbnUQwQYgW3oZn9LNjO7rMs5hLDGbvmYbKjjdXtOsCW9DWZ46i3sAqs41zPIocF/io1vbSrfXLoMSm0myXWu95SeF0btOeSybNyPqh/e280bCBQuTszf/qLXNfnkvbAm01VSeZPAffvLqxHeGmtTg5rOcYhrUX1kbg0Ptrpp3PVQlbptu1/zuGsyVsIngSNK3Wt3mmufsJYmcbKIlCVl4dnXVbmUMWpKR+DZr85RoIjWsV4pe15d8oczmrxNbu75Ot9U4p/Ebn8cK6luz3CWpwLnyQZEaFeEy0fMPCDmQz22GP2aGSJhVKYtwH/QEdLGbs+O/uKP1F0aiLMLwIb15NmEROlbR40rNDzke3RxRDXqlGuZGhHZH1/3rkld0nN2UZpzhgEOx9G0U3lNit33xm74ILStAEDdCtQXt7mljaociO9dV76oYNM+JKONVUuXGQu2hmHxmL83UmirQ8Uc0uC2M55jzK5ZTTbdx0P7/PcupPN0TH2hCSjYS4qUjE3de4p3U/aQxMzo5PDFFN6sc52jTkaprG28hPst1+Dp/8vtHfVdeOMDmX+GHAP2152C/ycdCqnPiSXL9XVrWqOxBEdLgVcXCcysl3ku52MDQYtv51zwIytCAZ3gyIUeQ032H9i/u3jp0z/6Ya6ZTWNH7JztWb2covOzudA0nG0G77PN6MnBNqHIxOdxaRBMo2pC5tZU8MudmPW2Sx+eLEnqn588VHuhCkHsz6N18g2reeh1f+Mrkf4hHEpD9rDBnzJ6qNru7+lCj1tW/xGODpJjrYy/uNJBQWLEcWe42ovol8tqkNf0N0q/MYZ39U9TgQ0hDYGNe9Fw/X2zTvExZpHOBE/x9xLe1MsBiN2zuZqmjacJF95zbuBtEb52JzUNDalFTErOiw4lYnUgsxSaiorOtrKZuFz0anAP2FC6RAuVCERl5JlmQpePcV8YyYrJP64EUGcmJX9KGyqXrnRX1C8BFuz6xxDwdWFtkr+X8zufAUL4lgIZb8ZqHv3o2h0tq02CS08TmdB4Oxqv9//RGSEK7pN1Dd4qq51y0hYYG8Wd6+vbg2ww91dR59kx5Ul1tM7ALfC1CocJjjeLnm8p+ohgfFAy9tboNgh9gn1dyqSSLN+BqJmOOuREwDgO7RkFVcHNS926CeZlM2exoGHE3aErYQ8iVzKiYKISHVw05ofq4ugdagO49xJmhfWdj6Z+dJviAUbqywz7/szIVjTui2F/ZVq87wdUxf3bP/w/iHnja4XTAvy4P53CeHDDeyNueNZkcfDRlrwFWJ1cjJlHixNOBnZmDmSOcJ1Mrio7YScIkizdq+rvY7JUtbC+oqqIyeLu7f2ODKsgRavggWcTn2TcFEKR6WsqvfXrYhpQT5ev/4A2r2EsON95Gk7QqHzx/yFMz8LrH53ZW99+qcUTUbMfQEaOYF46gvbrDcaA0s+i8lgxFLb8bwcibT8OmbDB1uOMnq+BYlW8mPTuOwrvqdroTu4EP8BInmJwI5jY5vrgoe+ePrYFt0lvE32eYbcDR33Dbsu7SnJ1TRAdCkCHCwplf0kymhFiplE+8EnzKTBP8R+fXjY6mjpCHi87s8tvt/HvC/hUKeH54PzgBRjwJm3wEz8BnNND7cRWIqsfTjXfIQ66RpAzB6Po5BionBQLiRTxDE7aiqEjW2oOuJlSWvdwTUoBKya2eWQ6+Dbw9naeW64so++ImPRpI2I+a5DtgJnSmFvU+dJOHUfgbIpiCadEV52ibnwI6TaaXe0FjXWremx9mXmFy/Rde0K16yBAy9i5O/HHQp6Uh3tywO1qz6OC4MKOs0Z2CpXInSYTbgd7bXk6oa1S4npMQXTxTehsLQUvvzr4EySyzCqkwhrYlFdgSO77XDipOLqPDhml2NvPdrZVkGQw1ZKWdDeOY01Va9Qu5gaTBXHk/wFpUVYZV5HGC7V2cuFSe5YDGJ4ULYvRvB9DgXgwdyHvF72CxzNydnN7K55jD17PH0RLE0cJjAHhMtxI9JOv4JMIogBotJkSXNty9pumfKmDVHMje4zjFPcjHY3R9e53xR2NaVNpzS9MviTiooOf17x0xhbNrWF4Cg4UbT29spcgm1aXVGRlVd8B0zuf2F+jjdO5VjLR2JsIzk2fZRcptJcKTlCJBXuDci5LnOdcqf2JP9hDzcdB+yp1BkRyn3czpnSJmdkbHGGBvrPglIxMJkcvCn97Rp99LA21AftK92HvFaatOPO0F5TsjbLsl8MsbZpG2qrYq7r0IIeTiaN18UXKycnAVdkHOvU17jNI7vfgpX5iE6+MNamowPEO321cesbqitfxCWAWdJWTcQ4esKCGz3fsHVwBNlWH0K7r4UP0OOM/ZRoMGzgyzifLcbqMira6COw0IodwjM7d5a30wTApSdxajQv1IHTXNX/7ZBmWEekwkkzxbNjJwt6sh/125mwapmpLw7UVmyODRouFZwfBrdCRLz+JMwFE5DugX+vTTZt2tSaWXDWMzCnT0M255NW99ogqrKpunL5qNxZhSms42bM6QYI7jgsPSmRYBDIdgjRJkjzqyGm/XVzTdfduC6weMTqAujvy5gxY1KTBw0fGg0fOso7Nm7seZms5hM21IubMlmj+h/QwJ7Q0FJSMpQ8LeaYLU+7bbS0tKyHNxo9hnjfdDdKeYYMiFcfq3x388bd5ITGqosuI5r4Bg+btnfH5sr+tonGEf6eJzKza0cIj3WetDwOjbnG9nFbfdpQs3QbYPo0/bHxJkoTFEhQIEGBBAW+QQrEXM++wfH01jVdLzo9pAtNtX72FTkxvQF/HXXwA063PGd5cdWWG5Ye7PXOdR8DKMVeee8RNdM6uqvyVMxl0qSr0qXRMrmhZlkFuu5yG3tsE/oY0zdanZUzFXe5ZJ1uWBXeQWdV42juQQyoX7uAzPyiP2bhIt7JTWCeYCmnv2p45HKPj72nkkKPngy+HTtCqVzwR4Q4e8jJ4HHbSi04DMGB3/r9t/WgSY8PF/jbmCvpSWG6OmqZR0p132kZuNuzOKug+EDDmspnaLyX5uWN8IhUhCdVEgL6CxDzdSJS47ILJiAAcxP38CH4pcD8QF1lI8Hjus4wnhSai4BCim2pZxtrYt9aJFhKZWVb+Pa97CyEAB/wCfEPpQzXc+UTcotvF0GxLBCo2JaV+/1bOe9Y+X519XZqh7tS44RPuw1bmd3BQ58/sWXLlhYqx/4YGz4ZwgUBR9vIo7c9g+aw9v1/se1zUvSU9jlD0sTCis7tVVZ+6bUIYBXjMGMN9sl/AwqnnT976kXc48M9azUcZRhTAFl3+s5osDNkXHBvrR24P7Cyql4p8RzidWXuVDztKdi62M5Rn63p97nlOk85Ath9CBC3maKjK9bcofMOUHgrTn4Q7JS/dOHj5eV0mwUH5EpX9weF9UpQM50oFeAV1/lIlSzvycwuvkAK/U4eCjl75bGTSzM0H1+II7mPEdUa6x2U8Yd4+JO/HN2KKE0ZTx04VTut/Vr8DikXzHVOpYi5istfSSkacTFvXlbhjGmEBz+VGawM318R89mHsOWnWG87Y4jdvXynGIyAuvyEhQMGuG80AgSnS3BOUklsHH6JMhJBC/xYiCW75YF1FdvA3O0I4b2yad2qD9xyj9ccgzjWKHzrXPE+jwlJg6ktYtWv2Io/Ko+yJheXEeSPg8ATmcEeA79fcvfiUNJMaNbewNqq+eqouh8/tZiMNjFpvqr5hSAY9biU+lzA3MhN40/IEVnBPyFn4EJfEBqPH6FxHN+GYybJaeoi+lVUYO3SR2So9b/RNzTYT026UszOumq/RS84uEBEV51BF80h0X8GEy/DWcYj7hAR6bkGhw3bceGnFozvcYdEA/URlL9sfM7U8S48TObV3GY7peT44RZ+bYVL6fQ7Jron3gUT/cKJOfx0TYmzuU92UbK+vmIncFSD+VmmanvBbabzlg8wzoxJhdOvYins3yAhdLbuMC0MgztAERcdeXDfEsAMQh/7A3VLnKXEgVMsgKhVEIc470iLv6lavlpF5UIe3gaBTsOFvznM8P4ENtt74ECVlpk3o8Q/rcwJ4HxnGAzbeAgm7CuY4bmYWbqyVHEDLrg7BMA/OyifgUbegwcX+niVW045TPRSaP8DmvAWuuXKFH8Bu24FictgZJcxXOvFLyZvF17+PRcmMse1XzrB+ABnJNfgfvVDwsNuiKwXlvYy8Dy2ubr6kFtO6zAWgAcQMV6EuqF6Cu9aCnw+HD8o1WSaqut82rm9IdV83Dh9zsVBeceatIVg3gZhiJWY+1WHhw51+IZrvHsRCr0XR6PzEe+/EKdmb6Snpwsl7B9poWA/rwNH9pR4T1AgQYEEBRIUSFAgQYEEBRIUSFAgQYEEBRIUSFAgQYEEBb7FFPg/dVlB0aejt4MAAAAASUVORK5CYII=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undo the deletion, if we will delete this integration and replace by another , then we will break backward compatibility

@content-bot
Copy link
Collaborator

This PR is starting to get a little stale. @ansfan are there any changes you wanted to make since @anara123's last comment?

@anara123
Copy link
Contributor

@anara123
Copy link
Contributor

Closing due to inactivity. @ansfan please reopen the PR when you have time to continue the review process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Contribution Thank you! Contributions are always welcome!
Projects
None yet
4 participants